-
-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
StackStorm v3.4.0 Pre-release Testing #66
Comments
st2ctl status showing
in the result above it is showing st2chatops not running.
Even after the machine restart, the same thing happens. |
@himadriganguly Thank you for testing v3.4a! That's normal when you haven't configured st2chatops by modifying We should try to improve that situation in the future, but for v3.4 I'm not considering that a bug. If you have configured st2chatops, then that's definitely a bug. In which case, please post your |
Running Self-Verification Result Click to expand!
|
@blag Thank you for the explanation. I will be testing other things according to the checklist and will post the details. I will try to contribute as much as I can. |
Just a short update since I ran into an issue with the load_diag workflow in the Linux pack and I'll look into that now:
Update: The issue with the load_diag workflow isn't a regression or issue specific to st2 3.4 and there are already open issues that cover these issues. So I'll proceed. |
I run tests on Ubuntu Xenial (16.04) here are the issues I found:
This may or may not be expected and desired, depending on how we envision whole Python 3 story. It returns python 2 since system default python binary in It does appear to be working fine for python runner actions though (which is likely the only thing we care about) - vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_version
.
id: 60319401b8c38397f0e40066
action.ref: examples.python_runner_print_python_version
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
end_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
result:
exit_code: 0
result: null
stderr: ''
stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26)
[GCC 5.4.0 20160609]
"
vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_environment
.
id: 60319560b8c38397f0e40069
action.ref: examples.python_runner_print_python_environment
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
end_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
result:
exit_code: 0
result: null
stderr: ''
stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26)
[GCC 5.4.0 20160609]
Platform: Linux-4.4.0-161-generic-x86_64-with-Ubuntu-16.04-xenial
PYTHONPATH: /opt/stackstorm/virtualenvs/examples/lib/python3.6:/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages:/opt/stackstorm/packs/examples/actions/lib::/opt/stackstorm/st2/lib/python3.6/site-packages
sys.path: ['/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner', '/opt/stackstorm/virtualenvs/examples/lib/python3.6', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/packs/examples/actions/lib', '/', '/opt/stackstorm/st2/lib/python3.6/site-packages', '/opt/stackstorm/virtualenvs/examples/lib/python36.zip', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/opt/stackstorm/packs/examples/actions/pythonactions'] As far as core.local "issue" goes, we need to decide what's the desired and expected behavior. I believe we should not add In short, we only care about python version being used for Python runner actions and StackStorm services. If that's the case, testing instructions need to be updated.
It could be that this has been broken for a long time already, but I do know it worked at some point in the distant past (perhaps we just don't source source the bash completion script by default, didn't dig in). |
@Kami I agree about |
I'm done with the tests of fresh st2 installations on CentOS7 and Ubuntu 16.04 now as well. I created an issue + PR for a small issue with the install script on Ubuntu 16.04 and can also confirm topics reported by @Kami This means:
I also did a cross-check with st2 v3.3 and the st2 auto-completion does not work, too. So this is not a new issue. I'll proceed with tests upgrading st2 v3.3 to 3.4 on both OS. |
Started testing on CentOS 8. |
StackStorm/st2docs#1055 - doc change about EWC components |
Simulated the E2E tests setup - and found that the ansible-script in the user-data didn't seem to get applied to the instance, hence there was no winrm listener on 5986.
|
Quick play with RBAC and ChatOps (hubot) on CentOS 8 ansible install, all good. !help still worked with rbac enabled. Ansible install on CentOS 7 - went through UI testing list on Chrome. Couldn't find a "Contact Us" - think its replaced by "Support" - so might want to update https://github.com/StackStorm/discussions/wiki/Release-Process#manual-testing-for-web-and-flow-ui Followed manual instructions for CentOS 7, all good, and setup ChatOps with manual instructions successfully. |
Possible upgrade or pythonpath issue. Upgraded StackStorm 3.3 on CentOS 7-> 3.4. Got error on aws pack after upgrade:
Problem is that the virtualenv of the pack from when it was installed on ST2 3.3 has python2.7 files in the lib area. Updated PR StackStorm/st2docs#1056 to include instructions that need to uninstall/re-install packs after upgrade on U16 and EL7. |
Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11 I ran into another issue during the upgrade von Ubuntu 16.04 with
And a second execution of
Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added. |
@winem StackStorm installer doesn't support curl-bash script re-run and idempotence, eg. resuming installation at this moment. Something that we could improve in the future for sure!
I think what happened there is failed |
@armab yes, it's fine that idempotence is not supported. I just forgot that when I ran it again. :) Anyway, the deadsnakes sources file in |
I have a fix @winem for this - StackStorm/st2-packages#691 I was looking into it as the E2E tests started failing... |
Updated the helm chart StackStorm/stackstorm-k8s#182.
Incorporated the suggested updates. Have this up and running in my K8s
cluster with RBAC and LDAP enabled, So far everything looks good. Just
noted some modifications to be made to the cli. Will submit a separate PR
for that.
Using Ubuntu 18.04 for the K8s cluster. Kubernetes and Ansible pack fail to
install with the error: *“error: command 'x86_64-linux-gnu-gcc' failed: No
such file or directory”*
Not sure if this needs to handled at the base image level. fixed it at our
end by creating our custom st2actionrunner image but it might make sense to
package this dependency in the base image as these are very commonly used
packs and pretty there would be other packs which would need this
dependency.
FROM stackstorm/st2actionrunner:3.4dev
ARG APP_VERSION=0.4
RUN sudo apt-get update
RUN apt-get install -y gcc libkrb5-dev
Regards
Harsh Nanchahal
…On Tue, Feb 23, 2021 at 2:48 PM Amanda McGuinness ***@***.***> wrote:
Upgrade on CentOS 7 went fine except for the issue(s) reported by
@amanda11 <https://github.com/amanda11>
I ran into another issue during the upgrade von Ubuntu 16.04 with
--u16-add-insecure-py3-ppa:
20210223T193442+0000 The repository is setup! You can now install packages.
20210223T193442+0000 Reading package lists...
20210223T193442+0000 Building dependency tree...
20210223T193442+0000 Reading state information...
20210223T193442+0000 Some packages could not be installed. This may mean that you have
20210223T193442+0000 requested an impossible situation or if you are using the unstable
20210223T193442+0000 distribution that some required packages have not yet been created
20210223T193442+0000 or been moved out of Incoming.
20210223T193442+0000 The following information may help to resolve the situation:
20210223T193442+0000
20210223T193442+0000 The following packages have unmet dependencies:
20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000 Depends: python3.6-dev but it is not installable
20210223T193442+0000 E: Unable to correct problems, you have held broken packages.
20210223T193442+0000 ############### ERROR ###############
20210223T193442+0000 # Failed on Install st2 #
20210223T193442+0000 #####################################
And a second execution of curl -sSL
https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin
***@***.*** --unstable --u16-add-insecure-py3-ppa fails due to
the already installed mongodb instance that was created during the first
try - incl. the users and databases:
20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY [js] Error: couldn't add user: command createUser requires authentication :
20210223T193946+0000 ***@***.***/mongo/shell/utils.js:25:13
20210223T193946+0000 ***@***.***/mongo/shell/db.js:1514:15
20210223T193946+0000 @(shell):1:1
20210223T193946+0000 bye
20210223T193946+0000 ############### ERROR ###############
20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) #
20210223T193946+0000 #####################################
Digging into it now. It looks like the check if Python3.6 is available or
not has some caveats and the ppa was not added.
I have a fix @winem <https://github.com/winem> for this -
StackStorm/st2-packages#691
<StackStorm/st2-packages#691>
I was looking into it as the E2E tests started failing...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#66 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AD2PWBSGGXDE44SPM2MJGA3TAQWFNANCNFSM4X5BVMYQ>
.
|
pip reaches for gcc only when it can't resolve a wheel to download so it needs to build a wheel for a package that is not pure-python. Test infra uses a newer version of pip than what we install in production. So we're unlikely to find such errors while testing (before we cut a release) because pip is new enough in testing. Please, let's merge these PRs to sanitize/make-consistent the pip version we use everywhere. |
I disagree. If we explicitly pin one version of a package, but we test with a different version, then that is a bug. |
As far as the ansible pack is concerned, that note in the readme is somewhat incorrect. gcc is only needed because the version of pip we're using in production is broken. Ansible is a pure-python package. It depends on cryptography and PyYAML which have non-python components. Both cryptography and PyYAML provide wheels, but the old versions of pip can't use them so it tries to build its own wheel instead. If we used the same version of pip in production that we have been running unit/integration tests with, then this would be a non-issue. |
Tested manual and ansible deployments on U16. All good. Not had chance to confirm upgrade on U16. Quick play in those environments with workflow designer, and the st2-self-check all ran successfully. |
@minsis This is great feedback, would you mind (if you haven't already) create a feature request on the st2web repository for these. They wouldn't be regression problems, but good points to improve the UI in future releases. |
I'm running into another issue with the upgrade from 3.3 to 3.4 on U16 now. It looks like an issue with stdin redirection again. It appears if the st2.conf was modified and does not match the one provided by st2:
So it does not wait for any user input and fails instead immediately. This seems to be the same as: StackStorm/st2-packages#688 The upgrade seems to work fine when I revert all changes on st2.conf and run the curl-bash-command. At least the self-verification was successful. Still have to do some more tests. |
It looks like we can avoid the stdin issue as described above and in StackStorm/st2-packages#688 by using the following command:
instead of the original one:
The one I suggested does not have any issues with interactions and does wait for the users input. I tested all available options Y/I, N/O, D and Z successfully. I do not see any additional issue or (security) risk by changing the commands but am happy about additional views on this suggestion. |
That enhancement would be helpful (please submit a PR), but we don't support upgrading st2 versions via |
PR provided. And yes, it's just for testing/development purpose. 👍 |
I'm done with the tests on different installations. Let me just summarize the tests here for a better overview: Testing means:
Tested Versions:
|
@winem - Did you follow the manual upgrade instructions? On CentOS 7 when following the manual upgrade instructions there is no problem. As the first instruction is to stop all ST2 services, therefore st2resultstracker is stopped at that point, and then never restarted when the ST2 services is restarted. |
I've just installed a pack on a CentOS 7 upgraded to 3.4, and I can edit the workflows. But I do need to make sure it has fully loaded everything, so when I click on action I can see that the action has loaded etc. I installed one of my own packs after upgrade, and could edit its workflows fine. And I can't see any difference in ownership of files. @winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention). |
Successfully performed U16 upgrade, using the new st2ctl reload register-recreate-virtualenvs. I couldn't necessarily find a suitable place to document the problems about editing workflows with the designer that had retry or delay parameters.... There are pull requests in progress for those, so could just create an issue in st2web and link the in-progress PRs to that? |
The error I saw was not related to these properties. But I was able to confirm that there is a very specific error message for these cases which is good! To make it short: Both of my the 2 issues are obsolete now. @amanda11 to me, it looks like we're fine with testing CentOS7 and U16 now. Do you see further test cases? If yes, just let me know and I'll proceed with them. |
@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕 |
@winem I think we've hit everything we need to, unless I'm missing something. The only thing of concern to me, is whether the issues mentioned on docker are blocking or not. Neither myself or @blag are that familiar on the docker use-cases. Other than that I think we've tested all the scenarios, and fixed the regressions.... |
Yeah - I see they were in progress. And those problems existed in EWC - so agree not blocking for 3.4 release. |
Thanks! There is no blocker from the Docker side for starting a release. In fact, all the deployment methods (Puppet, Ansible, Docker, K8s, Vagrant) really come as a secondary comparing to So we're good from that standpoint 👍 |
Ok I'm going to try and explain this the best I can here, sorry if it seems like ramblings. Something happened with my docker setup when I enabled RBAC with LDAP. I'm still trying to troubleshoot why this is happening but one of the containers is continually trying to authenticate my user and basically locked out my account. My password was reset by our IT team but my account is continually locked out (probably due to caching) and I had to disable LDAP to get this to stop. Here's the events in order:
So I'm not exactly sure where or why this happened. I looked at the logs of all the other containers and none of them were trying to auth my account. It was solely the auth container that was trying to auth my account every 5 seconds. |
So I have LDAP disabled and then reenabled RBAC only and its now failing again due to invalid login. So something has to be cached in the database and even with LDAP disabled its still trying to reauth me to the LDAP backend [auth]
mode = standalone
backend = ldap
backend_kwargs = <--some config stuff-->
enable = False
use_ssl = False
logging = /etc/st2/logging.auth.gunicorn.conf
debug = True
[rbac]
enable = True
backend = default I'm guessing even though enabled = False, its still trying to auth to ldap since backend is still set to ldap. Other than this issue I'm not even sure how to clear the cache to stop it from trying to auth with my old LDAP login. |
Thanks, @minsis. Because Docker and K8s might need proper implementation in the future, keeping in mind their multi-container specifics, I'd recommend testing LDAP/RBAC with the VM at this moment, following documentation: |
The st-docker#219 issue is just for getting a volume added to store the rbac role files. The current issue here is that something has broken with LADP and RBAC to where its locking out LDAP accounts because it tries to reauth every 5s (which I also put in an issue on that repo) on a cached credential. I'm just not sure where to go from here in terms of troubleshooting. Every time I turn on LDAP/RBAC it tries to auth my old credentials that are cached somewhere and I can't seem to clear them out. I would love to test directly on a VM, but at work I have limited resources and it takes our provisioning department weeks to get something built out. |
You can spin up a VM locally with st2vagrant and configure it that way. |
Since Saturday install on EL8 failing. Looks to be that we are picking up a new version of rabbitmq, which requires a new version of erlang, and that version of erlang is not available in CentOS or the epel repos. Also affecting old releases, unless pin rabbitmq version... |
I upgraded our dev box.
|
FWIW, I am on Centos 7 which now uses python3. I fired off a few large workflows and they all seem to be working as expected. |
Probably worth mentioning that if on a distro with python2.7 as the default that all sensor packs will need to be changed to python version 3 in the pack.yaml and reinstalled |
We're ready to prepare the StackStorm
v3.4
release and start pre-release testing..Release Process Preparation
Per Release Management Schedule @blag is the Release Manager and @amanda11 assisting for v3.4. They will freeze the
master
for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in#releasemgmt
and#development
Slack channels.The first step is pre-release manual user-acceptance testing for
v3.4dev
.Why Manual testing?
StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).
That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.
However it's not enough.
There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.
This is how StackStorm previously managed to land less major/critical bugs into production.
TL;DR
Install StackStorm
v3.4dev
unstable packages, try random things in random environments (different OS) and report any regressions found comparing tov3.3
(and/orv3.2
if you are still running ST2 Enterprise):curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable
You can also run StackStorm on a specific OS with st2vagrant:
Extra points for PR hotfixes and adding new or missing test cases.
Specific things to check
st2 --version
actions/lib
, please ensure that those actions still work. In the Python 2.7 removal, we adjusted how we handle this code, so we want to ensure that we do not have any regressions.actions/lib
issue, we haven't troubleshooted this enough to be sure).!help
(with st2chatops/hubot-stackstorm) or!st2help
(with err-stackstorm) commands work. See 500 on actionalias help endpoint with action_alias_help permission grant st2-rbac-backend#47 for more information.If you have successful test results, please post a summary of what all you tested (OSes, what features you tested).
If you run into any bugs, please open them in the respective repositories and link to this issue from there. I will add them to the list at the bottom of this description.
If you have any issues running StackStorm or running the tests, please post down below.
Major changes
timeout
parameter topacks.install
st2 action-alias execute
commandFull Changelog
Changes which are recommended to ack, explore, check and try in a random way.
st2
Added
Added support for GitLab SSH URLs on pack install and download actions. (improvement) #5050
Contributed by @asthLucas
Added st2-rbac-backend pip requirements for RBAC integration. (new feature) #5086
Contributed by @hnanchahal
Added notification support for err-stackstorm. (new feature) #5051
Added st2-auth-ldap pip requirements for LDAP auth integartion. (new feature) #5082
Contributed by @hnanchahal
Changed
Updated deprecation warning for python 2 pack installs, following python 2 support removal. #5099
Contributed by @amanda11
Improve the st2-self-check script to echo to stderr and exit if it isn't run with a
ST2_AUTH_TOKEN or ST2_API_KEY environment variable. (improvement) #5068
Added timeout parameter for packs.install action to help with long running installs that exceed the
default timeout of 600 sec which is defined by the python_script action runner (improvement) #5084
Contributed by @hnanchahal
Upgraded cryptography version to 3.2 to avoid CVE-2020-25659 (security) #5095
Converted most CI jobs from Travis to GitHub Actions (all except Integration tests).
Contributed by @nmaludy, @winem, and @blag
Updated cryptography dependency to version 3.3.2 to avoid CVE-2020-36242 (security) #5151
Fixed
Pin chardet version as newest version was incompatible with pinned requests version #5101
Contributed by @amanda11
Fixed issue were st2tests was not getting installed using pip because no version was specified.
Contributed by @anirudhbagri
Added monkey patch fix to st2stream to enable it to work with mongodb via SSL. (bug fix) #5078 #5091
Fix nginx buffering long polling stream to client. Instead of waiting for closed connection
wait for final event to be sent to client. (bug fix) #4842 #5042
Contributed by @guzzijones
StackStorm now explicitly decodes pack files as utf-8 instead of implicitly as ascii (bug fix)
#5106, #5107
Fix incorrect array parameter value casting when executing action via chatops or using
POST /aliasexecution/match_and_execute
API endpoint. The code would incorrectly assume thevalue is always a string, but that may not be the cast - they value could already be a list and
in this case we don't want any casting to be performed. (bug fix) #5141
Contributed by @Kami.
Fix
@parameter_name=/path/to/file/foo.json
notation in thest2 run
command which didn'twork correctly because it didn't convert read bytes to string / unicode type. (bug fix) #5140
Contributed by @Kami.
Fix broken
st2 action-alias execute
command and make sure it workscorrectly. (bug fix) #5138
Contributed by @Kami.
Removed
Removed --python3 pack install option #5100
Contributed by @amanda11
Removed submit-debug-info tool and the st2debug component #5103
Removed check-licence script (cleanup) #5092
Contributed by @kroustou
Updated Makefile and CI to use Python 3 only, removing Python 2 (cleanup) #5090
Contributed by @blag
Remove st2resultstracker from st2ctl, the development environment and the st2actions setup.py (cleanup) #5108
Contributed by @winem
orquesta 1.3.0
Added
Contributed by AJ (guzzijones)
Changed
deprecated in StackStorm and there's no intention to support it here. A mistral to orquesta
workflow conversion tool is available at https://github.com/StackStorm/orquestaconvert.
Fixed
st2chatops
No changes
StackStorm Exchange
Conclusion
Please report findings here and bugs/regressions in respective repositories.
Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.
Issues Found During Release
PRs Merged for Release
TODOs
The text was updated successfully, but these errors were encountered: