Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StackStorm v3.4.0 Pre-release Testing #66

Closed
13 of 30 tasks
blag opened this issue Feb 19, 2021 · 49 comments
Closed
13 of 30 tasks

StackStorm v3.4.0 Pre-release Testing #66

blag opened this issue Feb 19, 2021 · 49 comments

Comments

@blag
Copy link

blag commented Feb 19, 2021

We're ready to prepare the StackStorm v3.4 release and start pre-release testing..

Release Process Preparation

Per Release Management Schedule @blag is the Release Manager and @amanda11 assisting for v3.4. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels.
The first step is pre-release manual user-acceptance testing for v3.4dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough.
There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.4dev unstable packages, try random things in random environments (different OS) and report any regressions found comparing to v3.3 (and/or v3.2 if you are still running ST2 Enterprise):

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

You can also run StackStorm on a specific OS with st2vagrant:

git clone https://github.com/StackStorm/st2vagrant.git
cd st2vagrant
VERSION=3.4dev vagrant up  # Default OS is Ubuntu Xenial (16.04)

BOX=ubuntu/bionic64 VERSION=3.4dev vagrant up  # Ubuntu Bionic 18.04

BOX=centos/7 VERSION=3.4dev vagrant up  # CentOS 7

BOX=centos/8 VERSION=3.4dev vagrant up  # CentOS 8

Extra points for PR hotfixes and adding new or missing test cases.

Specific things to check

  • Run the self-verification script - see instructions here
    • Ensure that the st2resultstracker service doesn't exist and NOT running
    • Ensure that StackStorm is running on Python 3 - st2 --version
  • Test the web UI and the integrated flow UI - see instructions here
  • Test Ubuntu16.04 or CentOS7/RHEL7 installation and upgrade as they've moved from py2 to py3
  • If you have a pack that utilizes Python modules from actions/lib, please ensure that those actions still work. In the Python 2.7 removal, we adjusted how we handle this code, so we want to ensure that we do not have any regressions.
  • If you have access to a Windows server, please test out a few actions from the Windows pack. We have not been testing this in our end-to-end tests because the Windows instances were not responding to WinRM queries, so we want to ensure no regressions (this might be related to the actions/lib issue, we haven't troubleshooted this enough to be sure).
  • If you are using either the LDAP authentication or the RBAC authorization backends, please enable those backends and ensure that the integration continues to work.
  • StackStorm upgrade from 3.3 to 3.4, following the Upgrade Instructions: https://docs.stackstorm.com/latest/install/upgrades.html#v3-4

If you have successful test results, please post a summary of what all you tested (OSes, what features you tested).

If you run into any bugs, please open them in the respective repositories and link to this issue from there. I will add them to the list at the bottom of this description.

If you have any issues running StackStorm or running the tests, please post down below.

Major changes

  • Removed Python 2.7 support for all OSes
    • CentOS7/RHEL7 migrated from py2 to py3
    • Ubuntu 16.04 LTS migrated from py2 to py3
  • Integrated ST2 RBAC backend
  • Integrated LDAP authentication backend
  • Integrated Workflow Designer (st2flow) into Workflow Composer (st2web)
  • Added notification support for err-stackstorm
  • Updated deprecation warning for Python 2 packs
  • Added a timeout parameter to packs.install
  • Updated dependencies
  • Pack files are now interpreted as UTF-8 instead of ASCII
  • Fixed broken st2 action-alias execute command
  • Removed st2resultstracker

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

  • Added support for GitLab SSH URLs on pack install and download actions. (improvement) #5050
    Contributed by @asthLucas

  • Added st2-rbac-backend pip requirements for RBAC integration. (new feature) #5086
    Contributed by @hnanchahal

  • Added notification support for err-stackstorm. (new feature) #5051

  • Added st2-auth-ldap pip requirements for LDAP auth integartion. (new feature) #5082
    Contributed by @hnanchahal

Changed

  • Updated deprecation warning for python 2 pack installs, following python 2 support removal. #5099
    Contributed by @amanda11

  • Improve the st2-self-check script to echo to stderr and exit if it isn't run with a
    ST2_AUTH_TOKEN or ST2_API_KEY environment variable. (improvement) #5068

  • Added timeout parameter for packs.install action to help with long running installs that exceed the
    default timeout of 600 sec which is defined by the python_script action runner (improvement) #5084

    Contributed by @hnanchahal

  • Upgraded cryptography version to 3.2 to avoid CVE-2020-25659 (security) #5095

  • Converted most CI jobs from Travis to GitHub Actions (all except Integration tests).

    Contributed by @nmaludy, @winem, and @blag

  • Updated cryptography dependency to version 3.3.2 to avoid CVE-2020-36242 (security) #5151

Fixed

  • Pin chardet version as newest version was incompatible with pinned requests version #5101
    Contributed by @amanda11

  • Fixed issue were st2tests was not getting installed using pip because no version was specified.
    Contributed by @anirudhbagri

  • Added monkey patch fix to st2stream to enable it to work with mongodb via SSL. (bug fix) #5078 #5091

  • Fix nginx buffering long polling stream to client. Instead of waiting for closed connection
    wait for final event to be sent to client. (bug fix) #4842 #5042

    Contributed by @guzzijones

  • StackStorm now explicitly decodes pack files as utf-8 instead of implicitly as ascii (bug fix)
    #5106, #5107

  • Fix incorrect array parameter value casting when executing action via chatops or using
    POST /aliasexecution/match_and_execute API endpoint. The code would incorrectly assume the
    value is always a string, but that may not be the cast - they value could already be a list and
    in this case we don't want any casting to be performed. (bug fix) #5141

    Contributed by @Kami.

  • Fix @parameter_name=/path/to/file/foo.json notation in the st2 run command which didn't
    work correctly because it didn't convert read bytes to string / unicode type. (bug fix) #5140

    Contributed by @Kami.

  • Fix broken st2 action-alias execute command and make sure it works
    correctly. (bug fix) #5138

    Contributed by @Kami.

Removed

  • Removed --python3 pack install option #5100
    Contributed by @amanda11

  • Removed submit-debug-info tool and the st2debug component #5103

  • Removed check-licence script (cleanup) #5092

    Contributed by @kroustou

  • Updated Makefile and CI to use Python 3 only, removing Python 2 (cleanup) #5090

    Contributed by @blag

  • Remove st2resultstracker from st2ctl, the development environment and the st2actions setup.py (cleanup) #5108

    Contributed by @winem

orquesta 1.3.0

Added

  • Implement the orquesta-rehearse command to run unit tests for workflow. (new feature, beta)
    Contributed by AJ (guzzijones)

Changed

  • Remove mistral fixtures because they were used as ref to support multiple specs. Mistral is
    deprecated in StackStorm and there's no intention to support it here. A mistral to orquesta
    workflow conversion tool is available at https://github.com/StackStorm/orquestaconvert.

Fixed

  • Fix the custom flake8 extensions to check import statements. (bug fix)
  • Fix the deprecation warning that flags are not at the start of the expression. (bug fix)

st2chatops

No changes

StackStorm Exchange

  • Removed Python 2.7 support and testing from almost all packs
  • Added environment variable flag to avoid publishing test packs to exchange.stackstorm.com

Conclusion

Please report findings here and bugs/regressions in respective repositories.
Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.

Issues Found During Release

PRs Merged for Release

TODOs

  • Blog post for release
  • Blog post for exchange/community update
@himadriganguly
Copy link

st2ctl status showing

##### st2 components status #####
st2actionrunner PID: 18733
st2actionrunner PID: 18737
st2actionrunner PID: 18740
st2actionrunner PID: 18743
st2actionrunner PID: 18746
st2actionrunner PID: 18749
st2actionrunner PID: 18753
st2actionrunner PID: 18756
st2actionrunner PID: 18762
st2actionrunner PID: 18765
st2api PID: 18635
st2api PID: 18647
st2stream PID: 18583
st2stream PID: 18628
st2auth PID: 18523
st2auth PID: 18535
st2garbagecollector PID: 17552
st2notifier PID: 17555
st2rulesengine PID: 17558
st2sensorcontainer PID: 18653
st2chatops is not running.
st2timersengine PID: 17569
st2workflowengine PID: 18665
st2scheduler PID: 17575

in the result above it is showing st2chatops not running.

sudo service st2chatops status

● st2chatops.service - StackStorm service st2chatops
   Loaded: loaded (/lib/systemd/system/st2chatops.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sat 2021-02-20 16:04:35 UTC; 3s ago
  Process: 22638 ExecStart=/opt/stackstorm/chatops/bin/hubot $DAEMON_ARGS (code=exited, status=1/FAILURE)
 Main PID: 22638 (code=exited, status=1/FAILURE)
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Main process exited, code=exited, status=1/FAILURE
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Unit entered failed state.
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Failed with result 'exit-code'.

Even after the machine restart, the same thing happens.

@blag
Copy link
Author

blag commented Feb 20, 2021

@himadriganguly Thank you for testing v3.4a!

That's normal when you haven't configured st2chatops by modifying /opt/stackstorm/chatops/st2chatops.env. The default StackStorm installation will not configure ChatOps for you since that is an optional component that requires integration with an external service.

We should try to improve that situation in the future, but for v3.4 I'm not considering that a bug.

If you have configured st2chatops, then that's definitely a bug. In which case, please post your st2chatops.env file (but remember to redact any API keys or Slack tokens) so we can figure out if it's a bug in your configuration or in st2chatops.

@himadriganguly
Copy link

Running Self-Verification Result

Click to expand!
$ sudo ST2_AUTH_TOKEN=$(st2 auth st2admin -p 'Ch@ngeMe' -t) /opt/stackstorm/st2/bin/st2-self-check
~ ~
Cloning st2tests (branch=master) in /home/vagrant.
Cloning into 'st2tests'...
remote: Enumerating objects: 323, done.
remote: Counting objects: 100% (323/323), done.
remote: Compressing objects: 100% (184/184), done.
remote: Total 323 (delta 145), reused 251 (delta 121), pack-reused 0
Receiving objects: 100% (323/323), 703.27 KiB | 396.00 KiB/s, done.
Resolving deltas: 100% (145/145), done.
Checking connectivity... done.
Copying asserts, fixtures, tests and examples packs.
Installing asserts, fixtures, tests and examples packs.
..................
id: 60313654f6df70916a47edb3
action.ref: packs.setup_virtualenv
context.user: st2admin
parameters: 
  packs:
  - examples
  - tests
  - asserts
  - fixtures
  - webui
status: succeeded
start_timestamp: Sat, 20 Feb 2021 16:18:28 UTC
end_timestamp: Sat, 20 Feb 2021 16:19:05 UTC
result: 
  exit_code: 0
  result: 'Successfully set up virtualenv for the following packs: examples, tests, asserts, fixtures, webui'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "examples" (/opt/stackstorm/packs/examples)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/examples" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "examples" in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/examples" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --no-download /opt/stackstorm/virtualenvs/examples" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/examples/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/examples/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/examples/requirements.txt with command /opt/stackstorm/virtualenvs/examples/bin/pip install -U -r /opt/stackstorm/packs/examples/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "examples" successfully created in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "tests" (/opt/stackstorm/packs/tests)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/tests" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "tests" in "/opt/stackstorm/virtualenvs/tests"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/tests" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --no-download /opt/stackstorm/virtualenvs/tests" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/tests/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    No pack specific requirements found
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "tests" successfully created in "/opt/stackstorm/virtualenvs/tests"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "asserts" (/opt/stackstorm/packs/asserts)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/asserts" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "asserts" in "/opt/stackstorm/virtualenvs/asserts"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/asserts" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/asserts" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/asserts/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/asserts/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/asserts/requirements.txt with command /opt/stackstorm/virtualenvs/asserts/bin/pip install -U -r /opt/stackstorm/packs/asserts/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "asserts" successfully created in "/opt/stackstorm/virtualenvs/asserts"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "fixtures" (/opt/stackstorm/packs/fixtures)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/fixtures" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "fixtures" in "/opt/stackstorm/virtualenvs/fixtures"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/fixtures" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/fixtures" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/fixtures/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/fixtures/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/fixtures/requirements.txt with command /opt/stackstorm/virtualenvs/fixtures/bin/pip install -U -r /opt/stackstorm/packs/fixtures/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "fixtures" successfully created in "/opt/stackstorm/virtualenvs/fixtures"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "webui" (/opt/stackstorm/packs/webui)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/webui" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "webui" in "/opt/stackstorm/virtualenvs/webui"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/webui" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/webui" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/webui/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/webui/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/webui/requirements.txt with command /opt/stackstorm/virtualenvs/webui/bin/pip install -U -r /opt/stackstorm/packs/webui/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "webui" successfully created in "/opt/stackstorm/virtualenvs/webui"
    '
  stdout: ''
Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
2021-02-20 16:19:06,282 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2021-02-20 16:19:06,289 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2021-02-20 16:19:06,737 INFO [-] =========================================================
2021-02-20 16:19:06,738 INFO [-] ############## Registering triggers #####################
2021-02-20 16:19:06,738 INFO [-] =========================================================
2021-02-20 16:19:06,984 INFO [-] Registered 1 triggers.
2021-02-20 16:19:06,984 INFO [-] =========================================================
2021-02-20 16:19:06,984 INFO [-] ############## Registering sensors ######################
2021-02-20 16:19:06,984 INFO [-] =========================================================
2021-02-20 16:19:07,182 INFO [-] Registered 7 sensors.
2021-02-20 16:19:07,182 INFO [-] =========================================================
2021-02-20 16:19:07,183 INFO [-] ############## Registering runners ######################
2021-02-20 16:19:07,183 INFO [-] =========================================================
2021-02-20 16:19:07,977 INFO [-] Registered 14 runners.
2021-02-20 16:19:07,977 INFO [-] =========================================================
2021-02-20 16:19:07,977 INFO [-] ############## Registering actions ######################
2021-02-20 16:19:07,978 INFO [-] =========================================================
2021-02-20 16:19:15,491 INFO [-] Registered 230 actions.
2021-02-20 16:19:15,491 INFO [-] =========================================================
2021-02-20 16:19:15,491 INFO [-] ############## Registering rules ########################
2021-02-20 16:19:15,492 INFO [-] =========================================================
2021-02-20 16:19:15,767 INFO [-] Registered 9 rules.
2021-02-20 16:19:15,767 INFO [-] =========================================================
2021-02-20 16:19:15,767 INFO [-] ############## Registering aliases ######################
2021-02-20 16:19:15,767 INFO [-] =========================================================
2021-02-20 16:19:17,329 INFO [-] Registered 134 aliases.
2021-02-20 16:19:17,329 INFO [-] =========================================================
2021-02-20 16:19:17,329 INFO [-] ############## Registering policy types #################
2021-02-20 16:19:17,329 INFO [-] =========================================================
2021-02-20 16:19:17,364 INFO [-] Registered 3 policy types.
2021-02-20 16:19:17,364 INFO [-] =========================================================
2021-02-20 16:19:17,364 INFO [-] ############## Registering policies #####################
2021-02-20 16:19:17,364 INFO [-] =========================================================
2021-02-20 16:19:17,365 INFO [-] Registered 0 policies.
2021-02-20 16:19:17,365 INFO [-] =========================================================
2021-02-20 16:19:17,365 INFO [-] ############## Registering configs ######################
2021-02-20 16:19:17,366 INFO [-] =========================================================
2021-02-20 16:19:17,368 INFO [-] Registered 0 configs.
##### st2 components status #####
st2actionrunner PID: 1330
st2actionrunner PID: 1356
st2actionrunner PID: 1366
st2actionrunner PID: 1392
st2actionrunner PID: 1415
st2actionrunner PID: 1421
st2actionrunner PID: 1427
st2actionrunner PID: 1470
st2actionrunner PID: 1497
st2actionrunner PID: 1539
st2api PID: 1166
st2api PID: 1772
st2stream PID: 1226
st2stream PID: 1764
st2auth PID: 1234
st2auth PID: 1733
st2garbagecollector PID: 1169
st2notifier PID: 1205
st2rulesengine PID: 1220
st2sensorcontainer PID: 1179
st2chatops is not running.
st2timersengine PID: 1155
st2workflowengine PID: 1210
st2scheduler PID: 1185
Deleting 'st2tests' directory from /home/vagrant.
~
Attempting Test tests.test_inquiry_chain...OK! (55s)
Attempting Test tests.test_key_triggers...OK! (44s)
Attempting Test tests.test_pack_install_tool...OK! (17s)
Attempting Test tests.test_packs_pack...OK! (130s)
Attempting Test tests.test_quickstart...OK! (27s)
Attempting Test tests.test_quickstart_key...OK! (7s)
Attempting Test tests.test_quickstart_local_script_actions...OK! (13s)
Attempting Test tests.test_quickstart_passive_sensor...OK! (7s)
Attempting Test tests.test_quickstart_polling_sensor...OK! (5s)
Attempting Test tests.test_quickstart_python_actions...OK! (13s)
Attempting Test tests.test_quickstart_remote_script_actions...OK! (17s)
Attempting Test tests.test_quickstart_rules...OK! (64s)
Attempting Test tests.test_quickstart_trace...OK! (21s)
Attempting Test tests.test_render_config_context...OK! (5s)
Attempting Test tests.test_run_pack_tests_tool...OK! (82s)
Attempting Test tests.test_timer_rule...OK! (41s)
Skipping tests.test_windows_runners...
Skipping tests.test_winrm_large_script...
Skipping tests.test_winrm_runners...
Attempting Example examples.orquesta-examples...OK! (9s)
SELF CHECK SUCCEEDED!
st2-self-check succeeded.

#############################################################
###################################################   #######
###############################################   /~\   #####
############################################   _- `~~~', ####
##########################################  _-~       )  ####
#######################################  _-~          |  ####
####################################  _-~            ;  #####
##########################  __---___-~              |   #####
#######################   _~   ,,                  ;  `,,  ##
#####################  _-~    ;'                  |  ,'  ; ##
###################  _~      '                    `~'   ; ###
############   __---;                                 ,' ####
########   __~~  ___                                ,' ######
#####  _-~~   -~~ _                               ,' ########
##### `-_         _                              ; ##########
#######  ~~----~~~   ;                          ; ###########
#########  /          ;                        ; ############
#######  /             ;                      ; #############
#####  /                `                    ; ##############
###  /                                      ; ###############
#                                            ################

@himadriganguly
Copy link

himadriganguly commented Feb 20, 2021

@blag Thank you for the explanation. I will be testing other things according to the checklist and will post the details. I will try to contribute as much as I can.

@winem
Copy link

winem commented Feb 20, 2021

Just a short update since I ran into an issue with the load_diag workflow in the Linux pack and I'll look into that now:

  • Installation on CentOS7 works as expected (WebUI works, st2ctl commands work, running different actions of different packs works)

Update: The issue with the load_diag workflow isn't a regression or issue specific to st2 3.4 and there are already open issues that cover these issues. So I'll proceed.

@Kami
Copy link
Member

Kami commented Feb 20, 2021

I run tests on Ubuntu Xenial (16.04) here are the issues I found:

  1. st2 run core.local cmd='python --version' returns Python 2

This may or may not be expected and desired, depending on how we envision whole Python 3 story. It returns python 2 since system default python binary in PATH is used instead of python binary from /opt/stackstorm/st2/bin.

It does appear to be working fine for python runner actions though (which is likely the only thing we care about) - st2 run examples.python_runner_print_python_version, /opt/stackstorm/st2/bin/python --version.

vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_version
.
id: 60319401b8c38397f0e40066
action.ref: examples.python_runner_print_python_version
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
end_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26) 
[GCC 5.4.0 20160609]
"

vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_environment
.
id: 60319560b8c38397f0e40069
action.ref: examples.python_runner_print_python_environment
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
end_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26) 
[GCC 5.4.0 20160609]
Platform: Linux-4.4.0-161-generic-x86_64-with-Ubuntu-16.04-xenial
PYTHONPATH: /opt/stackstorm/virtualenvs/examples/lib/python3.6:/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages:/opt/stackstorm/packs/examples/actions/lib::/opt/stackstorm/st2/lib/python3.6/site-packages
sys.path: ['/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner', '/opt/stackstorm/virtualenvs/examples/lib/python3.6', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/packs/examples/actions/lib', '/', '/opt/stackstorm/st2/lib/python3.6/site-packages', '/opt/stackstorm/virtualenvs/examples/lib/python36.zip', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/opt/stackstorm/packs/examples/actions/pythonactions']

As far as core.local "issue" goes, we need to decide what's the desired and expected behavior. I believe we should not add /opt/stackstorm/st2/bin/ to PATH for core.local actions - core local actions should observe same PATH rules as any other script which is manually ran from a shell session on that server.

In short, we only care about python version being used for Python runner actions and StackStorm services.

If that's the case, testing instructions need to be updated.

  1. bash completion for st2 CLI doesn't seem to work out of the box

It could be that this has been broken for a long time already, but I do know it worked at some point in the distant past (perhaps we just don't source source the bash completion script by default, didn't dig in).

@arm4b
Copy link
Member

arm4b commented Feb 21, 2021

@Kami I agree about 1) /opt/stackstorm/st2/bin/python is something to verify for the new py env.
I've updated the instructions so the expectation is to make sure that st2 --version is showing py3.

@winem
Copy link

winem commented Feb 21, 2021

I'm done with the tests of fresh st2 installations on CentOS7 and Ubuntu 16.04 now as well. I created an issue + PR for a small issue with the install script on Ubuntu 16.04 and can also confirm topics reported by @Kami

This means:

  • st2 installation
  • use the web & Flow UI, st2 cli
  • install & removing packs
  • run actions & workflows
  • run the self-verification

I also did a cross-check with st2 v3.3 and the st2 auto-completion does not work, too. So this is not a new issue.

I'll proceed with tests upgrading st2 v3.3 to 3.4 on both OS.

@amanda11
Copy link

amanda11 commented Feb 22, 2021

Started testing on CentOS 8.
Found issue that the ansible install docs page hasn't been updated given the EWC donated features - PR in progress
Also should get StackStorm/st2docs#1022 reviewed and merged, as the matching code is merged but not the doc PR.

@arm4b
Copy link
Member

arm4b commented Feb 22, 2021

@Kami @winem Good find about the auto-completion, fresh perspective always helps 👍
Would be nice to document the bug somewhere so we could fix it in the next release.

@amanda11
Copy link

StackStorm/st2docs#1055 - doc change about EWC components

@amanda11
Copy link

amanda11 commented Feb 22, 2021

Simulated the E2E tests setup - and found that the ansible-script in the user-data didn't seem to get applied to the instance, hence there was no winrm listener on 5986.
Manually setup a windows machine, against a CentOS 8 3.4 dev installed via ansible.
Successfully ran the following tests from the st2tests/tests pack

  • test_winrm_large_script
  • test_winrm_runners
    Also on a bash installed CentOS 8 3.4dev, tested that the windows.wait_for_winrm action worked, after running the ansible winrm script manually.

@amanda11
Copy link

Quick play with RBAC and ChatOps (hubot) on CentOS 8 ansible install, all good. !help still worked with rbac enabled.

Ansible install on CentOS 7 - went through UI testing list on Chrome. Couldn't find a "Contact Us" - think its replaced by "Support" - so might want to update https://github.com/StackStorm/discussions/wiki/Release-Process#manual-testing-for-web-and-flow-ui

Followed manual instructions for CentOS 7, all good, and setup ChatOps with manual instructions successfully.

@amanda11
Copy link

amanda11 commented Feb 23, 2021

Possible upgrade or pythonpath issue.

Upgraded StackStorm 3.3 on CentOS 7-> 3.4.

Got error on aws pack after upgrade:

  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/netaddr/compat.py\", line 93, in <module>
    import importlib_resources as _importlib_resources
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/importlib_resources/__init__.py\", line 3, in <module>
    from ._common import (
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/importlib_resources/_common.py\", line 2, in <module>
    import pathlib
ImportError: No module named pathlib
",
  "exit_code": 1,
  "result": "None"
}

Problem is that the virtualenv of the pack from when it was installed on ST2 3.3 has python2.7 files in the lib area.
If I remove and then install the pack after upgrade it works fine.

Updated PR StackStorm/st2docs#1056 to include instructions that need to uninstall/re-install packs after upgrade on U16 and EL7.

@winem
Copy link

winem commented Feb 23, 2021

Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11

I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa:

20210223T193442+0000 The repository is setup! You can now install packages.
20210223T193442+0000 Reading package lists...
20210223T193442+0000 Building dependency tree...
20210223T193442+0000 Reading state information...
20210223T193442+0000 Some packages could not be installed. This may mean that you have
20210223T193442+0000 requested an impossible situation or if you are using the unstable
20210223T193442+0000 distribution that some required packages have not yet been created
20210223T193442+0000 or been moved out of Incoming.
20210223T193442+0000 The following information may help to resolve the situation:
20210223T193442+0000 
20210223T193442+0000 The following packages have unmet dependencies:
20210223T193442+0000  st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000        Depends: python3.6-dev but it is not installable
20210223T193442+0000 E: Unable to correct problems, you have held broken packages.
20210223T193442+0000 ############### ERROR ###############
20210223T193442+0000 # Failed on Install st2 #
20210223T193442+0000 #####################################

And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:

20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY    [js] Error: couldn't add user: command createUser requires authentication :
20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13
20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15
20210223T193946+0000 @(shell):1:1
20210223T193946+0000 bye
20210223T193946+0000 ############### ERROR ###############
20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) #
20210223T193946+0000 #####################################

Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.

@arm4b
Copy link
Member

arm4b commented Feb 23, 2021

@winem StackStorm installer doesn't support curl-bash script re-run and idempotence, eg. resuming installation at this moment. Something that we could improve in the future for sure!

20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000 Depends: python3.6-dev but it is not installable

I think what happened there is failed apt-get update which happens sometimes due to transient network/upstream issues.

@winem
Copy link

winem commented Feb 23, 2021

@armab yes, it's fine that idempotence is not supported. I just forgot that when I ran it again. :)

Anyway, the deadsnakes sources file in /etc/apt/sources.list.d/ was not created. I'll re-run the whole upgrade path and create a snapshot of running st2 3.3 on 16.04 before I try the update next time. This should speed up my troubleshooting (if it's reproducable and an actual issue).
I'll share the results latest tomorrow.

@minsis
Copy link

minsis commented Feb 23, 2021

Few things tested. As a note I'm using the docker version. Using Chrome Browser.

  • LDAP works with a simple setup non SSL connection to MS AD.
  • Workflow visualizer doesn't seem to pick up on previously created workflows. Meaning I had workflows created and registered via the CLI before the upgrade, and going to the workflow visualizer doesn't produce anything but a blank canvas with no actions. After some more poking around I'm getting a 404 when the visualizer opens my old workflow pack and can't find any of the files. I'm not exactly sure why, but the error in the API container is st2common.exceptions.db.StackStormDBObjectNotFoundError: Resource with a ref or id "webui" not found
  • I was unable to test the examples that come prepackaged as there's nothing there (see below)?
  • I'm unable to get a positive result from the test script. I'm guessing this is looking locally inside the container itself for the services
  • In the workflow visualizer the "name" field is required, but not marked as such so saving fails the first time around.
    image

Some random feedback:

  • The save button in the visualizer is easy to miss as it looks like its more of a navigation button. Maybe put a big button that says "Save"; same goes for the run button
  • I know workflows have to have an action behind it, but it seems like it would be cleaner to maybe have the visualizer in its own tab?
  • When hovering over the anchor point on a task object maybe have the icon change to a move icon.
  • When closing the visualizer after making a change there should be a warning telling you that you haven't saved anything.

examples pack missing in container?

root@42118eabf091:/opt/stackstorm# ll -d packs/examples2
lrwxrwxrwx 1 root root 27 Feb 23 21:30 packs/examples2 -> /usr/share/doc/st2/examples/
root@42118eabf091:/opt/stackstorm# ll packs/examples2/*
packs/examples2/actions:
total 40
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x  9 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 bash_exit_code/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 bash_ping/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 bash_random/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 chains/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 pythonactions/
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 ubuntu_pkg_info/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 windows/
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 workflows/

packs/examples2/aliases:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/lib:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/rules:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/sensors:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/tests:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/triggers:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
root@42118eabf091:/opt/stackstorm# ll packs/examples2/actions/*
packs/examples2/actions/bash_exit_code:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/bash_ping:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/bash_random:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/chains:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/pythonactions:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/ubuntu_pkg_info:
total 12
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 lib/

packs/examples2/actions/windows:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/workflows:
total 12
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 tests/
root@42118eabf091:/opt/stackstorm# ll packs/examples2/actions/*/*
packs/examples2/actions/ubuntu_pkg_info/lib:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/workflows/tests:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ../

@amanda11
Copy link

Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11

I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa:

20210223T193442+0000 The repository is setup! You can now install packages.
20210223T193442+0000 Reading package lists...
20210223T193442+0000 Building dependency tree...
20210223T193442+0000 Reading state information...
20210223T193442+0000 Some packages could not be installed. This may mean that you have
20210223T193442+0000 requested an impossible situation or if you are using the unstable
20210223T193442+0000 distribution that some required packages have not yet been created
20210223T193442+0000 or been moved out of Incoming.
20210223T193442+0000 The following information may help to resolve the situation:
20210223T193442+0000 
20210223T193442+0000 The following packages have unmet dependencies:
20210223T193442+0000  st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000        Depends: python3.6-dev but it is not installable
20210223T193442+0000 E: Unable to correct problems, you have held broken packages.
20210223T193442+0000 ############### ERROR ###############
20210223T193442+0000 # Failed on Install st2 #
20210223T193442+0000 #####################################

And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:

20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY    [js] Error: couldn't add user: command createUser requires authentication :
20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13
20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15
20210223T193946+0000 @(shell):1:1
20210223T193946+0000 bye
20210223T193946+0000 ############### ERROR ###############
20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) #
20210223T193946+0000 #####################################

Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.

I have a fix @winem for this - StackStorm/st2-packages#691

I was looking into it as the E2E tests started failing...

@hnanchahal
Copy link

hnanchahal commented Feb 24, 2021 via email

@cognifloyd
Copy link
Member

cognifloyd commented Feb 24, 2021

Using Ubuntu 18.04 for the K8s cluster. Kubernetes and Ansible pack fail to
install with the error: “error: command 'x86_64-linux-gnu-gcc' failed: No
such file or directory”

pip reaches for gcc only when it can't resolve a wheel to download so it needs to build a wheel for a package that is not pure-python. Test infra uses a newer version of pip than what we install in production. So we're unlikely to find such errors while testing (before we cut a release) because pip is new enough in testing.

Please, let's merge these PRs to sanitize/make-consistent the pip version we use everywhere.
StackStorm/st2packaging-dockerfiles#103
StackStorm/st2-packages#687
StackStorm-Exchange/ci#102
StackStorm/st2#5165

@cognifloyd
Copy link
Member

@cognifloyd It has nothing to do with the pip version and all those PRs are not bugfixes, but maintenance tasks that are not blockers/requirements for this release.

I disagree. If we explicitly pin one version of a package, but we test with a different version, then that is a bug.

@cognifloyd
Copy link
Member

cognifloyd commented Feb 24, 2021

As far as the ansible pack is concerned, that note in the readme is somewhat incorrect. gcc is only needed because the version of pip we're using in production is broken.

Ansible is a pure-python package. It depends on cryptography and PyYAML which have non-python components. Both cryptography and PyYAML provide wheels, but the old versions of pip can't use them so it tries to build its own wheel instead. If we used the same version of pip in production that we have been running unit/integration tests with, then this would be a non-issue.

@amanda11
Copy link

Tested manual and ansible deployments on U16. All good. Not had chance to confirm upgrade on U16. Quick play in those environments with workflow designer, and the st2-self-check all ran successfully.

@amanda11
Copy link

Some random feedback:

  • The save button in the visualizer is easy to miss as it looks like its more of a navigation button. Maybe put a big button that says "Save"; same goes for the run button
  • I know workflows have to have an action behind it, but it seems like it would be cleaner to maybe have the visualizer in its own tab?
  • When hovering over the anchor point on a task object maybe have the icon change to a move icon.
  • When closing the visualizer after making a change there should be a warning telling you that you haven't saved anything.

@minsis This is great feedback, would you mind (if you haven't already) create a feature request on the st2web repository for these. They wouldn't be regression problems, but good points to improve the UI in future releases.

@winem
Copy link

winem commented Feb 27, 2021

I'm running into another issue with the upgrade from 3.3 to 3.4 on U16 now. It looks like an issue with stdin redirection again. It appears if the st2.conf was modified and does not match the one provided by st2:

20210227T212722+0000 Unpacking st2 (3.4dev-58) over (3.3.0-4) ...
20210227T212730+0000 Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
20210227T212730+0000 Setting up libpython3.6:amd64 (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up libpython3.6-dev:amd64 (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up python3.6-dev (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up st2 (3.4dev-58) ...
20210227T212730+0000 Installing new version of config file /etc/logrotate.d/st2 ...
20210227T212730+0000 
20210227T212730+0000 Configuration file '/etc/st2/st2.conf'
20210227T212730+0000  ==> Modified (by you or by a script) since installation.
20210227T212730+0000  ==> Package distributor has shipped an updated version.
20210227T212730+0000    What would you like to do about it ?  Your options are:
20210227T212730+0000     Y or I  : install the package maintainer's version
20210227T212730+0000     N or O  : keep your currently-installed version
20210227T212730+0000       D     : show the differences between the versions
20210227T212730+0000       Z     : start a shell to examine the situation
20210227T212730+0000  The default action is to keep your current version.
20210227T212730+0000 *** st2.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package st2 (--configure):
20210227T212730+0000  end of file on stdin at conffile prompt
20210227T212730+0000 Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
20210227T212730+0000 Errors were encountered while processing:
20210227T212730+0000  st2
20210227T212731+0000 E: Sub-process /usr/bin/dpkg returned an error code (1)
20210227T212731+0000 ############### ERROR ###############
20210227T212731+0000 # Failed on Install st2 #
20210227T212731+0000 #####################################

So it does not wait for any user input and fails instead immediately. This seems to be the same as: StackStorm/st2-packages#688

The upgrade seems to work fine when I revert all changes on st2.conf and run the curl-bash-command. At least the self-verification was successful. Still have to do some more tests.

@winem
Copy link

winem commented Feb 27, 2021

It looks like we can avoid the stdin issue as described above and in StackStorm/st2-packages#688 by using the following command:

bash <(curl -sSL https://stackstorm.com/packages/install.sh) --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa

instead of the original one:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

The one I suggested does not have any issues with interactions and does wait for the users input. I tested all available options Y/I, N/O, D and Z successfully.

I do not see any additional issue or (security) risk by changing the commands but am happy about additional views on this suggestion.

@arm4b
Copy link
Member

arm4b commented Feb 28, 2021

That enhancement would be helpful (please submit a PR), but we don't support upgrading st2 versions via curl|bash installer as it doesn't know about the migration path, nor idempotent for that operation.
The user needs to follow the https://docs.stackstorm.com/latest/install/upgrades.html#general-upgrade-procedure instead.

@winem
Copy link

winem commented Feb 28, 2021

PR provided. And yes, it's just for testing/development purpose. 👍

@winem
Copy link

winem commented Feb 28, 2021

I'm done with the tests on different installations. Let me just summarize the tests here for a better overview:

Testing means:

  • st2 installation
  • use the web & Flow UI, st2 cli
  • install & removing packs
  • run actions & workflows
  • run the self-verification
  • restart, reload & register components + common (maintenance) tasks via st2ctl
  • ensure service states incl. the resultstracker & used Python versions

Tested Versions:

  • Ubuntu 16.04
    • manual 3.4dev installation
    • installation via ansible-st2
    • manual upgrade from 3.3 to 3.4dev
    • upgrade via ansible-st2 from 3.3 to 3.4dev
  • Ubuntu 18.04
    • manual 3.4dev installation
    • installation via ansible-st2
  • CentOS 7
    • manual installation of 3.4dev
    • manual upgrade from 3.3 to 3.4dev

I just found some (minor?) issues:
1. After the manual upgrade on U16 st2resultstracker was still running. The running service was not stopped by the script. The service was disabled and did not start automatically again after a reboot. st2resultstracker was stopped during and no longer running after the upgrade via ansible-st2, so this is just an issue with the bash script.
2. The FlowUI works fine for viewing existing Orquesta workflows but you can't save any changes due to a permission issue. Example: "Error saving workflow.: Unable to write data to "/opt/stackstorm/packs/packs/actions/workflows/install.yaml" (permission denied). Make sure permissions for that pack directory are configured correctly so st2api can write to it." Creating new workflows and updating them works fine. This issue is not limited to Ubuntu 16.04 and was also reproducable on 18.04.

So, is 1) an actual issue? Do we expect the same issue when st2 3.4 will be installed via package managers like apt, yum, etc? Do we want to add a note to the migration docs?
Is 2) known & expected and I just missed it in the documentation? I couldn't find a corresponding issue. I'll create one tomorrow unless we figure this out earlier than that here.

@amanda11
Copy link

amanda11 commented Mar 1, 2021

  1. After the manual upgrade on U16 st2resultstracker was still running. The running service was not stopped by the script. The service was disabled and did not start automatically again after a reboot. st2resultstracker was stopped during and no longer running after the upgrade via ansible-st2, so this is just an issue with the bash script.

@winem - Did you follow the manual upgrade instructions? On CentOS 7 when following the manual upgrade instructions there is no problem. As the first instruction is to stop all ST2 services, therefore st2resultstracker is stopped at that point, and then never restarted when the ST2 services is restarted.
If you used the bash installer for upgrade, then that isn't the documented upgrade procedure - so I think that isn't relevant.

@amanda11
Copy link

amanda11 commented Mar 1, 2021

2. The FlowUI works fine for viewing existing Orquesta workflows but you can't save any changes due to a permission issue. Example: "Error saving workflow.: Unable to write data to "/opt/stackstorm/packs/packs/actions/workflows/install.yaml" (permission denied). Make sure permissions for that pack directory are configured correctly so st2api can write to it." Creating new workflows and updating them works fine. This issue is not limited to Ubuntu 16.04 and was also reproducable on 18.04.

I've just installed a pack on a CentOS 7 upgraded to 3.4, and I can edit the workflows. But I do need to make sure it has fully loaded everything, so when I click on action I can see that the action has loaded etc.
I can't save packs.install workflow but I think that might be due to: StackStorm/st2web#859 or StackStorm/st2web#858. (And was same on EWC). So there are some fields I think workflow composer can't handle yet. (which we should probably document).

I installed one of my own packs after upgrade, and could edit its workflows fine. And I can't see any difference in ownership of files.

@winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention).

@amanda11
Copy link

amanda11 commented Mar 1, 2021

Successfully performed U16 upgrade, using the new st2ctl reload register-recreate-virtualenvs.
After upgrade, installed a pack of mine that had an Orquesta workflow and was able to edit it.

I couldn't necessarily find a suitable place to document the problems about editing workflows with the designer that had retry or delay parameters.... There are pull requests in progress for those, so could just create an issue in st2web and link the in-progress PRs to that?

@winem
Copy link

winem commented Mar 1, 2021

@winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention).

The error I saw was not related to these properties. But I was able to confirm that there is a very specific error message for these cases which is good!

To make it short: Both of my the 2 issues are obsolete now.
The issue saving the workflow was caused by a permission issue. Investigation has shown that files of packs that were installed during the self-verification are owned by root:root while those of packs installed via the UI or st2 pack install are owned by root:st2packs. The FlowUI process running under the st2 user does not have the required write permission on those workflows installed during the self-verification.

@amanda11 to me, it looks like we're fine with testing CentOS7 and U16 now. Do you see further test cases? If yes, just let me know and I'll proceed with them.

@blag
Copy link
Author

blag commented Mar 1, 2021

@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕

@amanda11
Copy link

amanda11 commented Mar 1, 2021

@winem I think we've hit everything we need to, unless I'm missing something.

The only thing of concern to me, is whether the issues mentioned on docker are blocking or not.
@armab Do you have a view on that?

Neither myself or @blag are that familiar on the docker use-cases.

Other than that I think we've tested all the scenarios, and fixed the regressions....

@amanda11
Copy link

amanda11 commented Mar 1, 2021

@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕

Yeah - I see they were in progress. And those problems existed in EWC - so agree not blocking for 3.4 release.

@arm4b
Copy link
Member

arm4b commented Mar 1, 2021

Thanks! There is no blocker from the Docker side for starting a release.

In fact, all the deployment methods (Puppet, Ansible, Docker, K8s, Vagrant) really come as a secondary comparing to st2 core, unless there's some terribly wrong that's failing CI and needs fixing.

So we're good from that standpoint 👍

@minsis
Copy link

minsis commented Mar 1, 2021

Ok I'm going to try and explain this the best I can here, sorry if it seems like ramblings.

Something happened with my docker setup when I enabled RBAC with LDAP. I'm still trying to troubleshoot why this is happening but one of the containers is continually trying to authenticate my user and basically locked out my account. My password was reset by our IT team but my account is continually locked out (probably due to caching) and I had to disable LDAP to get this to stop.

Here's the events in order:

  1. LDAP was enabled and working correctly
  2. RBAC was then enabled but wasn't running correctly because I didn't have "backend" set
  3. Set the backend to "default"
  4. My account in LDAP is now locked out.
  5. I restarted containers, rebuilt containers, closed all browsers, etc but I couldn't recover this. Eventually had to disable LDAP all together to get it to stop.

So I'm not exactly sure where or why this happened. I looked at the logs of all the other containers and none of them were trying to auth my account. It was solely the auth container that was trying to auth my account every 5 seconds.

@minsis
Copy link

minsis commented Mar 1, 2021

So I have LDAP disabled and then reenabled RBAC only and its now failing again due to invalid login. So something has to be cached in the database and even with LDAP disabled its still trying to reauth me to the LDAP backend

[auth]
mode = standalone
backend = ldap
backend_kwargs = <--some config stuff-->
enable = False
use_ssl = False
logging = /etc/st2/logging.auth.gunicorn.conf
debug = True

[rbac]
enable = True
backend = default

I'm guessing even though enabled = False, its still trying to auth to ldap since backend is still set to ldap. Other than this issue I'm not even sure how to clear the cache to stop it from trying to auth with my old LDAP login.

@arm4b
Copy link
Member

arm4b commented Mar 1, 2021

Thanks, @minsis.
Looks like you already documented the Feature request/Issue in the respective Docker repository: StackStorm/st2-docker#219.
There is a somewhat similar WIP PR contribution in the K8s from the Community: StackStorm/stackstorm-k8s#182 Though both are not tied to any release.

Because Docker and K8s might need proper implementation in the future, keeping in mind their multi-container specifics, I'd recommend testing LDAP/RBAC with the VM at this moment, following documentation:

@minsis
Copy link

minsis commented Mar 1, 2021

The st-docker#219 issue is just for getting a volume added to store the rbac role files.

The current issue here is that something has broken with LADP and RBAC to where its locking out LDAP accounts because it tries to reauth every 5s (which I also put in an issue on that repo) on a cached credential. I'm just not sure where to go from here in terms of troubleshooting. Every time I turn on LDAP/RBAC it tries to auth my old credentials that are cached somewhere and I can't seem to clear them out.

I would love to test directly on a VM, but at work I have limited resources and it takes our provisioning department weeks to get something built out.

@blag
Copy link
Author

blag commented Mar 1, 2021

You can spin up a VM locally with st2vagrant and configure it that way.

@amanda11
Copy link

amanda11 commented Mar 2, 2021

Since Saturday install on EL8 failing. Looks to be that we are picking up a new version of rabbitmq, which requires a new version of erlang, and that version of erlang is not available in CentOS or the epel repos.
https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.13

Also affecting old releases, unless pin rabbitmq version...

@guzzijones
Copy link

guzzijones commented Mar 2, 2021

I upgraded our dev box.

  1. if you use coordination you will need to reinstall redis using the documented instructions. After the upgrade my st2api stopped working with no module named redis stack trace. pip install redis intructions
  2. also if you were using the 'Enterprise RBAC' or 'Enterprise LDAP' you will be forced to upgrade to one of the unstable dev packages for rbac now. The 'enterprise' backend config option will not work anymore.

@guzzijones
Copy link

FWIW, I am on Centos 7 which now uses python3. I fired off a few large workflows and they all seem to be working as expected.

@guzzijones
Copy link

guzzijones commented Mar 3, 2021

Probably worth mentioning that if on a distro with python2.7 as the default that all sensor packs will need to be changed to python version 3 in the pack.yaml and reinstalled

@blag blag closed this as completed Mar 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants