StackStorm v3.4.0 Pre-release Testing

[blag]

We're ready to prepare the StackStorm v3.4 release and start pre-release testing..

Release Process Preparation

Per Release Management Schedule @blag is the Release Manager and @amanda11 assisting for v3.4. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels.
The first step is pre-release manual user-acceptance testing for v3.4dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough.
There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.4dev unstable packages, try random things in random environments (different OS) and report any regressions found comparing to v3.3 (and/or v3.2 if you are still running ST2 Enterprise):

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

You can also run StackStorm on a specific OS with st2vagrant:

git clone https://github.com/StackStorm/st2vagrant.git
cd st2vagrant
VERSION=3.4dev vagrant up  # Default OS is Ubuntu Xenial (16.04)

BOX=ubuntu/bionic64 VERSION=3.4dev vagrant up  # Ubuntu Bionic 18.04

BOX=centos/7 VERSION=3.4dev vagrant up  # CentOS 7

BOX=centos/8 VERSION=3.4dev vagrant up  # CentOS 8

Extra points for PR hotfixes and adding new or missing test cases.

Specific things to check

• Run the self-verification script - see instructions here
  • Ensure that the st2resultstracker service doesn't exist and NOT running
  • Ensure that StackStorm is running on Python 3 - st2 --version
• Test the web UI and the integrated flow UI - see instructions here
• Test Ubuntu16.04 or CentOS7/RHEL7 installation and upgrade as they've moved from py2 to py3
• If you have a pack that utilizes Python modules from actions/lib, please ensure that those actions still work. In the Python 2.7 removal, we adjusted how we handle this code, so we want to ensure that we do not have any regressions.
• If you have access to a Windows server, please test out a few actions from the Windows pack. We have not been testing this in our end-to-end tests because the Windows instances were not responding to WinRM queries, so we want to ensure no regressions (this might be related to the actions/lib issue, we haven't troubleshooted this enough to be sure).
• If you are using either the LDAP authentication or the RBAC authorization backends, please enable those backends and ensure that the integration continues to work.
  • LDAP backend (https://docs.stackstorm.com/latest/authentication.html#ldap)
    • - users can login with their LDAP credentials
  • RBAC backend (https://docs.stackstorm.com/latest/rbac.html#enabling-rbac)
    • users without permission grants are not allowed to access resources
    • users with permission grants are allowed to access protected resources
    • if you are also running ChatOps, ensure that the !help (with st2chatops/hubot-stackstorm) or !st2help (with err-stackstorm) commands work. See 500 on actionalias help endpoint with action_alias_help permission grant st2-rbac-backend#47 for more information.
• StackStorm upgrade from 3.3 to 3.4, following the Upgrade Instructions: https://docs.stackstorm.com/latest/install/upgrades.html#v3-4

If you have successful test results, please post a summary of what all you tested (OSes, what features you tested).

If you run into any bugs, please open them in the respective repositories and link to this issue from there. I will add them to the list at the bottom of this description.

If you have any issues running StackStorm or running the tests, please post down below.

Major changes

• Removed Python 2.7 support for all OSes
  • CentOS7/RHEL7 migrated from py2 to py3
  • Ubuntu 16.04 LTS migrated from py2 to py3
• Integrated ST2 RBAC backend
• Integrated LDAP authentication backend
• Integrated Workflow Designer (st2flow) into Workflow Composer (st2web)
• Added notification support for err-stackstorm
• Updated deprecation warning for Python 2 packs
• Added a timeout parameter to packs.install
• Updated dependencies
• Pack files are now interpreted as UTF-8 instead of ASCII
• Fixed broken st2 action-alias execute command
• Removed st2resultstracker

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

• Added support for GitLab SSH URLs on pack install and download actions. (improvement) #5050
  Contributed by @asthLucas

• Added st2-rbac-backend pip requirements for RBAC integration. (new feature) #5086
  Contributed by @hnanchahal

• Added notification support for err-stackstorm. (new feature) #5051

• Added st2-auth-ldap pip requirements for LDAP auth integartion. (new feature) #5082
  Contributed by @hnanchahal

Changed

• Updated deprecation warning for python 2 pack installs, following python 2 support removal. #5099
  Contributed by @amanda11

• Improve the st2-self-check script to echo to stderr and exit if it isn't run with a
  ST2_AUTH_TOKEN or ST2_API_KEY environment variable. (improvement) #5068

• Added timeout parameter for packs.install action to help with long running installs that exceed the
  default timeout of 600 sec which is defined by the python_script action runner (improvement) #5084

  Contributed by @hnanchahal

• Upgraded cryptography version to 3.2 to avoid CVE-2020-25659 (security) #5095

• Converted most CI jobs from Travis to GitHub Actions (all except Integration tests).

  Contributed by @nmaludy, @winem, and @blag

• Updated cryptography dependency to version 3.3.2 to avoid CVE-2020-36242 (security) #5151

Fixed

• Pin chardet version as newest version was incompatible with pinned requests version #5101
  Contributed by @amanda11

• Fixed issue were st2tests was not getting installed using pip because no version was specified.
  Contributed by @anirudhbagri

• Added monkey patch fix to st2stream to enable it to work with mongodb via SSL. (bug fix) #5078 #5091

• Fix nginx buffering long polling stream to client. Instead of waiting for closed connection
  wait for final event to be sent to client. (bug fix) #4842 #5042

  Contributed by @guzzijones

• StackStorm now explicitly decodes pack files as utf-8 instead of implicitly as ascii (bug fix)
  #5106, #5107

• Fix incorrect array parameter value casting when executing action via chatops or using
  POST /aliasexecution/match_and_execute API endpoint. The code would incorrectly assume the
  value is always a string, but that may not be the cast - they value could already be a list and
  in this case we don't want any casting to be performed. (bug fix) #5141

  Contributed by @Kami.

• Fix @parameter_name=/path/to/file/foo.json notation in the st2 run command which didn't
  work correctly because it didn't convert read bytes to string / unicode type. (bug fix) #5140

  Contributed by @Kami.

• Fix broken st2 action-alias execute command and make sure it works
  correctly. (bug fix) #5138

  Contributed by @Kami.

Removed

• Removed --python3 pack install option #5100
  Contributed by @amanda11

• Removed submit-debug-info tool and the st2debug component #5103

• Removed check-licence script (cleanup) #5092

  Contributed by @kroustou

• Updated Makefile and CI to use Python 3 only, removing Python 2 (cleanup) #5090

  Contributed by @blag

• Remove st2resultstracker from st2ctl, the development environment and the st2actions setup.py (cleanup) #5108

  Contributed by @winem

orquesta 1.3.0

Added

• Implement the orquesta-rehearse command to run unit tests for workflow. (new feature, beta)
  Contributed by AJ (guzzijones)

Changed

• Remove mistral fixtures because they were used as ref to support multiple specs. Mistral is
  deprecated in StackStorm and there's no intention to support it here. A mistral to orquesta
  workflow conversion tool is available at https://github.com/StackStorm/orquestaconvert.

Fixed

• Fix the custom flake8 extensions to check import statements. (bug fix)
• Fix the deprecation warning that flags are not at the start of the expression. (bug fix)

st2chatops

No changes

StackStorm Exchange

• Removed Python 2.7 support and testing from almost all packs
• Added environment variable flag to avoid publishing test packs to exchange.stackstorm.com

Conclusion

Please report findings here and bugs/regressions in respective repositories.
Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.

Issues Found During Release

PRs Merged for Release

TODOs

• Blog post for release
• Blog post for exchange/community update

[himadriganguly]

st2ctl status showing

##### st2 components status #####
st2actionrunner PID: 18733
st2actionrunner PID: 18737
st2actionrunner PID: 18740
st2actionrunner PID: 18743
st2actionrunner PID: 18746
st2actionrunner PID: 18749
st2actionrunner PID: 18753
st2actionrunner PID: 18756
st2actionrunner PID: 18762
st2actionrunner PID: 18765
st2api PID: 18635
st2api PID: 18647
st2stream PID: 18583
st2stream PID: 18628
st2auth PID: 18523
st2auth PID: 18535
st2garbagecollector PID: 17552
st2notifier PID: 17555
st2rulesengine PID: 17558
st2sensorcontainer PID: 18653
st2chatops is not running.
st2timersengine PID: 17569
st2workflowengine PID: 18665
st2scheduler PID: 17575

in the result above it is showing st2chatops not running.

sudo service st2chatops status

● st2chatops.service - StackStorm service st2chatops
   Loaded: loaded (/lib/systemd/system/st2chatops.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sat 2021-02-20 16:04:35 UTC; 3s ago
  Process: 22638 ExecStart=/opt/stackstorm/chatops/bin/hubot $DAEMON_ARGS (code=exited, status=1/FAILURE)
 Main PID: 22638 (code=exited, status=1/FAILURE)
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Main process exited, code=exited, status=1/FAILURE
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Unit entered failed state.
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Failed with result 'exit-code'.

Even after the machine restart, the same thing happens.

[blag]

@himadriganguly Thank you for testing v3.4a!

That's normal when you haven't configured st2chatops by modifying /opt/stackstorm/chatops/st2chatops.env. The default StackStorm installation will not configure ChatOps for you since that is an optional component that requires integration with an external service.

We should try to improve that situation in the future, but for v3.4 I'm not considering that a bug.

If you have configured st2chatops, then that's definitely a bug. In which case, please post your st2chatops.env file (but remember to redact any API keys or Slack tokens) so we can figure out if it's a bug in your configuration or in st2chatops.

[himadriganguly]

Running Self-Verification Result

Click to expand!

$ sudo ST2_AUTH_TOKEN=$(st2 auth st2admin -p 'Ch@ngeMe' -t) /opt/stackstorm/st2/bin/st2-self-check
~ ~
Cloning st2tests (branch=master) in /home/vagrant.
Cloning into 'st2tests'...
remote: Enumerating objects: 323, done.
remote: Counting objects: 100% (323/323), done.
remote: Compressing objects: 100% (184/184), done.
remote: Total 323 (delta 145), reused 251 (delta 121), pack-reused 0
Receiving objects: 100% (323/323), 703.27 KiB | 396.00 KiB/s, done.
Resolving deltas: 100% (145/145), done.
Checking connectivity... done.
Copying asserts, fixtures, tests and examples packs.
Installing asserts, fixtures, tests and examples packs.
..................
id: 60313654f6df70916a47edb3
action.ref: packs.setup_virtualenv
context.user: st2admin
parameters: 
  packs:
  - examples
  - tests
  - asserts
  - fixtures
  - webui
status: succeeded
start_timestamp: Sat, 20 Feb 2021 16:18:28 UTC
end_timestamp: Sat, 20 Feb 2021 16:19:05 UTC
result: 
  exit_code: 0
  result: 'Successfully set up virtualenv for the following packs: examples, tests, asserts, fixtures, webui'
  stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "examples" (/opt/stackstorm/packs/examples)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/examples" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "examples" in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/examples" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --no-download /opt/stackstorm/virtualenvs/examples" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/examples/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/examples/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/examples/requirements.txt with command /opt/stackstorm/virtualenvs/examples/bin/pip install -U -r /opt/stackstorm/packs/examples/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "examples" successfully created in "/opt/stackstorm/virtualenvs/examples"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "tests" (/opt/stackstorm/packs/tests)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/tests" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "tests" in "/opt/stackstorm/virtualenvs/tests"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/tests" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --no-download /opt/stackstorm/virtualenvs/tests" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/tests/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    No pack specific requirements found
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "tests" successfully created in "/opt/stackstorm/virtualenvs/tests"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "asserts" (/opt/stackstorm/packs/asserts)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/asserts" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "asserts" in "/opt/stackstorm/virtualenvs/asserts"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/asserts" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/asserts" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/asserts/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/asserts/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/asserts/requirements.txt with command /opt/stackstorm/virtualenvs/asserts/bin/pip install -U -r /opt/stackstorm/packs/asserts/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "asserts" successfully created in "/opt/stackstorm/virtualenvs/asserts"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "fixtures" (/opt/stackstorm/packs/fixtures)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/fixtures" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "fixtures" in "/opt/stackstorm/virtualenvs/fixtures"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/fixtures" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/fixtures" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/fixtures/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/fixtures/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/fixtures/requirements.txt with command /opt/stackstorm/virtualenvs/fixtures/bin/pip install -U -r /opt/stackstorm/packs/fixtures/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "fixtures" successfully created in "/opt/stackstorm/virtualenvs/fixtures"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "webui" (/opt/stackstorm/packs/webui)
    st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/webui" doesn''t exist
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "webui" in "/opt/stackstorm/virtualenvs/webui"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/webui" using Python binary "/opt/stackstorm/st2/bin/python"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/webui" to create virtualenv.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/webui/bin/pip install six>=1.9.0,<2.0.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/webui/requirements.txt"
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/webui/requirements.txt with command /opt/stackstorm/virtualenvs/webui/bin/pip install -U -r /opt/stackstorm/packs/webui/requirements.txt.
    st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Virtualenv for pack "webui" successfully created in "/opt/stackstorm/virtualenvs/webui"
    '
  stdout: ''
Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
2021-02-20 16:19:06,282 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2021-02-20 16:19:06,289 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2021-02-20 16:19:06,737 INFO [-] =========================================================
2021-02-20 16:19:06,738 INFO [-] ############## Registering triggers #####################
2021-02-20 16:19:06,738 INFO [-] =========================================================
2021-02-20 16:19:06,984 INFO [-] Registered 1 triggers.
2021-02-20 16:19:06,984 INFO [-] =========================================================
2021-02-20 16:19:06,984 INFO [-] ############## Registering sensors ######################
2021-02-20 16:19:06,984 INFO [-] =========================================================
2021-02-20 16:19:07,182 INFO [-] Registered 7 sensors.
2021-02-20 16:19:07,182 INFO [-] =========================================================
2021-02-20 16:19:07,183 INFO [-] ############## Registering runners ######################
2021-02-20 16:19:07,183 INFO [-] =========================================================
2021-02-20 16:19:07,977 INFO [-] Registered 14 runners.
2021-02-20 16:19:07,977 INFO [-] =========================================================
2021-02-20 16:19:07,977 INFO [-] ############## Registering actions ######################
2021-02-20 16:19:07,978 INFO [-] =========================================================
2021-02-20 16:19:15,491 INFO [-] Registered 230 actions.
2021-02-20 16:19:15,491 INFO [-] =========================================================
2021-02-20 16:19:15,491 INFO [-] ############## Registering rules ########################
2021-02-20 16:19:15,492 INFO [-] =========================================================
2021-02-20 16:19:15,767 INFO [-] Registered 9 rules.
2021-02-20 16:19:15,767 INFO [-] =========================================================
2021-02-20 16:19:15,767 INFO [-] ############## Registering aliases ######################
2021-02-20 16:19:15,767 INFO [-] =========================================================
2021-02-20 16:19:17,329 INFO [-] Registered 134 aliases.
2021-02-20 16:19:17,329 INFO [-] =========================================================
2021-02-20 16:19:17,329 INFO [-] ############## Registering policy types #################
2021-02-20 16:19:17,329 INFO [-] =========================================================
2021-02-20 16:19:17,364 INFO [-] Registered 3 policy types.
2021-02-20 16:19:17,364 INFO [-] =========================================================
2021-02-20 16:19:17,364 INFO [-] ############## Registering policies #####################
2021-02-20 16:19:17,364 INFO [-] =========================================================
2021-02-20 16:19:17,365 INFO [-] Registered 0 policies.
2021-02-20 16:19:17,365 INFO [-] =========================================================
2021-02-20 16:19:17,365 INFO [-] ############## Registering configs ######################
2021-02-20 16:19:17,366 INFO [-] =========================================================
2021-02-20 16:19:17,368 INFO [-] Registered 0 configs.
##### st2 components status #####
st2actionrunner PID: 1330
st2actionrunner PID: 1356
st2actionrunner PID: 1366
st2actionrunner PID: 1392
st2actionrunner PID: 1415
st2actionrunner PID: 1421
st2actionrunner PID: 1427
st2actionrunner PID: 1470
st2actionrunner PID: 1497
st2actionrunner PID: 1539
st2api PID: 1166
st2api PID: 1772
st2stream PID: 1226
st2stream PID: 1764
st2auth PID: 1234
st2auth PID: 1733
st2garbagecollector PID: 1169
st2notifier PID: 1205
st2rulesengine PID: 1220
st2sensorcontainer PID: 1179
st2chatops is not running.
st2timersengine PID: 1155
st2workflowengine PID: 1210
st2scheduler PID: 1185
Deleting 'st2tests' directory from /home/vagrant.
~
Attempting Test tests.test_inquiry_chain...OK! (55s)
Attempting Test tests.test_key_triggers...OK! (44s)
Attempting Test tests.test_pack_install_tool...OK! (17s)
Attempting Test tests.test_packs_pack...OK! (130s)
Attempting Test tests.test_quickstart...OK! (27s)
Attempting Test tests.test_quickstart_key...OK! (7s)
Attempting Test tests.test_quickstart_local_script_actions...OK! (13s)
Attempting Test tests.test_quickstart_passive_sensor...OK! (7s)
Attempting Test tests.test_quickstart_polling_sensor...OK! (5s)
Attempting Test tests.test_quickstart_python_actions...OK! (13s)
Attempting Test tests.test_quickstart_remote_script_actions...OK! (17s)
Attempting Test tests.test_quickstart_rules...OK! (64s)
Attempting Test tests.test_quickstart_trace...OK! (21s)
Attempting Test tests.test_render_config_context...OK! (5s)
Attempting Test tests.test_run_pack_tests_tool...OK! (82s)
Attempting Test tests.test_timer_rule...OK! (41s)
Skipping tests.test_windows_runners...
Skipping tests.test_winrm_large_script...
Skipping tests.test_winrm_runners...
Attempting Example examples.orquesta-examples...OK! (9s)
SELF CHECK SUCCEEDED!
st2-self-check succeeded.

#############################################################
###################################################   #######
###############################################   /~\   #####
############################################   _- `~~~', ####
##########################################  _-~       )  ####
#######################################  _-~          |  ####
####################################  _-~            ;  #####
##########################  __---___-~              |   #####
#######################   _~   ,,                  ;  `,,  ##
#####################  _-~    ;'                  |  ,'  ; ##
###################  _~      '                    `~'   ; ###
############   __---;                                 ,' ####
########   __~~  ___                                ,' ######
#####  _-~~   -~~ _                               ,' ########
##### `-_         _                              ; ##########
#######  ~~----~~~   ;                          ; ###########
#########  /          ;                        ; ############
#######  /             ;                      ; #############
#####  /                `                    ; ##############
###  /                                      ; ###############
#                                            ################

[himadriganguly]

@blag Thank you for the explanation. I will be testing other things according to the checklist and will post the details. I will try to contribute as much as I can.

[winem]

Just a short update since I ran into an issue with the load_diag workflow in the Linux pack and I'll look into that now:

• Installation on CentOS7 works as expected (WebUI works, st2ctl commands work, running different actions of different packs works)

Update: The issue with the load_diag workflow isn't a regression or issue specific to st2 3.4 and there are already open issues that cover these issues. So I'll proceed.

[Kami]

I run tests on Ubuntu Xenial (16.04) here are the issues I found:

1. st2 run core.local cmd='python --version' returns Python 2

This may or may not be expected and desired, depending on how we envision whole Python 3 story. It returns python 2 since system default python binary in PATH is used instead of python binary from /opt/stackstorm/st2/bin.

It does appear to be working fine for python runner actions though (which is likely the only thing we care about) - st2 run examples.python_runner_print_python_version, /opt/stackstorm/st2/bin/python --version.

vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_version
.
id: 60319401b8c38397f0e40066
action.ref: examples.python_runner_print_python_version
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
end_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26) 
[GCC 5.4.0 20160609]
"

vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_environment
.
id: 60319560b8c38397f0e40069
action.ref: examples.python_runner_print_python_environment
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
end_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26) 
[GCC 5.4.0 20160609]
Platform: Linux-4.4.0-161-generic-x86_64-with-Ubuntu-16.04-xenial
PYTHONPATH: /opt/stackstorm/virtualenvs/examples/lib/python3.6:/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages:/opt/stackstorm/packs/examples/actions/lib::/opt/stackstorm/st2/lib/python3.6/site-packages
sys.path: ['/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner', '/opt/stackstorm/virtualenvs/examples/lib/python3.6', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/packs/examples/actions/lib', '/', '/opt/stackstorm/st2/lib/python3.6/site-packages', '/opt/stackstorm/virtualenvs/examples/lib/python36.zip', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/opt/stackstorm/packs/examples/actions/pythonactions']

As far as core.local "issue" goes, we need to decide what's the desired and expected behavior. I believe we should not add /opt/stackstorm/st2/bin/ to PATH for core.local actions - core local actions should observe same PATH rules as any other script which is manually ran from a shell session on that server.

In short, we only care about python version being used for Python runner actions and StackStorm services.

If that's the case, testing instructions need to be updated.

2. bash completion for st2 CLI doesn't seem to work out of the box

It could be that this has been broken for a long time already, but I do know it worked at some point in the distant past (perhaps we just don't source source the bash completion script by default, didn't dig in).

[arm4b]

@Kami I agree about 1) /opt/stackstorm/st2/bin/python is something to verify for the new py env.
I've updated the instructions so the expectation is to make sure that st2 --version is showing py3.

[winem]

I'm done with the tests of fresh st2 installations on CentOS7 and Ubuntu 16.04 now as well. I created an issue + PR for a small issue with the install script on Ubuntu 16.04 and can also confirm topics reported by @Kami

This means:

• st2 installation
• use the web & Flow UI, st2 cli
• install & removing packs
• run actions & workflows
• run the self-verification

I also did a cross-check with st2 v3.3 and the st2 auto-completion does not work, too. So this is not a new issue.

I'll proceed with tests upgrading st2 v3.3 to 3.4 on both OS.

[amanda11]

Started testing on CentOS 8.
Found issue that the ansible install docs page hasn't been updated given the EWC donated features - PR in progress
Also should get StackStorm/st2docs#1022 reviewed and merged, as the matching code is merged but not the doc PR.