Filter test collection based on files changed in upstream PRs

[tpapaioa]

Problem Statement

Filter tests based on the files modified in an upstream (foreman or katello) PR.

Solution

The new upstream_pr pytest plugin uses the PyGithub library to get the list of files modified by the given PR, and filters the test collection based on the mapping of upstream repo paths/filenames to test markers in robottelo.

For example, with the following new settings block settings.github:

$ cat conf/github.yaml
GITHUB:
  FOREMAN:
    REPO: foreman
    ORG: theforeman
    RULES:
      - PATH: "app/models/concerns/hostext/"
        MARKER: component
        MARKER_ARG: Hosts
      - PATH: "webpack/assets/javascripts/react_app/components/SettingUpdateModal/"
        MARKER: component
        MARKER_ARG: Settings
  KATELLO:
    REPO: katello
    ORG: Katello
    RULES:
      - PATH: "app/controllers/katello/api/v2/errata_controller.rb"
        MARKER: component
        MARKER_ARG: ErrataManagement

then the following Foreman PR, which modifies files under webpack/assets/javascripts/react_app/components/SettingUpdateModal/, should filter the tests on the Settings component. In this case, --upstream-pr foreman/9969 is equivalent to --component Settings:

Fixes #37013 - change the 'All hosts' menu item's url based on the new_host_page setting
theforeman/foreman#9969

>>> FOREMAN_PR_ID = 9969
>>> from github import Github
>>> from pprint import pp
>>> pr = Github().get_repo("theforeman/foreman").get_pull(FOREMAN_PR_ID)
>>> filenames = [file.filename for commit in pr.get_commits() for file in commit.files]
>>> pp(filenames)
['app/registries/foreman/settings/general.rb',
 'webpack/assets/javascripts/react_app/Root/Context/ForemanContext.js',
 'webpack/assets/javascripts/react_app/Root/ReactApp.js',
 'webpack/assets/javascripts/react_app/components/Layout/LayoutConstants.js',
 'webpack/assets/javascripts/react_app/components/Layout/LayoutHelper.js',
 'webpack/assets/javascripts/react_app/components/Layout/Navigation.js',
 'webpack/assets/javascripts/react_app/components/Layout/index.js',
 'webpack/assets/javascripts/react_app/components/SettingUpdateModal/SettingUpdateModalConstants.js',
 'webpack/assets/javascripts/react_app/components/SettingUpdateModal/components/SettingForm/SettingForm.js',
 'webpack/test_setup.js']

$ pytest tests/foreman --co --upstream-pr foreman/9969
[...]
=== 54/5426 tests collected (5372 deselected) in 16.91s ===

$ pytest tests/foreman --co --component Settings
[...]
=== 54/5426 tests collected (5372 deselected) in 15.14s ===

Similarly, for this Katello PR:

Fixes #37394 - content_view_filter_id only works for ID-type filters
Katello/katello#10985

the file app/controllers/katello/api/v2/errata_controller.rb is matched exactly, and tests with the ErrataManagement component are selected:

>>> KATELLO_PR_ID = 10985
>>> pr = Github().get_repo("Katello/katello").get_pull(KATELLO_PR_ID)
>>> filenames = [file.filename for commit in pr.get_commits() for file in commit.files]
>>> pp(filenames)
['app/controllers/katello/api/v2/debs_controller.rb',
 'app/controllers/katello/api/v2/errata_controller.rb',
 'app/controllers/katello/api/v2/packages_controller.rb']

$ pytest tests/foreman --co --upstream-pr katello/10985
[...]
=== 70/5426 tests collected (5356 deselected) in 15.45s ===

$ pytest tests/foreman --co --component ErrataManagement
[...]
== 70/5426 tests collected (5356 deselected) in 13.78s ===

[ogajduse]

The idea looks good to me overall, but I do not like the config maintenance that goes with the change and that someone needs to do once this patch gets merged. If the structure of the upstream repository changes, we need to update our config too. If that author is an RH associate or someone from ATIX, I can imagine them updating the config here. But if it is another community member, they might not even know that robottelo exists. Also, I see that you added only a template config. That indicates that we will store the actual config in our internal repository. If that is the case, community contributors do not have a chance to update it and someone from SatQE has to watch for upstream repository structure changes and update the config.

I would say that it is worth opening an RFC in the Foreman community to discuss this first before we proceed further here.

Summed up:

• maintenance of the rules part of the config
  • when do we update it
  • who should update it
    • what if it is a community contributor changing the repo structure, how do they know they should update robottelo configuration?
  • non-public config
  • filing an RFC to get more attention and ideas on how to implement it would be nice

[github-actions]

This pull request has not been updated in the past 45 days.

[github-actions]

This pull request has not been updated in the past 45 days.

[github-actions]

This pull request is now being closed after stale warnings.