Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve nightly functional tests #1331

Merged
merged 23 commits into from
Oct 25, 2024
Merged

Improve nightly functional tests #1331

merged 23 commits into from
Oct 25, 2024

Conversation

james-garriss
Copy link
Collaborator

@james-garriss james-garriss commented Sep 24, 2024 •
edited
Loading

🗣 Description

There are several improvements to the nightly functional tests that are all rolled into this PR:

  • Publish to the private gallery, thus ensuring that our code is in a state that can be published to PSGallery.
  • Email TCOs for each failing set of tests, thus allowing them to know when there are issues that may need their attention.
  • Use GitHub caching, thus speeding up testing time by only installing ScubaGear and various dependencies once.
  • Handle concurrency problems within a run by using GitHub's "needs" features for product/tenant pairing, thus ensuring testing one variant doesn't mess with testing a different variant.
  • Handle concurrency problems between runs by using GitHub's "concurrency" features for product/tenant pairing, thus ensuring that one tester doesn't mess with the tests of another tester.
  • Added every combination of products/tenants/variants, thus ensuring more complete coverage. This required additional test parameters for both dev and prod.

The matrix solution that I was using doesn't support concurrency, so I had to replace it with a set of reusable workflows and composite actions. The resulting solution violates DRY more than I want, so I may eventually create an exploratory issue to replace the reusable workflows with composite action.

I also fixed a permissions problem with Checkov and fixed a pathing problem with the CI and workflow pipelines, as they were impacting my work.

💭 Motivation and context

The nightly functional tests were good, but they lacked some features that were causing issues (e.g., concurrency), and they were incomplete.

Closes: #1235
Closes: #1237
Closes: #1278

🧪 Testing

Development was initially done w a temp pipeline whose runs can be found here:
https://github.com/cisagov/ScubaGear/actions/workflows/test_concurrency.yaml
Note: At some point these runs will be cleaned up and deleted.

Once the work was completed, the code was transitioned to the nightly workflow:
https://github.com/cisagov/ScubaGear/actions/workflows/test_production_function.yaml

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • PR targets the correct parent branch (e.g., main or release-name) for merge.
  • Changes are limited to a single goal - eschew scope creep!

Well, I did clean up some other issues, too, but they were impacting my testing.

  • Changes are sized such that they do not touch excessive number of files.
  • All future TODOs are captured in issues, which are referenced in code comments.

I will need to create the TODO I mentioned above when I get back.

  • These code changes follow the ScubaGear [content style guide]
  • Related issues these changes resolve are linked preferably via [closing keywords]
  • All relevant type-of-change labels added.
  • All relevant project fields are set.
  • All relevant functional tests passed.

Actually, the Entra ID functional tests are all failing at this time. But Ted is aware of the issues and is actively working them.

  • All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

  • PR passed smoke test check.
  • Feature branch has been rebased against changes from parent branch, as needed
  • Resolved all merge conflicts on branch
  • Notified merge coordinator that PR is ready for merge via comment mention

✅ Post-merge checklist

  • Feature branch deleted after merge to clean up repository.
  • Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

@james-garriss james-garriss added the enhancement This issue or pull request will add new or improve existing functionality label Sep 24, 2024
@james-garriss james-garriss added this to the Jellyfish milestone Sep 24, 2024
@james-garriss james-garriss self-assigned this Sep 24, 2024
mitchelbaker-cisa pushed a commit that referenced this pull request Nov 6, 2024
@james-garriss @mitchelbaker-cisa
Improve nightly functional tests (#1331)
10c8dae 
* Add a badge for the functional tests (#1319)

* add badge

* fix badge URL

* fix badge url

* fix pipeline badge

* Fix typo in pipeline badge markdown

* Update workflow names for the badges

* Update badge names

* Undo name change

* redo name changes

* use reference links

create workflow

fix lint

set cancel to false

add needs

add more jobs

add composite action

fix typos

add hello name

add create pfx

fix tabbing in create PFX

push on action updates

move env to main workflow

add import pfx

fix typo in actions

add shell to import pfx

add get thumbprint

test pwsh

correct action names

fix env variable for import

fix step names

try real import action

use pw w import

fix indents in thumbprint

add install selenium

add setup sg

combine some steps

add missing env

test product

add missing apos

mod on push

add missing id

try id on action

fix on push

write output

move id

test tp

remove pipe to null

cat

get env

comment out extra steps

look at files

look at more files

look at x

more write host

put tp back

move id

id on both

test input

try env

123

fix 123 typo

use output

test id on step

cleanup output

setup params

test the product

test 2 products

fix product alias variable

switch to teams

switch to prod params, add email

fix indent

use email action not composite

fix comment

Test Defender for email notify

add schedule back

fail on push

use dev params

updated email

test teams

pub to private repo

restore from cache

add with path repo

cd repo

more repo

add repo to import mod

re run

change order

add quotes

change module name

create dir first

remove create dir

use full key

delete caches

set tokn

install gh

comment

reorder

set repo

Comment better

add write privs

change restore order

add quotes to dirs

cleanup the clear cache

use double quotes for program files

test cache

ignore program files

list modules

Change SG dir

add program files back

fix typo

align cache restore

fix path

import selenium

install selenium

hide ast and azure

delete unused steps

fix lint

remove 2 steps

clear cache

fix needs

name clear cache

remove comment

delete 3 caches

remove commas

fix keys

fix keys

debug

fix keys

fix keys

remove newlines

fix keys

use bash array

fix var typo

fix variable

test w 3 products

add email

rewrite cache

comment out exch

fix needs

fix cache key

improve cache

fix email typo

add break to drop out of loop

move teams to separate workflow

set unique name

add push path

comment out all but checkout

comment out clear cache

comment out everything

slim and trim

add permissions

fix permissions

restore publish

fix permissions

add needs

remove workflow call

fix needs

back to workflow

pass secrets to workflow call

add secrets to action

use with for passing secrets

add missing $

fix lint

use env

from reusable workflow to composite

test var

fix value

use secrets

remove echo

use env

wip

wip

wip

wip

fix thumb

complete test

fix inputs

move thumb to env

remove extra inputs

get thumb from env

fix colon

env for thumb

new step for thumb

use with

use output

fix needs

fix needs

input the thumb

fix thumb

fix thumb

fix outputs

read all

fix perms

fix perms

put publish into reusable workflow

fix permissions

add push

put cache in own workflow

set cache permissions

add push

cache write all

constrain delete cache

use job id

test tenant 2

add sharepoint

add exchange

add defender

fix cache dependencies

added all nightly tests - ka-boom!

fix lint

pass github.job

add new inputs

fix needs

hardcode alias

hardcode alias

hard card alias

add entra

fix cache

fix bugs in inputs

fix issues in other products

Fix Checkov

Comment out debugging.

remove debug, fix checkov

fix pipelines runs for the new dirs

fix merge conflicts

fix checkov

suppress checkov error for cache

fix lint

clean up everything for release to prod

Set Defender back to dev params

Set Defender back to prod

always run variants

* change job permissions

* fix paths

* fix cleanup permissions

* fix publish permissions

* fix all test permissions

* set specific cache permissions

* clean up permissions

* test permissions for cache

* fix needs

* remove cache

* update job name

* Removed or for email when pushing code.

* remove comments

* change email type

* simplify

* schedule equal empty

* Print Values

* fix schedule

* fix all if conditions

* uncomment

* Remove debugging

* remove comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dagarwal-mitre dagarwal-mitre dagarwal-mitre approved these changes

@schrolla schrolla schrolla approved these changes

@nanda-katikaneni nanda-katikaneni Awaiting requested review from nanda-katikaneni

Assignees

@james-garriss james-garriss

Labels
enhancement This issue or pull request will add new or improve existing functionality
Projects
None yet
Milestone
Kraken
4 participants
@james-garriss @schrolla @dagarwal-mitre @nanda-katikaneni