FI-2937 inferno execute

[Shaumik-Ashraf]

Summary

Putting up an early PR for feedback. This implements a very basic bundle exec inferno execute command that will run an entire test suite synchronously and output color-coded test results, messages, and requests. Note:

• Test results not exactly in same order as web ui, I didn't walk through the test entities I just ran the whole suite
• I couldn't set Inferno::Application['async_jobs] properly when booting inferno from CLI, so I ended up adding a force_synchronous kwarg to Jobs
• Colors from from Pastel gem instead of Thor or colorize, Pastel is made from the same person as tty-markdown

TODOs

• Factorize code
• Clean code
• Testing & RSpec

Testing Guidance OUTDATED SEE BELOW FOR NEW TESTING GUIDANCE

1. checkout branch
2. bundle exec rspec
3. bundle exec inferno services start
4. copy and paste command below to run infrastructure dev suite, which has same results as web UI:

bundle exec ./bin/inferno execute --suite infra_test --inputs suite_input:a outer_group_input:b inner_group_input:c test_input:d external_test1_input:e external_inner_group_input:f external_outer_group_input:g

5. run new validator dev suite on US Core patient, which correctly fails since no IG is loaded

bundle exec inferno execute -s dev_validator -i "url:https://inferno.healthit.gov/reference-server/r4" access_token:SAMPLE_TOKEN patient_id:85

6. run again in verbose mode to see error messages

bundle exec inferno execute -s dev_validator -i "url:https://inferno.healthit.gov/reference-server/r4" access_token:SAMPLE_TOKEN patient_id:85 --verbose

7. run validator dev suite on generic patient with all tests passing

bundle exec inferno execute -s dev_validator -i "url:https://hapi.fhir.org/baseR4" patient_id:1234321

8. run a group

bundle exec inferno execute -g dev_validator-patient_group -i "url:https://hapi.fhir.org/baseR4" patient_id:1234321

9. run help

bundle exec inferno help execute

10. see this github action on our reference server for an example in CI, this action correctly fails.

Updated testing guidance 8/9/2024. Ready for review.
Updated again 8/28/2024. Ready for review.

[codecov]

Codecov Report

Attention: Patch coverage is 98.55072% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.99%. Comparing base (e5dd3cb) to head (f796e1d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/inferno/jobs.rb	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #517      +/-   ##
==========================================
+ Coverage   79.86%   79.99%   +0.12%     
==========================================
  Files         253      257       +4     
  Lines       13352    13403      +51     
  Branches     1290     1290              
==========================================
+ Hits        10664    10722      +58     
+ Misses       1939     1932       -7     
  Partials      749      749              

Flag	Coverage Δ
backend	92.11% <98.55%> (+0.26%)	⬆️
frontend	74.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Shaumik-Ashraf]

Updated Testing Guidance

1. Checkout branch
2. bundle exec rspec
3. bundle exec inferno services start
4. Run test suite with validator and inputs, all tests pass correctly:

bundle exec inferno execute --suite dev_validator --inputs "url:https://hapi.fhir.org/baseR4" patient_id:1234321

5. Run test suite on US Core patient with tests that fail correctly:

bundle exec inferno execute -s dev_validator -i "url:https://inferno.healthit.gov/reference-server/r4" access_token:SAMPLE_TOKEN patient_id:85

6. Run test suite with suite options:

bundle exec inferno execute --suite options --suite-options ig_version:1 another_option:1 --inputs all_versions_input:AAA v1_input:BBB

7. Run a test suite with specific groups by short id

bundle exec inferno execute -s dev_validator -i "url:https://hapi.fhir.org/baseR4" patient_id:1234321 --groups 1

8. Run a test suite with specific tests by short id

bundle exec inferno execute -s dev_validator -i "url:https://hapi.fhir.org/baseR4" patient_id:1234321 --tests 1.01 1.02

[Jammjammjamm]

I don't think we should add abstract classes. If you think we need some sort of really basic output class, make this some sort of null output class that works but doesn't print anything.

[Jammjammjamm]

Don't make a bang method if there isn't a version without a bang.

[Jammjammjamm]

This dosen't seem right:

ᐅ bundle exec inferno execute -s dev_validator -i "url:https://hapi.fhir.org/baseR4" patient_id:1234321 --tests 1.01 1.02

================================================================================
Testing dev_validator
================================================================================
Running tests. This may take a while...
================================================================================
Test Results:
================================================================================
1.01 Patient Read Test: ✓ pass
Validator Suite: ✓ pass
1 Patient Test Group: ✓ pass
1.02 Patient Validate Test: ✓ pass
================================================================================

If short ids are provided, I wouldn't expect the suite to run.

Also, users shouldn't have to use a different command line argument depending on whether the short ids belong to tests or groups. There should be a single argument for short ids which can take both.

[Shaumik-Ashraf]

The implementation is literally using test_runs_repo.results_for_test_run, and doesn't actually run the suite if short ids are specified. Inferno seems to rollup results to all parents whenever a single test it run, and if this is a bug I feel like it should be a separate ticket.

[Jammjammjamm]

Gotcha, that is necessary behavior for the UI, but is confusing here, so can you make a ticket for that?

If you want to get this in and unify the short id input in a follow-up ticket, that is fine with me, I'd just want it to be the next one you pick up.