FI-2728: Prevent Test Runner from Crashing When Validator Response Contains Bad Characters

[vanessuniq]

Summary

Ticket Description: We saw an issue where a validator response contained a null character and it crashed the test runner, so it kept spinning forever. Since we don't control the validator and especially the tx server, make sure an unexpected response doesn't crash anything.

I was able to reproduce the issue by mocking the validator service with various unexpected response formats, including non-printable/control characters, which led to crashes when the test runner attempted to persist the response. By using Mockoon to simulate these edge cases, I confirmed that the issue was specifically related to non-printable characters in the response body, which caused SQLite exceptions during persistence.

Changes Made:

• Sanitized the response body: The response body is now sanitized to remove non-printable/control characters before parsing or persisting, preventing crashes and database exceptions.
• Improved JSON Parsing Error Handling: The JSON parsing process has been refined, ensuring that malformed responses are caught, and users are provided with clear, actionable error messages.
• Prevent storing null validator session: Added a null check to prevent storing null validator session IDs (This should fix ticket FI-2914)
• Update unit tests accordingly

[codecov]

Codecov Report

Attention: Patch coverage is 70.83333% with 7 lines in your changes missing coverage. Please review.

Project coverage is 79.66%. Comparing base (e1b8fb6) to head (a894c0c).
Report is 1 commits behind head on main.

Files	Patch %	Lines
lib/inferno/jobs/invoke_validator_session.rb	0.00%	6 Missing ⚠️
lib/inferno/dsl/fhir_resource_validation.rb	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #518      +/-   ##
==========================================
+ Coverage   79.65%   79.66%   +0.01%     
==========================================
  Files         249      249              
  Lines       13177    13179       +2     
  Branches     1289     1283       -6     
==========================================
+ Hits        10496    10499       +3     
+ Misses       1932     1931       -1     
  Partials      749      749              

Flag	Coverage Δ
backend	91.69% <70.83%> (+0.02%)	⬆️
frontend	74.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dehall]

Ok I was able to reproduce the issue while testing this. The ticket says "bad characters" but I think the only real culprit is the null character. Using mockoon I wasn't able to get a realistic response containing the null character, but I was able to get *a* response containing the null character, and that produced the same problem as Steve originally reported on the bulk data test kit PR a while back: the test runner hits an exception in SQLite and then the UI spins forever. Can you try again with a response specifically containing a null character? I was able to get a null character to paste with the (mac) terminal command echo "\u0000" | pbcopy .

[dehall]

Because this runnable gets persisted to the DB, I think this might be where the change to handle null characters will need to be.

[vanessuniq]

Ok I was able to reproduce the issue while testing this. The ticket says "bad characters" but I think the only real culprit is the null character. Using mockoon I wasn't able to get a realistic response containing the null character, but I was able to get a response containing the null character, and that produced the same problem as Steve originally reported on the bulk data test kit PR a while back: the test runner hits an exception in SQLite and then the UI spins forever. Can you try again with a response specifically containing a null character? I was able to get a null character to paste with the (mac) terminal command echo "\u0000" | pbcopy .

Thanks! that helped.

[Jammjammjamm]

Get rid of the comment and make a method called remove_invalid_characters or something like that.