[Breaking Change] Runtime exceptions aggregate

[konrad-jamrozik]

Note: the list below is competing for priority with

• Aggregate issue for kojamroz's Dilithium semester backlog #7795

This is an aggregate issue for Breaking Change runtime exceptions:

• [Breaking Change] Swagger BreakingChange validation failure with runtime exception: incompatible properties : tags #5981
• [Breaking Change] ERROR: Schema violation: Data does not match any schemas from 'oneOf' #6036
  • [Breaking Change] Misleading error message: Runtime exception: ERROR: Schema violation: Data does not match any schemas from 'oneOf' #6998
• [Breaking Change] ERROR: Schema violation: Array is too short (0) #6168

And from the old, pre-transfer backlog:

• oad runtime exception caused by retry-after header with integer value  openapi-diff#271
• Runtime Exception - JSON integer is too large or small for an Int64 openapi-diff#258
• [Rule] breaking changes, when x-ms-discriminator-value changes openapi-diff#241
• Runtime Error - something to do with Autorest openapi-diff#239

Related issue:

• [Breaking Change] Breaking change presence should not always result in a required review #6859

Errors reported after 9/11

I.e. after the runtime exception fix got merged and deployed.

• [Breaking Change] Misleading error message: Runtime exception: ERROR: Schema violation: Data does not match any schemas from 'oneOf' #6998
• [Breaking Change] Missing label in spite of label-adding rule XmsEnumChanged failing #6999

References

For more, see comments below.

Konrad's private OneNote page for this issue is #6169 [Breaking Change] Runtime exceptions aggregate

[konrad-jamrozik]

Another case:

• Add single execution history endpoint, refactor copy paste code for j… azure-rest-api-specs#23816 (comment)

[konrad-jamrozik]

Another runtime failure seen here:

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2786392&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03&l=192

Albeit this time the tool did generate some results:

• https://github.com/Azure/azure-rest-api-specs/pull/24059/checks?check_run_id=13597926900
• Adding put and delete BMI operations to baremetalinfrastructure.json spec azure-rest-api-specs#24059

[konrad-jamrozik]

Another occurrence, by Jessica Li on 6/29/2023:

• https://github.com/Azure/azure-rest-api-specs/pull/24387/checks?check_run_id=14608165051
  • Team discussion

[konrad-jamrozik]

Info from Ray about known limitation in this Teams discussion:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/12695

There's a known limitation in the pipeline to remove the label when the new commit was pushed to the PR to remove the breaking change.

[konrad-jamrozik]

Another case:

• [Breaking Change] Runtime exception on breaking change check #6526

[konrad-jamrozik]

Another case:

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2962440&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03&l=52
• Context on Teams

Some clues from the error log:

ERROR: Schema violation: Missing required property: name\n - file:///mnt/vss/work/1/azure-rest-api-specs/specification/cognitiveservices/data-plane/Language/stable/2022-05-01/analyzeconversations.json:2077:10 ($.definitions.apimkey.properties.name["x-ms-enum"])\nFATAL: swagger-document/individual/schema-validator - FAILED\nFATAL: Error: [OperationAbortedException] Error occurred. Exiting.\nProcess() cancelled due to exception : [OperationAbortedException] Error occurred. Exiting.\n"

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs/pull/25016/checks?check_run_id=15442460997
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2958286&view=logs&j=8f628bfe-7d8d-5bf3-e141-37106862c4f5&t=723dfc11-d28f-5d74-45f1-663cde6746b4

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/13923/checks?check_run_id=15390299761
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2952865&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03

The tool detected the breaking change, but then failed with runtime error, so it never produced the signal required for automation to add necessary labels.

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/11374
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2933861&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03&l=202

[konrad-jamrozik]

Another one:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/13606/checks?check_run_id=16018267282
• Email thread: RE: Breaking change approval request - new APIs

This case denotes the runtime exception may possibly be caused by the tool trying to compare to a non-existent file.

[konrad-jamrozik]

• [Breaking Change] Cross-Version Runtime Exception: Error: Error retrieving property #6447

[konrad-jamrozik]

I did analysis and bucketing of some of the runtime errors. See here:

• https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_git/openapi-alps/pullRequest/500754#1695931281

[konrad-jamrozik]

So far big chunk of this work was addressed by:

• Pull Request 500754: Implement support for pretty-printing runtime errors originating from AutoRest invocation from within OAD (breaking change detector)

[konrad-jamrozik]

Closing for now. May revisit with future work later. For now we identified more urgent work.

[konrad-jamrozik]

Reopening for Dilithium