[FEATURE] Support correctness validation in OSB workloads

[jainankitk]

Is your feature request related to a problem?

The OSB workloads do not validate response of the search requests. Any 200 OK response work well from the benchmark perspective.

What solution would you like?

Given we have these workloads running nightly, having correctness validation also will prevent regressions like opensearch-project/OpenSearch#15169. And I believe, we should be able to validate correctness with minimal effort without impacting the performance numbers. We can have static file containing expected response OR validate the correctness between baseline and candidate in addition to performance numbers.

What alternatives have you considered?

Ideally, integration and unit tests cover most of these scenarios, but it is impractical to have such large amounts of data as part of such tests

[IanHoang]

@jainankitk OSB comes with --enable-assertions flag which tells OSB you want to validate that responses from a operation meet a criteria. Ensure you are running the test with --enable-assertions and have defined asssertions in the workload for each operation you want to validate:

{
  "name": "term",
  "operation-type": "search",
  "assertions": [
    {
      "property": "hits",
      "condition": ">",
      "value": 0
    }
  ],
}

We will add documentation for this to the documentation website.

[jainankitk]

@IanHoang - This approach involves adding assertions for the whole response, for all the operations, which is very complex IMO. Can we have expected response file for each operations, so that the operation response can be compared with the expected response?

[jainankitk]

Also, that feature should be enabled by default and there should be option to disable that if needed