Benchmark runner instantiates annotators #510
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| @@ -36,26 +36,35 @@ def fake_all_secrets(value="some-value") -> RawSecrets: | |||
| return raw_secrets | |||
|
|
|||
|
|
|||
| class FakeExplodingAnnotator(FakeAnnotator): | |||
There was a problem hiding this comment.
This pollutes a global variable with test-specific data, and as a general rule tests should not have side effects.
There was a problem hiding this comment.
Are you referring to the registration of the fake annotators? If so, I agree that it's problematic. However, tests' get_annotators just returns UIDs and then the runners get the actual annotators from the registry now.
Do you have suggestions to avoid test side effects on ANNOTATORS while still testing the runners' annotator-specific functionality?
There was a problem hiding this comment.
In general, I prefer to avoid global variables, which neatly solves this problem. But since fixing that is probably out of scope for this, I think the next-best solution is either setup/teardown (which add the fakes and then restore state after the test has run) or mocking such that the test diverts the production code so that it uses a fake registry which is discarded after the test has run.
There was a problem hiding this comment.
That makes sense! Updated.
PR #487 did not apply its test annotator instantiation change to modelbench's runner. This fixes that.
I also snuck in a bug fix for the "sxc" prompt issue that was causing some tests to fail.