Use GenAI to make test scenarios

[sloorush]

Extremely chaotic but worth considering with how fast the new AI models run.

My idea is to use the OpenAI Package on Tatin(or any other LLM) and make a call to OpenAI(or similar) and ask it to give us some data that it thinks the APL primitive might break over. We should log everything this does.

I think it will give a lot of false positives but non-deterministic testing is a new way of doing things. Its on the same line as chaos testing.

Pros:

• Lots of new data
• Lots of uncommon scenarios that might give aplcores (or errors that are incorrect) because its gonna write something that does not work a lot of times
• Its a new thing ¯\(⍣)/¯
• (I think) We need more chaos testing

Cons:

• Expensive because it will need a subscription to an AI model (we can use an open source one like LLAMA3, its pretty good)
• Time taking- the time taken for each test will be increased (can be dealt with these only running with the slow tests)

[sloorush]

I got this idea from a project that makes non-deterministic testing for GenAI applications. But this is the other way round, I would appreciate more opinions on this if you think it will make a major difference...

[sloorush]

It comes from an ideology that randomly generated deterministic data might not be the best test data but AI generated random data that might be relevant and it will try to generate the best random data possible.

(Considering a world with next to no AI hallucinations)
(Hallucinations are not bad either because it will test for known errors) (?)

[sloorush]

This can be added as an addition to RunVariations where it checks what the primitive can do, and gives it the hardest expression it thinks the primitive can solve