Feature/maestro ai

[bartekpacia]

#1222 (comment)

Intro

This PR adds 2 new commands: assertWithAI and assertNoDefectsWithAI.

Documentation PR here. Read it for intro to these 2 commands.

Notes

• at least for now, we don't consider AI output to be part of the test results. It's an additional stream of information, just like logs or screenshots. Reason: we're not sure of relationship of AI output with e.g. JUnit - how it should be serialized into that format.

  Data structure of AI output

  flow 1 \
     assertVisualAI 1
       screenshot
         - defect 1
         - defect 2
         - defect 3
  flow 2 \
     assertVisualAI 1
       screenshot
         - defect 1
         - defect 2
         - defect 3
     assertVisualAI 2
       screenshot
         - defect 1
         - defect 2
         - defect 3
  
  

• In the longer run, I'd like to integrate normal test results with AI test results (integrate meaning e.g. link to each other from HTML reports. Or, preferably, maybe merge their HTML reports)

To test

• works in Maestro Studio
• works in Maestro Cloud todo in the near future
• works in continuous mode

Off-top things noticed

Output directories

• Maestro test output + AI output should be placed in the root directory of the project. This will allow to use relative paths in e.g. test outputs. Benefit: possible to move files out of CI without breaking paths.
• Maestro debug output can stay where it is right now.

coroutines

Maestro uses a lot of runBlocking { }, which sometimes is justified, but often it's a bad practice.

I think most of code in maestro-client and maestro-orchestra (Maestro, Driver) should be using suspending functions.

[amanjeetsingh150]

What should be the expected output? would be helpful to document that as well

[bartekpacia]

Added expected output.

Also, I think we should add a link where the fixtures can be downloaded from. Otherwise the barrier entry to this DemoApp is very high, because of the need to take the screenshots manually.

Currently the fixtures dataset I use is from https://github.com/mobile-dev-inc/copilot/pull/188.

I will upload that dataset to GCP and paste the link here. Do you see any problem with it? I think the customer apps that are in https://github.com/mobile-dev-inc/copilot/pull/188 are okay with that?

[amanjeetsingh150]

Lets not public the GCS link for now, you can make it work for any storage link. Lets write a runbook internally for this in case some one from us wants to evaluate. This repository should ideally have access to general data (from open source apps) instead of customer apps.

[bartekpacia]

Good points, though I don't agree.

I'd prefer to have this dataset public, internal runbooks tend to rot and no one uses them unless really needed. But if we made this testing dataset public, anybody could play around with LLM outptus and submit a PR that improves it.

If this requires removing apps from our customers, and using screenshots from more popular, well known apps instead (like Uber, Bolt), then I still think it's worth the effort. We should strive to make working on Maestro possible and easy to people outside of @mobile-dev-inc.

PS 1 The perfect situation would be that generating screenshots for each app with takeScreenshot. But I think it's too much work to automate this, and what we have now is enough.

[bartekpacia]

That said: I will take no action for now.

[amanjeetsingh150]

Better to create issue as well after release!

[bartekpacia]

Created #1972 to track this