AI QA Agent for GitHub.

Website | Join our Discord

Manual testing is reptitive and slow. Automated tests take siginifigant investment time and need to be maintained. The TestDriver QA Agent can test your app with natural language prompts and AI vision, which is quicker to set up and more resiliant to changes.

• No code or frameworks to invest in
• Describe test steps with nautural lanuage
• Resiliant to changes in color, positioning, etc
• Can interpret objectives that get in the way of the goal
• High powered Silicon Mac M1 VM
• Supports full stack applications (backend and frontend) as well as desktop app
• Records video and logs of test results (powered by Dashcam.io)
  • Desktop Video Replay
  • AI Logs
  • Chrome Developer Console
  • Chrome Network Requests
  • Universal Logfile Ingestion

1. focus the Wave application with Spotlight
2. click "Continue"
3. focus the Wave input with the keyboard shorcut Command + I
4. type 'ls' into the input
5. press return
6. validate Wave shows the result of 'ls'

TestDriver is a productized infrastructure pipeline created by Dashcam.io. In short, this is how it works:

1. Spawn a Mac1 VM
2. Clone your repository (optional)
3. Runs prerun.sh
4. Spawns AI Agent with prompt
5. Reads step
6. Looks at screen, reads text and describes images
7. Determines what actions it needs to take to reach goal of prompt step
8. Executes actions
9. Agent summarizes results

This is an example workflow that Wave Terminal uses to test their electron application nightly and on every feature branch and send the results to Slack.

name: TestDriver.ai

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]
  workflow_dispatch:

jobs:
  test:
    name: "TestDriver"
    runs-on: ubuntu-latest
    steps:
      - uses: replayableio/testdriver-action@main
        id: testdriver
        with:
          prompt: |
            1. focus the Wave application with Spotlight
            2. click "Continue"
            3. focus the Wave input with the keyboard shorcut Command + I
            4. type 'ls' into the input
            5. press return
            6. validate Wave shows the result of 'ls'
      - name: Send custom JSON data to Slack workflow
        id: slack
        if: ${{ always() }}
        uses: slackapi/[email protected]
        with:
          # This data can be any valid JSON from a previous step in the GitHub Action
          payload: |
            {
              "link": "${{ steps.testdriver.outputs.link }}",
              "summary": ${{ toJSON(steps.testdriver.outputs.summary)}}
            }
        env:
          SLACK_WEBHOOK_URL: "https://hooks.slack.com/triggers/xxx/yyy/zzz"

TestDriver will look for a script in ./testdriver/prerun.sh and execute this before the AI prompt.

npm install dashcam-chrome --save
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --load-extension=./node_modules/dashcam-chrome/build/ 1>/dev/null 2>&1 &
exit

brew install go
brew tap scripthaus-dev/scripthaus
brew install scripthaus
npm install -g yarn
mkdir ~/build
cd ~/build
git clone https://github.com/wavetermdev/waveterm.git
cd waveterm
scripthaus run build-backend
echo "Yarn"
yarn
echo "Rebuild"
scripthaus run electron-rebuild
echo "Webpack"
scripthaus run webpack-build
echo "Starting Electron"
scripthaus run electron 1>/dev/null 2>&1 &
echo "Electron Done"
exit