Add eval workflow that triggers remote eval job

[mamoodi]

End-user friendly description of the problem this fixes or functionality that this introduces

• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

---

Give a summary of what the PR does, explaining any non-trivial design decisions

This workflow triggers a remote eval job that runs an evaluation of 1 instance for now. This is to get the basic setup working.

---

Link of any specific issues this addresses

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:b2e8440-nikolaik   --name openhands-app-b2e8440   docker.all-hands.dev/all-hands-ai/openhands:b2e8440

[mamoodi]

I'm working on PR comments for start and finish....

[github-actions]

Running evaluation on the PR. Once eval is done, the results will be posted.

[openhands-agent]

Evaluation results: ## Summary

• submitted instances: 1
• empty patch instances: 0
• resolved instances: 1
• unresolved instances: 0
• error instances: 0

[rbren]

LGTM!

@enyst we will definitely continue enhancing this (e.g. with a link to download the full report)--lmk if you have lingering concerns

[github-actions]

Running evaluation on the PR. Once eval is done, the results will be posted.

[enyst]

OK, thank you. Yes, I think we can do this better, but I really appreciate the work on this @mamoodi <3

I started one to see if it works for me. I would suggest to merge if it ends successfully.

[rbren]

@enyst we currently have it limited to just me, Graham, Xingyao, and Mahmoud, given the potential to accidentally spend several hundred dollars 😅

[github-actions]

Running evaluation on the PR. Once eval is done, the results will be posted.

[rbren]

Maybe we can have tiers here, something like:

• anyone can run an xs
• maintainers can run s or m
• admins can run large

[enyst]

I think we definitely need something, because if anyone wants to run stuff in private, one can always run stuff in private. But if this workflow is defined here on this repo, then I do think it should be useful somehow here.

[enyst]

To clarify: I already talked to mamoodi before and we agreed that from my perspective, this PR is mergeable as it is. Not because it's entirely ready, but we know what we need to think about. 😹

Sorry, I did want to see if it works. 😅

I think Haiku is actually relatively expensive, after all we changed from deepseek where the difference is very significant.

I'll give some thought to this:

anyone can run an xs
maintainers can run s or m
admins can run large

Edited to add:
I'll go ahead and merge, because it's just a workflow... But I think we kinda should follow up soon with some decision IMHO. Because I think this is unique: a set of labels (run-eval-) that are useless to maintainers/collaborators who can set them. It feels bugged. 😀