[Bug]: run-eval workflow is failing on git pull (due to fork?)

[neubig]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Describe the bug and reproduction steps

I tried to run the eval workflow on this PR (from a fork): #5220

It is failing with the following error:

Fetching the repository
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/add_thinking*:refs/remotes/origin/add_thinking* +refs/tags/add_thinking*:refs/tags/add_thinking*
  The process '/usr/bin/git' failed with exit code 1
  Waiting 19 seconds before trying again
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/add_thinking*:refs/remotes/origin/add_thinking* +refs/tags/add_thinking*:refs/tags/add_thinking*
  The process '/usr/bin/git' failed with exit code 1
  Waiting 16 seconds before trying again
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/add_thinking*:refs/remotes/origin/add_thinking* +refs/tags/add_thinking*:refs/tags/add_thinking*

I suspect that this might be due to the eval workflow not working on forks. @mamoodi it'd be great if you can investigate!

OpenHands Installation

Other

OpenHands Version

No response

Operating System

None

Logs, Errors, Screenshots, and Additional Context

No response

[mamoodi]

To comment, I'm aware of the issue. The fix needs a little pondering.

[mamoodi]

The reason is github doesn't allow access to secrets from outside the repo, which is understandable.

[mamoodi]

Might be a will not do for now. Will discuss with the team how to approach this.
For forks, the job can be run manually.

[enyst]

@mamoodi Can you please tell, what do you mean by manually, how does that work or what does it imply?

[mamoodi]

The remote job will have to be run manually by someone from the All Hands AI team if an eval needs to be run on a fork for now. Investigated a bit on how we can make this automatic and because secrets are not available in forks, it makes things hard.

Looking at the original eval-runner workflow, that one has secrets as well so that one also has the same problem.

[mamoodi]

Going to close this as will not do for now. Running evals automatically through labelling on forks is not supported.

[mamoodi]

Reopening since enyst says this might be possible with a github app.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.