[Bug]: Error in certain Docker images for SWE-Bench instance evaluations

[sani903]

Is there an existing issue for the same bug?

• I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
• I have checked the existing issues.

Describe the bug

When initializing the environment for instance-level SWE-Bench Docker images, the code to go to the desired directory as base throws an error. It might be related to swe-bench-instance.json. More specifically, for certain instances from repositories like pytest, matplotlib, sphinx, etc. the directory name to change directory to might not be constructed properly with an error concatenating the repo name and its version which leads to an error in making the required directory in line 31 of evaluation/swe_bench/scripts/setup/instance_swe_entry.sh

Some instances which throw this error:

• matplotlib__matplotlib-23314
• sphinx-doc__sphinx-7757
• scikit-learn__scikit-learn-9288
• pytest-dev__pytest-7432
• pytest-dev__pytest-5262
• matplotlib__matplotlib-26466

From all the instances of django and sympy that I have tested, none throw this error.

Current OpenHands version

0.9.7

Installation and Configuration

git clone
make build

Model and Agent

No response

Operating System

Linux

Reproduction Steps

1. Create config.toml in evaluation/swe_bench and add the instance IDs mentioned as "selected_ids = [ "<error_instance_ids> "]
2. Run ./evaluation/swe_bench/scripts/run_infer.sh <llm.config>

Logs, Errors, Screenshots, and Additional Context

[neubig]

@xingyaoww it'd be great if you could take a look

[xingyaoww]

Haven't test it yet (will do soon) - but this should fix it: #4241

[xingyaoww]

I tested #4241 on matplotlib__matplotlib-23314 which is working. I'll close this for now - @sani903 lmk if this works for you, feel free to re-open if not!