Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(runtime): use micromamba instead of mamba and fix build issue #4154

Merged
merged 31 commits into from
Oct 2, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ffa94d9
use micromamba instead of mamba for effeciency
xingyaoww Oct 1, 2024
dfbc062
(debug) ls
xingyaoww Oct 1, 2024
33c5494
(debug) try directly using install script (likely more robust long term)
xingyaoww Oct 1, 2024
374e1ac
fix test runtime
xingyaoww Oct 1, 2024
cf13e56
fix test runtime build
xingyaoww Oct 1, 2024
156a679
conda forge yes
xingyaoww Oct 1, 2024
a95245c
tweak conda forge to no
xingyaoww Oct 1, 2024
ef76b1e
tweak to install py together with poetry & set forge_yes to true again
xingyaoww Oct 1, 2024
d883679
re-order env var
xingyaoww Oct 1, 2024
23d755d
always remove commercial channel 'defaults'
xingyaoww Oct 2, 2024
dc8db3f
try log stderrr
xingyaoww Oct 2, 2024
c68efe3
remove python unbufferred to fix remote runtime
xingyaoww Oct 2, 2024
f10f98a
add more printing
xingyaoww Oct 2, 2024
7d0fa1d
fix print
xingyaoww Oct 2, 2024
08b042e
fix runtime build tests
xingyaoww Oct 2, 2024
b778f44
fix test
xingyaoww Oct 2, 2024
6999e90
update CalledProcessError
xingyaoww Oct 2, 2024
334389a
add test rerun
xingyaoww Oct 2, 2024
cdabea0
attempt to fix ci by set n=1
xingyaoww Oct 2, 2024
f959421
Revert "attempt to fix ci by set n=1"
xingyaoww Oct 2, 2024
d732d3b
force rebuild for browsergym test
xingyaoww Oct 2, 2024
e04c84e
try print stderr
xingyaoww Oct 2, 2024
3fbff15
force rebuild for dummy agent
xingyaoww Oct 2, 2024
bc39c1d
include stdout/stderr for failed test
xingyaoww Oct 2, 2024
33c5238
always disable ps1
xingyaoww Oct 2, 2024
6330b52
tweak pytests
xingyaoww Oct 2, 2024
8a9033d
install curl too
xingyaoww Oct 2, 2024
811ba82
fix runtime build tests
xingyaoww Oct 2, 2024
61c6ec9
Revert "try print stderr"
xingyaoww Oct 2, 2024
42157a9
Revert "tweak pytests"
xingyaoww Oct 2, 2024
099aec9
Merge branch 'main' into xw/micromamba
xingyaoww Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion openhands/runtime/client/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ def _init_container(
container = self.docker_client.containers.run(
self.runtime_container_image,
command=(
f'/openhands/miniforge3/bin/mamba run --no-capture-output -n base '
f'/openhands/micromamba/bin/micromamba run -n openhands '
f'poetry run '
f'python -u -m openhands.runtime.client.client {self._container_port} '
f'--working-dir "{sandbox_workspace_dir}" '
Expand Down
3 changes: 2 additions & 1 deletion openhands/runtime/plugins/jupyter/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ async def initialize(self, username: str, kernel_id: str = 'openhands-default'):
'cd /openhands/code\n'
'export POETRY_VIRTUALENVS_PATH=/openhands/poetry;\n'
'export PYTHONPATH=/openhands/code:$PYTHONPATH;\n'
'/openhands/miniforge3/bin/mamba run -n base '
'export MAMBA_ROOT_PREFIX=/openhands/micromamba;\n'
'/openhands/micromamba/bin/micromamba run -n openhands '
'poetry run jupyter kernelgateway '
'--KernelGatewayApp.ip=0.0.0.0 '
f'--KernelGatewayApp.port={self.kernel_gateway_port}\n'
Expand Down
2 changes: 1 addition & 1 deletion openhands/runtime/remote/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ def __init__(
start_request = {
'image': self.container_image,
'command': (
f'/openhands/miniforge3/bin/mamba run --no-capture-output -n base '
f'/openhands/micromamba/bin/micromamba run -n openhands '
'PYTHONUNBUFFERED=1 poetry run '
f'python -u -m openhands.runtime.client.client {self.port} '
f'--working-dir {self.config.workspace_mount_path_in_sandbox} '
Expand Down
33 changes: 16 additions & 17 deletions openhands/runtime/utils/runtime_templates/Dockerfile.j2
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,15 @@ RUN mkdir -p /openhands && \
# Directory containing subdirectories for virtual environment.
ENV POETRY_VIRTUALENVS_PATH=/openhands/poetry

RUN if [ ! -d /openhands/miniforge3 ]; then \
wget --progress=bar:force -O Miniforge3.sh "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" && \
bash Miniforge3.sh -b -p /openhands/miniforge3 && \
rm Miniforge3.sh && \
chmod -R g+w /openhands/miniforge3 && \
bash -c ". /openhands/miniforge3/etc/profile.d/conda.sh && conda config --set changeps1 False && conda config --append channels conda-forge"; \
fi
# Install micromamba
RUN mkdir -p /openhands/micromamba/bin && \
/bin/bash -c "PREFIX_LOCATION=/openhands/micromamba BIN_FOLDER=/openhands/micromamba/bin INIT_YES=no CONDA_FORGE_YES=no $(curl -L https://micro.mamba.pm/install.sh)"

# Set up micromamba
ENV MAMBA_ROOT_PREFIX=/openhands/micromamba
RUN /openhands/micromamba/bin/micromamba create -n openhands python=3.11 -y && \
/openhands/micromamba/bin/micromamba install -n openhands conda-forge::poetry -y

# Install Python and Poetry
RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y
# ================================================================
# END: Build Runtime Image from Scratch
# ================================================================
Expand All @@ -60,26 +59,26 @@ COPY ./code /openhands/code
WORKDIR /openhands/code
RUN \
# Configure Poetry and create virtual environment
/openhands/miniforge3/bin/mamba run -n base poetry config virtualenvs.path /openhands/poetry && \
/openhands/miniforge3/bin/mamba run -n base poetry env use python3.11 && \
/openhands/micromamba/bin/micromamba run -n openhands poetry config virtualenvs.path /openhands/poetry && \
/openhands/micromamba/bin/micromamba run -n openhands poetry env use python3.11 && \
# Install project dependencies
/openhands/miniforge3/bin/mamba run -n base poetry install --only main,runtime --no-interaction --no-root && \
/openhands/micromamba/bin/micromamba run -n openhands poetry install --only main,runtime --no-interaction --no-root && \
# Update and install additional tools
apt-get update && \
/openhands/miniforge3/bin/mamba run -n base poetry run pip install playwright && \
/openhands/miniforge3/bin/mamba run -n base poetry run playwright install --with-deps chromium && \
/openhands/micromamba/bin/micromamba run -n openhands poetry run pip install playwright && \
/openhands/micromamba/bin/micromamba run -n openhands poetry run playwright install --with-deps chromium && \
# Set environment variables
echo "OH_INTERPRETER_PATH=$(/openhands/miniforge3/bin/mamba run -n base poetry run python -c "import sys; print(sys.executable)")" >> /etc/environment && \
echo "OH_INTERPRETER_PATH=$(/openhands/micromamba/bin/micromamba run -n openhands poetry run python -c "import sys; print(sys.executable)")" >> /etc/environment && \
# Install extra dependencies if specified
{{ extra_deps }} {% if extra_deps %} && {% endif %} \
# Clear caches
/openhands/miniforge3/bin/mamba run -n base poetry cache clear --all . && \
/openhands/micromamba/bin/micromamba run -n openhands poetry cache clear --all . && \
# Set permissions
{% if not skip_init %}chmod -R g+rws /openhands/poetry && {% endif %} \
mkdir -p /openhands/workspace && chmod -R g+rws,o+rw /openhands/workspace && \
# Clean up
apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
/openhands/miniforge3/bin/mamba clean --all
/openhands/micromamba/bin/micromamba clean --all
# ================================================================
# END: Copy Project and Install/Update Dependencies
# ================================================================
2 changes: 1 addition & 1 deletion tests/runtime/test_browsing.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# Browsing tests
# ============================================================================================================================

PY3_FOR_TESTING = '/openhands/miniforge3/bin/mamba run -n base python3'
PY3_FOR_TESTING = '/openhands/micromamba/bin/micromamba run -n openhands python3'


def test_simple_browse(temp_dir, box_class, run_as_openhands):
Expand Down
20 changes: 7 additions & 13 deletions tests/unit/test_runtime_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -156,15 +156,13 @@ def test_generate_dockerfile_scratch():
assert base_image in dockerfile_content
assert 'apt-get update' in dockerfile_content
assert 'apt-get install -y wget sudo apt-utils' in dockerfile_content
assert (
'RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y'
in dockerfile_content
)
assert 'conda-forge::poetry' in dockerfile_content
assert 'python=3.11' in dockerfile_content

# Check the update command
assert 'COPY ./code /openhands/code' in dockerfile_content
assert (
'/openhands/miniforge3/bin/mamba run -n base poetry install'
'/openhands/micromamba/bin/micromamba run -n openhands poetry install'
in dockerfile_content
)

Expand All @@ -178,17 +176,13 @@ def test_generate_dockerfile_skip_init():

# These commands SHOULD NOT include in the dockerfile if skip_init is True
assert 'RUN apt update && apt install -y wget sudo' not in dockerfile_content
assert (
'RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y'
not in dockerfile_content
)
assert 'conda-forge::poetry' not in dockerfile_content
assert 'python=3.11' not in dockerfile_content
assert 'https://micro.mamba.pm/install.sh' not in dockerfile_content

# These update commands SHOULD still in the dockerfile
assert 'COPY ./code /openhands/code' in dockerfile_content
assert (
'/openhands/miniforge3/bin/mamba run -n base poetry install'
in dockerfile_content
)
assert 'poetry install' in dockerfile_content


def test_get_runtime_image_repo_and_tag_eventstream():
Expand Down