Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop containers on interrupts #26

Merged
merged 2 commits into from
Apr 17, 2024
Merged

Stop containers on interrupts #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
728500d
docker_shell: Implement SIGTERM handler removing Docker containers
eszpotanski Apr 16, 2024
6e1428a
openroad.bzl: Add supports-graceful-termination tag for Docker flow t…
eszpotanski Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions docker_shell.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,17 @@
#!/usr/bin/env bash

set -ex
uuid=$(uuidgen)

function handle_sigterm() {
# Wait if container is not created
if [[ ! "$( docker container inspect -f '{{.State.Status}}' bazel-orfs-$uuid )" =~ ^(running|created)$ ]]; then
sleep 3s
fi
# Stop or remove container
docker container rm -f "bazel-orfs-$uuid" || true
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rework so that the handler is installed only when the container is running?


#!/bin/bash

# Function to handle SIGTERM signal
cleanup() {
    echo "Stopping container..."
    # Command to stop the container gracefully
    docker stop <container_name_or_id>
    exit 0
}

# Main
echo "Starting container..."
# Command to start your container, for example:
# docker run -d --name <container_name> <image_name>
docker run -d --name my_container my_image

# Wait until the container is either running or stopped
while true; do
    container_status=$(docker inspect -f '{{.State.Status}}' my_container)
    if [[ "$container_status" == "running" || "$container_status" == "exited" ]]; then
        break
    fi
    sleep 1
done

# Registering the cleanup function to handle SIGTERM
trap cleanup SIGTERM

# Wait for the script to be terminated
echo "Script is running. Press Ctrl+C to stop the container gracefully."
while true; do
    sleep 1
done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, SIGTERM can be received after docker run and before trap, then signal won't be handled, but container will be at least created.

I recreated this approach and tested it on BoomTile_synth, where a lot of containers are created, and usually at least one created container remained, but all running containers were stopped and removed.

Current solution stops and removes all containers created during bazel build.

If you prefer to add handler only when container is running and possibly not remove all created containers, I will commit these changes.

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

WORKSPACE_ROOT=$(pwd)/../..
Expand Down Expand Up @@ -47,9 +58,14 @@ export STAGE_CONFIG_PREFIXED=$WORKSPACE_EXECROOT/$STAGE_CONFIG
# Make bazel-bin writable
chmod -R +w $WORKSPACE_EXECROOT/bazel-out/k8-fastbuild/bin

# Handle TERM signals
# this option requires `supports-graceful-termination` tag in Bazel rule
trap handle_sigterm SIGTERM

# Most of these options below has to do with allowing to
# run the OpenROAD GUI from within Docker.
docker run --rm -u $(id -u ${USER}):$(id -g ${USER}) \
docker run --name "bazel-orfs-$uuid" --rm \
-u $(id -u ${USER}):$(id -g ${USER}) \
-e LIBGL_ALWAYS_SOFTWARE=1 \
-e "QT_X11_NO_MITSHM=1" \
-e XDG_RUNTIME_DIR=/tmp/xdg-run \
Expand All @@ -76,4 +92,9 @@ docker run --rm -u $(id -u ${USER}):$(id -g ${USER}) \
. ./env.sh
cd \$FLOW_HOME
$ARGUMENTS
"
" &

# Wait for Docker container to finish
# Docker container has to be run in subprocess,
# otherwise signal will not be handled immediately
wait $!
3 changes: 3 additions & 0 deletions openroad.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,7 @@ def mock_area_stages(
([name + "_synth_mock_area"] if stage == "floorplan" else []),
cmd = get_entrypoint_cmd(make_pattern, design_config, stage_config, True, make_targets, docker_image = docker_image, mock_area = (stage == "floorplan")),
outs = [s.replace("/" + variant + "/", "/mock_area/") for s in outs.get(stage, [])],
tags = ["supports-graceful-termination"],
)

def init_stage_dict(all_stage_names, stage_dict_init):
Expand Down Expand Up @@ -692,6 +693,7 @@ def build_openroad(
srcs = [Label("//:scripts/mem_dump.py"), Label("//:scripts/mem_dump.tcl"), target_name + "_clock_period", design_config, stage_config, make_pattern],
cmd = get_entrypoint_cmd(make_pattern, design_config, stage_config, True, "memory", docker_image = docker_image),
outs = outs["memory"],
tags = ["supports-graceful-termination"],
)

# Stage (Docker) targets
Expand Down Expand Up @@ -721,4 +723,5 @@ def build_openroad(
([name + target_ext + "_generate_abstract_mock_area"] if mock_area != None and stage == "generate_abstract" else []),
cmd = get_entrypoint_cmd(make_pattern, design_config, stage_config, True, make_targets, docker_image = docker_image),
outs = outs.get(stage, []),
tags = ["supports-graceful-termination"],
)