Add Containerized Benchmarking Support for GuideLLM #123

wangchen615 · 2025-04-17T21:56:38Z

Add Containerized Benchmarking Support for GuideLLM

Description

This PR adds containerization support for GuideLLM benchmarking, making it easier to run benchmarks in containerized environments. The changes include:

A new Dockerfile that:
- Uses Python 3.11 slim as the base image
- Installs necessary dependencies, including git and curl
- Sets up a non-root user for security
- Includes healthcheck functionality
- Properly configures the benchmark script
A new run_benchmark.sh script that:
- Provides a flexible way to run benchmarks with configurable parameters
- Supports environment variable overrides for customization
- Handles different output formats (json, yaml)
- Includes proper error handling and logging

Testing

The container can be built and tested using:

docker build -t <container-image-repo>/guidellm-benchmark:latest .
docker run -e TARGET="http://localhost:8000" -e MODEL="neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16" <container-image-repo>/guidellm-benchmark:latest

Configuration Options

The container supports the following environment variables:

TARGET: The target endpoint for benchmarking (default: http://localhost:8000)
MODEL: The model to benchmark (default: neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16)
RATE_TYPE: The rate type for benchmarking (default: sweep)
DATA: The data configuration (default: prompt_tokens=256,output_tokens=128)
MAX_REQUESTS: Maximum number of requests (default: 100)
MAX_SECONDS: Maximum duration in seconds
OUTPUT_PATH: Path for output files (default: /results/guidellm_benchmark_results)
OUTPUT_FORMAT: Output format (json, yaml, or yml)

Security Considerations

Uses a non-root user for running the benchmark
Includes proper file permissions
Implements healthcheck for container monitoring
Follows container security best practices

Related Issues

Checklist

Added Dockerfile with proper security configurations
Added run_benchmark.sh script with environment variable support
Added documentation for configuration options
Tested container build and run
Followed security best practices

…arking

sjmonson · 2025-04-17T23:25:58Z

Could you put these files under a /build/ directory? Also why python 3.11 and not 3.13 since its the latest version we offer support for?

Dockerfile

run_benchmark.sh

sjmonson · 2025-04-21T14:12:40Z

Dockerfile

+# Healthcheck
+HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+


I assume this was an accidental inclusion since GuideLLM does not provide a web server.

Suggested change

# Healthcheck

HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \

CMD curl -f http://localhost:8000/health || exit 1

Yes, I remove this in the new build/Dockerfile.

sjmonson · 2025-04-21T14:25:19Z

Dockerfile

+WORKDIR /app
+
+# Install GuideLLM
+RUN pip install git+https://github.com/neuralmagic/guidellm.git


Rather than installing main, install our current HEAD. Also cp from within the container to avoid having two COPY layers.

Suggested change

RUN pip install git+https://github.com/neuralmagic/guidellm.git

COPY . /source

RUN pip install /source && \

cp /source/build/run_benchmark.sh /app/

sjmonson · 2025-04-21T14:36:22Z

run_benchmark.sh

If this script is going to be the ENTRYPOINT and not the CMD than it needs to be more flexible. Either add an extra arguments variable or append command line arguments to the command.

Yes, I allow it to add extra arguments.

…arking fix pre-commit issue? Add Docker build files and update .gitignore

markurtz

Let some comments throughout. Additionally, I know we had a request to move it to build/, but this is a commonly named folder that python and potential future builds will use.

I've generally seen the parent folder be docker/ at the root and then different python versions could be either nested folders under that with a Dockerfile or multiple Dockerfiles a the root with extensions of the python version such as Dockerfile.py39

markurtz · 2025-04-22T13:38:19Z

build/Dockerfile

@@ -0,0 +1,32 @@
+FROM python:3.12-slim


I think it would be better to have this at one of the extremes for the supported versions rather than somewhere in the middle, especially since the max and min version are tested much more; ie 3.9 or 3.13

markurtz · 2025-04-22T13:39:13Z

build/Dockerfile

+FROM python:3.12-slim
+
+LABEL org.opencontainers.image.source="https://github.com/neuralmagic/guidellm"
+LABEL org.opencontainers.image.description="GuideLLM Benchmark Container"


NIT: can we remove the benchmark from this so we keep it flexible towards future goals of evals and things like that?

markurtz · 2025-04-22T13:40:24Z

build/Dockerfile

+RUN apt-get update && apt-get install -y \
+    git \
+    curl \
+    && pip install git+https://github.com/neuralmagic/guidellm.git \


I'm concerned that this is too restrictive in that it always builds from the latest main, so it will be tough to specify what should be built here other than from the latest main either with the build system for a specific release or for a user.

Can we expand this with a build-arg for the package version to build? Which we can set as default to build from the latest main, but otherwise can utilize it to specify the release tag/branch to build from.

markurtz · 2025-04-22T14:55:49Z

build/Dockerfile

+    CMD curl -f http://localhost:8000/health || exit 1
+
+# Set the entrypoint
+ENTRYPOINT ["/app/run_benchmark.sh"]


Can we change this out so that we are setting the entrypoint to "guidellm" or "guidellm benchmark"? We can then set the CMD after to include any specific args we need passed in / surface from the settings or set it to --help for users to get started with. This way we can enable and passthrough and remove duplication of arguments we need to pass through and another script to maintain and add functionality to.

markurtz · 2025-04-22T14:56:24Z

build/run_benchmark.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash


See note on the entrypoint for the Dockerfile, I'd like to see if we can remove this script fully or make it optional to package in

markurtz · 2025-04-22T14:56:51Z

build/Dockerfile

@@ -0,0 +1,32 @@
+FROM python:3.12-slim


Also, what are our plans for supporting the range of python versions? Anything we should do here to create specific Dockerfiles for each supported python version?

Normally I would recommend handling this either though build args or postfixes on the Containerfile name. E.g.

Suggested change

FROM python:3.12-slim

ARG python_version=3.13

FROM python:${python_version}-slim

Or name this file something like Containerfile.python-3.13

wangchen615 added 2 commits April 17, 2025 21:49

Add Dockerfile and benchmark script for containerized GuideLLM benchm…

29aaf7b

…arking

fix pre-commit issue?

9ca1e8f

surajssd reviewed Apr 20, 2025

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

Dockerfile Outdated Show resolved Hide resolved

run_benchmark.sh Outdated Show resolved Hide resolved

sjmonson requested changes Apr 21, 2025

View reviewed changes

Add Dockerfile and benchmark script for containerized GuideLLM benchm…

2fd8c0c

…arking fix pre-commit issue? Add Docker build files and update .gitignore

wangchen615 force-pushed the main branch from 9ca1e8f to 2fd8c0c Compare April 22, 2025 03:52

wangchen615 added 2 commits April 22, 2025 04:02

Merge branch 'main' of https://github.com/wangchen615/guidellm

5abf27a

remote original dockerfile and run_benchmark.sh

661e8e4

wangchen615 requested a review from sjmonson April 22, 2025 04:08

markurtz requested changes Apr 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Containerized Benchmarking Support for GuideLLM #123

Add Containerized Benchmarking Support for GuideLLM #123

wangchen615 commented Apr 17, 2025 •

edited by sjmonson

Loading

sjmonson commented Apr 17, 2025

sjmonson Apr 21, 2025

wangchen615 Apr 22, 2025

sjmonson Apr 21, 2025

sjmonson Apr 21, 2025

wangchen615 Apr 22, 2025

markurtz left a comment

markurtz Apr 22, 2025

markurtz Apr 22, 2025

markurtz Apr 22, 2025

markurtz Apr 22, 2025

markurtz Apr 22, 2025

markurtz Apr 22, 2025

sjmonson Apr 22, 2025

	# Healthcheck
	HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
	CMD curl -f http://localhost:8000/health \|\| exit 1

-RUN pip install git+https://github.com/neuralmagic/guidellm.git
+COPY . /source
+RUN pip install /source && \
+         cp /source/build/run_benchmark.sh /app/

-FROM python:3.12-slim
+ARG python_version=3.13
+FROM python:${python_version}-slim

Add Containerized Benchmarking Support for GuideLLM #123

Are you sure you want to change the base?

Add Containerized Benchmarking Support for GuideLLM #123

Conversation

wangchen615 commented Apr 17, 2025 • edited by sjmonson Loading

Add Containerized Benchmarking Support for GuideLLM

Description

Testing

Configuration Options

Security Considerations

Related Issues

Checklist

sjmonson commented Apr 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markurtz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangchen615 commented Apr 17, 2025 •

edited by sjmonson

Loading