GuideLLM Project Roadmap

[markurtz]

This roadmap outlines the project's active planning, releases, and future development. It follows agile principles to allow for flexibility in prioritization and planning. This document will be updated periodically to reflect priorities, progress, and community feedback.

📊 Stack Rank

Active

The following features, improvements, and bug fixes are actively being worked on or are in the planning phase. Items are listed in priority order, and priorities are based on business value, user feedback, and technical debt reduction.

1. General Improvements

• General Multi Process Scheduler Implementation and Report Generation Updates
• Calculate "peak token throughput" number that is closer to vLLM generation throughput

2. Dataset Standardization & Synthetic Data – Define standard dataset “profiles” and determine if synthetic data generation is needed.

• Base enablement to enable a dataset configuration file to be passed in rather than standard args, allow alias's for the folder structure
• Text-dataset performance profile creation - decide what the profiles are
• Scope out and propose Community-oriented text dataset profiles for model validation

3. Batch Inference Handling – Improve support for batch inferencing within GuideLLM.

• Enable Batch Inferencing Benchmarking Support

4. Multi-Modal Model Support – Add benchmarking support for vision-language, image generation, and audio models.

• Multimodal Support

5. Concurrent User-Based Load Generation – Improve GuideLLM’s capability to simulate concurrent user-based load testing.

• [Rate Type] Concurrencies
• ^^ Landing in General Multi Process Scheduler Implementation and Report Generation Updates
• Max RPS in sweep mode could be higher

6. GuideLLM UI Enhancements – Improve visualization and data integration for inference benchmarking.

• Land GuideLLM UI in Upstream Repository

Backlog / Known Issues

The backlog contains tasks and known issues that have been identified but are not actively being worked on. These will be prioritized in future planning cycles.

1. LLM Hardware Acceleration + vLLM Metrics Benchmarking – Adding support for benchmarking inference acceleration using different hardware setups and provide more detail on vLLM specific scheduler metrics.

• [Scoping] Enable Hardware Accelerator Monitoring metrics to be captured in benchmarks
• Enable vLLM scheduler specific Metrics

2. Cost Estimation - Develop a cost estimation feature within the GuideLLM UI to provide insights into the cost per million tokens across different hardware configurations.

• TBD

3. UX Improvements

• Enable pass-through tags as an extra field to allow users to get custom metadata output
• Have GuideLLM kick off a vLLM server automatically to avoid having the user install vLLM and assign the target themselves
• Add flag to put guildellm into "non-interactive" mode

4. Accuracy Eval Integration

• Get Accuracy Metrics reported
• Integrate lm-eval into GuideLLM

🗂️ Releases

Planned

Upcoming/planned releases for the project are linked below for further details.

1. v0.2.0 – This release of GuideLLM focuses on general improvements, enhancing concurrency handling, and refining dataset standardization. Key updates include:
   • Multi-process scheduler improvements for better inference load balancing.
     • General Multi Process Scheduler Implementation and Report Generation Updates
   • Introduction of "peak token throughput" calculation for more precise performance tracking.
     • Calculate "peak token throughput" number that is closer to vLLM generation throughput
   • Enhanced dataset configuration and profiling for standardized benchmarking.
     • Base enablement to enable a dataset configuration file to be passed in rather than standard args
     • Text-dataset performance profile creation
     • Community-oriented text dataset profiles
   • Batch inference benchmarking support for optimized performance evaluation.
     • Enable Batch Inferencing Benchmarking Support
   • Multi-modal model support for benchmarking vision-language, image generation, and audio models.
     • Multimodal Support
   • Enhanced concurrent user-based load testing capabilities.
     • [Rate Type] Concurrencies
     • Max RPS in sweep mode could be higher
   • Project Board for v0.2.0

Previous

Previous releases for the project are linked below for further details.

• v0.1.0 - Initial release of GuideLLM with version 0.1.0! This core release adds the basic structure, infrastructure, and code for benchmarking LLM deployments across several different use cases utilizing a CLI interface and terminal output. Further improvements are coming soon!
  • https://github.com/neuralmagic/guidellm/releases/tag/v0.1.0

📅 Future Roadmap

3 Months

• Multi-Modal Model Support – Add benchmarking support for vision-language, image generation, and audio models

  • Multimodal Support

• GuideLLM UI Release for better data visualization and usability.

  • Land GuideLLM UI in Upstream Repository

6 Months

The following prioritized milestones are planned for development and release within the next six months.

• Cost Estimation - Develop a cost estimation feature within the GuideLLM UI to provide insights into the cost per million tokens across different hardware configurations.

  • TBD

• LLM Hardware Acceleration + vLLM Metrics Benchmarking – Adding support for benchmarking inference acceleration using different hardware setups and provide more detail on vLLM specific scheduler metrics.

  • [Scoping] Enable Hardware Accelerator Monitoring metrics to be captured in benchmarks
    • Enable vLLM scheduler specific Metrics

• UX Improvements

  • Enable pass-through tags as an extra field to allow users to get custom metadata output
  • Have GuideLLM kick off a vLLM server automatically to avoid having the user install vLLM and assign the target themselves
  • Add flag to put guildellm into "non-interactive" mode