Skip to content

feat: Ensemble async callback execution (rework) #438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

yinggeh
Copy link
Contributor

@yinggeh yinggeh commented May 14, 2025

What does the PR do?

Reduce e2e latency in ensemble model by executing callbacks asynchronously at the end of each ensemble step. Excluding models that require responses to have the same order of requests.

Improvement: maximum throughput of sample ensemble model increased from 39k infer/sec to 50k infer/sec.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • feat

Related PRs:

triton-inference-server/common#133
Previous PR: #429

Where should the reviewer start?

Reviewer should start from the second commit.
Attention to the preserve_responses_order logic.

Test plan:

L0_simple_ensemble
L0_sequence_batcher
L0_backend_python

  • CI Pipeline ID:
    28454142

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #7650

@yinggeh yinggeh self-assigned this May 14, 2025
@yinggeh yinggeh added the PR: feat A new feature label May 14, 2025
@yinggeh yinggeh requested review from tanmayv25, GuanLuo and ziqif-nv May 14, 2025 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: feat A new feature
Development

Successfully merging this pull request may close these issues.

1 participant