-
Notifications
You must be signed in to change notification settings - Fork 104
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix state transitions for re-running requests #251
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kthui
previously approved these changes
Sep 1, 2023
nnshah1
reviewed
Sep 1, 2023
nnshah1
reviewed
Sep 1, 2023
…tion before Run()
nnshah1
reviewed
Sep 5, 2023
nnshah1
reviewed
Sep 5, 2023
nnshah1
approved these changes
Sep 5, 2023
kthui
approved these changes
Sep 5, 2023
rmccorm4
changed the title
Allow transitioning from RELEASED to STARTED state for re-using requests
Fix state transitions for re-running requests
Sep 6, 2023
Tabrizian
added a commit
that referenced
this pull request
Sep 7, 2023
* Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <[email protected]>
Tabrizian
added a commit
that referenced
this pull request
Sep 13, 2023
* Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <[email protected]>
Tabrizian
added a commit
that referenced
this pull request
Sep 13, 2023
* Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <[email protected]>
Tabrizian
added a commit
that referenced
this pull request
Sep 13, 2023
* Add inference request cancellation APIs (#249) * Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <[email protected]> * Fix request re-use when cancelling a request * Review edit * Fix warmup request * Fix null requests * Review edit --------- Co-authored-by: Ryan McCormick <[email protected]>
tanmayv25
added a commit
that referenced
this pull request
Oct 4, 2023
* Add inference request cancellation APIs (#249) * Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <[email protected]> * Fix request re-use when cancelling a request (#253) * Add inference request cancellation APIs (#249) * Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <[email protected]> * Fix request re-use when cancelling a request * Review edit * Fix warmup request * Fix null requests * Review edit --------- Co-authored-by: Ryan McCormick <[email protected]> * Dynamic batch scheduler request cancellation (#257) * Add adapter for IsCancelled() * Move cancel functions outside TRITON_ENABLE_STATS * Add request cancellation to dynamic batch scheduler * Refactor cancelled requests to use rejected requests routine * Remove shared pointer wrapper for rejected and cancelled requests * Ensemble scheduler request cancellation (#263) * Add adapter for IsCancelled() * Move cancel functions outside TRITON_ENABLE_STATS * Add request cancellation to ensemble scheduler * Sequence batch scheduler request cancellation (#260) * Add request cancellation to sequence batch scheduler * Add request cancellation to in-flight sequences * Refactor on how a request is cancelled * Fix issue when request is timeout dummy * Always mark request cancelled before cancelling * Mark immutable static status as const --------- Co-authored-by: Iman Tabrizian <[email protected]> Co-authored-by: Ryan McCormick <[email protected]> Co-authored-by: Jacky <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes L0_simple_lib--base, and allows requests to be re-used for multiple inferences.
Adds a sanity check that pending count isn't incremented multiple times without getting decremented in some edge case where a request is PENDING, fails to get to the PENDING state, and ends up in PENDING state again (re-used).
General logic:
If a request were to be transitioned from RELEASED to PENDING/EXECUTING directly, this should indicate a logical error as the request should only execute via InferAsync entrypoint, which calls PrepareForInference ([re]sets to INITIALIZED state) then InferenceRequest::Run (sets to PENDING state).