Add websocket-based streaming chat to prototype #163
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR abandons the idea of a streaming langchain-js Lambda Function URL (which was proving to be both complicated and unreliable) in favor of using the python-based langchain to set up an API Gateway V2 Websocket API to handle chats.
It adds:
/chat
directory, which is deployed as a sub-stack of the main dc-api stack./chat-endpoint
route to the main API, which returns the URI of the above websocket API and an authentication token to pass to itAfter connecting to the websocket URI, the client can issue a message in the format:
The socket will respond with a number of messages containing the question, the context documents, a stream of tokens making up the answer, and finally the full answer. Each of these responses will include the
ref
provided in the request. This allows multiple requests to run simultaneously on the same socket connection without losing track of which response goes with which request.