Enhancement: Add Support for Conversational Continuity Detection Using finish_reason

[RodrigoX1]

What features would you like to see added?

Background

In the context of using the OpenAI API for conversational applications, it's crucial to determine when a response from the API indicates that the conversation should continue. The finish_reason field provided in the API response helps identify whether the response was truncated, blocked, or completed naturally.

Proposed Improvement

I propose adding functionality to detect the finish_reason from the OpenAI API response and handle each case appropriately. This would enhance the user experience by ensuring that conversations continue smoothly and that users are informed when a response is incomplete or blocked.

Examples of finish_reason in OpenAI API Responses

1. stop Finish Reason

The text generation was completed because the API encountered one of the stop tokens provided or reached the end of the prompt.

{
  "id": "cmpl-6rTtSuVw2GHrJH2B3hzdsjdhsuad",
  "object": "text_completion",
  "created": 1658492036,
  "model": "gpt-4o",
  "choices": [
    {
      "text": "Gravity is a force that attracts two bodies towards each other. It is responsible for the falling of objects to the ground.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 7,
    "completion_tokens": 22,
    "total_tokens": 29
  }
}

The finish_reason is "stop" because the API completed the response as intended.

2. length Finish Reason

The text generation was cut off because it reached the maximum token limit for the response.

{
  "id": "cmpl-6rTtSuVw2GHrJH2B3hzdsjdhsuad",
  "object": "text_completion",
  "created": 1658492036,
  "model": "gpt-4o",
  "choices": [
    {
      "text": "The theory of relativity, developed by Albert Einstein, describes the laws of physics in",
      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 10,
    "total_tokens": 16
  }
}

The finish_reason is "length" because the response was truncated when it hit the token limit.

3. content_filter Finish Reason

The text generation was stopped by OpenAI's content filter because the generated text violated safety and use guidelines.

{
  "id": "cmpl-6rTtSuVw2GHrJH2B3hzdsjdhsuad",
  "object": "text_completion",
  "created": 1658492036,
  "model": "gpt-4o",
  "choices": [
    {
      "text": "",
      "index": 0,
      "logprobs": null,
      "finish_reason": "content_filter"
    }
  ],
  "usage": {
    "prompt_tokens": 3,
    "completion_tokens": 0,
    "total_tokens": 3
  }
}

The finish_reason is "content_filter" because the content filter detected potentially harmful or inappropriate content.

4. null Finish Reason

No specific reason was provided for the completion of the response. This might be used in cases where the response is terminated for unspecified reasons.

{
  "id": "cmpl-6rTtSuVw2GHrJH2B3hzdsjdhsuad",
  "object": "text_completion",
  "created": 1658492036,
  "model": "gpt-4o",
  "choices": [
    {
      "text": "",
      "index": 0,
      "logprobs": null,
      "finish_reason": "null"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 0,
    "total_tokens": 10
  }
}

The finish_reason is "null" because there wasn't a clear reason for the termination of the response.

Finish Reason	Description	Example Text
stop	The text generation was completed because a stop token was encountered or the prompt ended.	"The response is complete and successfully concluded."
length	The text generation was cut off because the token limit was reached.	"This is a very long response and was cut off because it reached the maximum token limit..."
content_filter	The text generation was stopped by OpenAI's content filter.	"This text was interrupted because the generated content violated the API use policies."
null	No specific reason was provided for the completion.	"The response was finished without a specific reason provided."

More details

Example Code

Here is a sample implementation of the proposed functionality in Python:

import openai
import time

class ConversationAgent:
    def __init__(self, api_key, model="gpt-4-0613"):
        self.client = openai.OpenAI(api_key=api_key)
        self.model = model

    def continue_conversation(self, conversation, next_prompt, max_tokens=150):
        try:
            response = self.client.chat.completions.create(
                model=self.model,
                messages=conversation,
                temperature=1,
                stream=True,   
                max_tokens=max_tokens
            )
        except Exception as e:
            print(f"Error creating chat completion: {e}")
            return ""

        message_content = ""
        finish_reason = None

        try:
            for chunk in response:
                if chunk.choices:
                    choice = chunk.choices[0]
                    if choice.delta:
                        if choice.delta.content:
                            message_content += choice.delta.content
                    if choice.finish_reason:
                        finish_reason = choice.finish_reason 
        except Exception as e:
            print(f"Error processing response chunks: {e}")
            return ""

        print(f"Finish reason: {finish_reason}")
      
        # Check if the response is incomplete
        if finish_reason == 'length':
            print("Incomplete response, pausing before continuing...")
            time.sleep(5)  # Pause for 5 seconds
            conversation.append({'role': 'user', 'content': next_prompt})

            try:
                next_response = self.client.chat.completions.create(
                    model=self.model,
                    messages=conversation,
                    temperature=1,
                    stream=True, 
                    max_tokens=max_tokens
                )
            except Exception as e:
                print(f"Error creating chat completion for next response: {e}")
                return ""

            next_message_content = ""
            try:
                for chunk in next_response:
                    if chunk.choices:
                        choice = chunk.choices[0]
                        if choice.delta and choice.delta.content:
                            next_message_content += choice.delta.content                            
                        if choice.finish_reason:
                            finish_reason = choice.finish_reason
                            print(f"Updating finish_reason: {finish_reason}")
            except Exception as e:
                print(f"Error processing next response chunks: {e}")
                return ""

            return next_message_content

        return message_content


api_key = "OPENAI-API-KEY"  

agent = ConversationAgent(api_key)
conversation = [{'role': 'user', 'content': 'A very long conversation...'}]
next_prompt = 'The next part of the conversation...'
print(agent.continue_conversation(conversation, next_prompt, max_tokens=150))



### Which components are impacted by your request?

_No response_

### Pictures

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

[danny-avila]

Finish reason is partly used already, primarily for triggering the "Continue" button.

LibreChat/client/src/hooks/useGenerationsByLatest.ts

Line 19 in 270c6d2

const { error, messageId, searchResult, finish_reason, isCreatedByUser } = message ?? {};

LibreChat/api/app/clients/OpenAIClient.js

Line 661 in 270c6d2

const { finish_reason } = streamResult.choices[0];

I think the easiest implementation would be provide the metadata via hover somewhere on messages if finish_reason is found (which isn't always and especially across providers)