Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for message history to be omitted when making request to llm #1468

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

DarioPanada
Copy link

Feature: Add support for appending messages to existing conversations while providing the option to include only the latest message in the conversation history.

Change: Introduces an optional boolean flag, exclude_history. When set to true, only the most recent message in the conversation is passed instead of the complete history.

Benefits:

  1. Reduced Payload Size: Enables compliance with services that have payload size limits (e.g., AWS CloudFront).
  2. Optimized Performance: Can potentially improve request processing speed and efficiency by reducing data transmission overhead.
  3. Developer Flexibility: Offers developers the choice to manage conversation history as needed in their backend systems.

Use Case

In scenarios where back-end services impose payload size restrictions, this flag allows developers to selectively exclude potentially large conversation histories. A typical use case is sending inference requests containing only the latest user prompt, while managing a more extensive conversation history in the backend.

Important Note: Developers will still need to implement their own backend persistence mechanisms if they require LLM access to the complete conversation history beyond the latest user message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant