Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return usage stats on AssistantMessage #214

Merged
merged 22 commits into from
May 26, 2024
Merged

Return usage stats on AssistantMessage #214

merged 22 commits into from
May 26, 2024

Conversation

jackmpcollins
Copy link
Owner

@jackmpcollins jackmpcollins commented May 16, 2024 •
edited

Resolves #74

  • Create new Usage class
  • Add this as an optional attribute of AssistantMessage
  • Wrap openai/anthropic response stream to set usage when encountered

Example of non-streamed response with usage immediately available

from magentic import OpenaiChatModel, UserMessage

chat_model = OpenaiChatModel("gpt-3.5-turbo", seed=42)
message = chat_model.complete(messages=[UserMessage("Say hello!")])

print(message.usage)
# > Usage(input_tokens=10, output_tokens=9)

Example of streamed response where usage only becomes available after the stream has been processed

from magentic import OpenaiChatModel, UserMessage
from magentic.streaming import StreamedStr

chat_model = OpenaiChatModel("gpt-3.5-turbo", seed=42)
message = chat_model.complete(messages=[UserMessage("Say hello!")], output_types=[StreamedStr])

print(message.usage)
# > `None` because stream has not be processed yet

# Process the stream (convert StreamedStr to str)
str(message.content)

print(message.usage)
# > Usage(input_tokens=10, output_tokens=9)

@jackmpcollins
Copy link
Owner Author

Published as https://github.com/jackmpcollins/magentic/releases/tag/v0.25.0a0 for testing

@jackmpcollins jackmpcollins mentioned this pull request May 16, 2024
Closed
@Lawouach
Copy link

Just wanted to circle back @jackmpcollins I haven't had the chance to try yet. Feel free to push when you want because I'm not sure I'll have a window soon-ish.

@jackmpcollins jackmpcollins marked this pull request as ready for review May 26, 2024 04:33
@jackmpcollins jackmpcollins merged commit 8e7ebbf into main May 26, 2024
2 checks passed
@jackmpcollins jackmpcollins deleted the add-usage-to-messages branch May 26, 2024 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Would it be possible to expose the usage payload of the OpenAI response?
2 participants
@jackmpcollins @Lawouach