feat: Added setLlmTokenCountCallback API endpoint to register a callback for calculating token count when none is provided #2065
Conversation
…ack for calculating token counts when none is provided
…ll `agent.llm.tokenCountCallback` when token_count is not present
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2065 +/- ##
=======================================
Coverage 97.17% 97.18%
=======================================
Files 248 248
Lines 41880 41936 +56
=======================================
+ Hits 40698 40754 +56
Misses 1182 1182
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| * @example | ||
| * // @param {string} model - name of model (i.e. gpt-3.5-turbo) | ||
| * // @param {string} content - prompt or completion response | ||
| * function tokenCallback(model, content) { |
There was a problem hiding this comment.
The value of content is extremely varied. I can see us fielding plenty of issues asking what it will be.
There was a problem hiding this comment.
It must be a string. In the openai cases they are always strings. I know for langchain they aren't always strings but we don't assign tokens. Looking at bedrock they are strings as well
There was a problem hiding this comment.
There was a problem hiding this comment.
it shouldn't be the parsed body. it should be the key in the body that contains the content.
Description
Adds
setLlmTokenCountCallbackto register callback to be called to get token counts when they are not present on events. This function must be synchronous and return token count.Related Issues
Closes #2055