-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Function streaming #232
base: main
Are you sure you want to change the base?
Function streaming #232
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Thanks for this request. I see it's failing the check, let me know if you have an update. It's very important to get the function streaming right, and in particular to not overfit the architecture to openai (as we have many vendors). I do like the functionality, but OpenAI tends to overload APIs which won't even work between models (e.g. -vision- has different types), which makes it a mess for the ecosystem. I'll review this, but not with the highest priority. If you have an update that solves the build issue (some issues with the message type not in sync), and that offers a great abstraction so that other vendors one day will be able to use streaming functions, I'ld love to see it! |
@ramicaza I intend starting to merge this as soon. Important questions:
|
Large progress has been made. Not function streaming yet, but almost. Every model vendor has been abstracted to support a better Streaming and Non-Streaming generation API. Continuing... ⌛ |
Note - evaluating this. Uncertain whether to terminate (execute the corresponding functions) on the client side, or on the server side. |
Function calling is extremely useful but it's a little tricky to implement it together with message streaming because of the weird way functions are streamed by openai. By streaming both messages and functions, it allows for fluid UX like the one displayed below. Notice how the model goes from streaming a response, right to calling the function it needs.
This PR is to enable function call streaming such that modules like ReAct and other future ones can have fluid UIs like the one below.
Screen.Recording.2023-11-19.at.1.29.49.PM.mov