Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.0.20 create a breaking change for StreamingTextResponse #1316

Open
arvinxx opened this issue Apr 10, 2024 · 33 comments
Open

3.0.20 create a breaking change for StreamingTextResponse #1316

arvinxx opened this issue Apr 10, 2024 · 33 comments
Assignees

Comments

@arvinxx
Copy link

arvinxx commented Apr 10, 2024

Description

the correct response with ai@3.0.19:

image

the error response with ai@3.0.20:

image

it cause an issue with it: lobehub/lobe-chat#1945

Code example

import { OpenAIStream, StreamingTextResponse } from 'ai';
import OpenAI, { ClientOptions } from 'openai';

const response = await this.client.chat.completions.create(postPayload);

return new StreamingTextResponse(OpenAIStream(response));

Additional context

I check the update code, I think it's a breaking change for StreamingTextResponse :

image

please consider to revert it. Or I have to stay with 3.0.19.

a6b2500#diff-ee443c120877f90b068a98d03b338a35958a3953db7e0159035ae060b5b9052b

@arvinxx arvinxx changed the title 3.0.20 create error response data 3.0.20 create a breaking change for StreamingTextResponse Apr 10, 2024
@lgrammel
Copy link
Collaborator

lgrammel commented Apr 10, 2024

Solutions

  • if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh
  • If you use your own server-side code, you can
    • pin the version of the AI SDK to 3.0.19
    • use streamMode: "text" in useChat / useCompletion (requires v3.0.23)
  • If you use your own client-side code, you can

(old)

useChat and useCompletion support the stream data protocol and should continue to work. Can you elaborate on how you are using the result of StreamingTextResponse?

@arvinxx
Copy link
Author

arvinxx commented Apr 10, 2024

this is how I use the Response of StreamingTextResponse.

/**
 * Fetch data using stream method
 */
export const fetchSSE = async (fetchFn: () => Promise<Response>, options: FetchSSEOptions = {}) => {
  const response = await fetchFn();

  if (!response.ok) {
    const chatMessageError = await getMessageError(response);

    options.onErrorHandle?.(chatMessageError);
    return;
  }

  const returnRes = response.clone();

  const data = response.body;

  if (!data) return;
  let output = '';
  const reader = data.getReader();
  const decoder = new TextDecoder();

  let done = false;
  let finishedType: SSEFinishType = 'done';

  while (!done) {
    try {
      const { value, done: doneReading } = await reader.read();
      done = doneReading;
      const chunkValue = decoder.decode(value, { stream: true });

      output += chunkValue;
      options.onMessageHandle?.(chunkValue);
    } catch (error) {
      done = true;

      if ((error as TypeError).name === 'AbortError') {
        finishedType = 'abort';
        options?.onAbort?.(output);
      } else {
        finishedType = 'error';
        console.error(error);
      }
    }
  }

  await options?.onFinish?.(output, { observationId, traceId, type: finishedType });

  return returnRes;
};

I don't use useChat and useCompletion because of I need lots of customization and hooks are too weak for lobe-chat's usage. I just use StreamingTextResponse to simplify the handle of openai 's response.

@lgrammel
Copy link
Collaborator

I see. We have switched to a new protocol that supports different types of messages (tool calls, etc). This provides the foundation to robustly support richer LLM functionality such as tool calling, annotations, etc.

If you use StreamData without useChat / useCompletion, you need to update your code to the new protocol.

In the stream data protocol, each message is a line with a type (indicated by a number) separated from the value.

E.g. 0:some chunk is a text chunk some chunk. (message types: https://github.com/vercel/ai/blob/main/packages/core/shared/stream-parts.ts )

With readDataStream you can parse the chunks: https://github.com/vercel/ai/blob/main/packages/core/shared/read-data-stream.ts

Example usage: https://github.com/vercel/ai/blob/main/packages/core/shared/parse-complex-response.ts#L52

For the basic use case that resembles the original text streaming, you just need to process text parts.

 for await (const { type, value } of readDataStream(reader, {
    isAborted: () => abortControllerRef?.current === null,
  })) {
    if (type === 'text') {
       doSomething(value);
    }
}

@lgrammel lgrammel self-assigned this Apr 10, 2024
@arvinxx
Copy link
Author

arvinxx commented Apr 10, 2024

I agree with the new protocol to support more types of messages and actually LobeChat needs it too.

But I don't think the StreamingTextResponse should be changed in this PATCH version. It's a really breaking change for others depend on the ai sdk.

Even use major version, I still think there is demand on pure text streaming response. Other library even has document how to intergate with StreamingTextResponse like pro-chat. Most users are just need a text output. If I need to handle with the protocol, why to use this StreamingTextResponse?

I really suggestion another method to add with the new protocol, like StreamingChatResponse and make the StreamingTextResponse still pure text output.

@lgrammel
Copy link
Collaborator

I agree. This was not intended to be a breaking change. StreamingTextResponse is meant to be used by useChat/useCompletion, and your use case unfortunately is outside the envisioned use. Because of some other changes that we are currently making (splitting out packages), it is sadly not possible to revert at the moment.

I suggest either pinning v3.0.19 for the time being or moving to the stream data protocol.

We'll include the simple text stream use case in the new streamText API.

@arvinxx
Copy link
Author

arvinxx commented Apr 10, 2024

Ok, I will pin to v3.0.19 currently. By the way, is there any document about the new stream data protocol? I will need it when I working on the claude function call feature.

@lgrammel
Copy link
Collaborator

There is no documentation yet other than the code. It was intended to be internal - so pinning the version will be important if you use it. Given your use case, it makes sense to standardize and document it, so we'll look into it.

PR for future raw text stream response: #1318

@arvinxx
Copy link
Author

arvinxx commented Apr 10, 2024

That's a nice direction to standardize the data stream protocol. I had just used regex to match the function call output and it can't be extandible. I'm glad if there is something more powerful then pure text.

You can check our document If you are interested in the LobeChat Plugin. And the function call I implemented is here: https://github.com/lobehub/lobe-chat/blob/main/src/store/chat/slices/plugin/action.ts#L181-L264

@lgrammel
Copy link
Collaborator

Interesting - love the store!

Not sure how well this suites your use case, and it is not fully integrated with the stream data protocol yet, but in the future you can also define tools with the new streamText API: https://sdk.vercel.ai/docs/ai-core/stream-text#terminal-chatbot-with-tools

@alishobeiri
Copy link

For what it's worth, I actually had the same issue here as well. We parse the output of the stream ourselves and the update to 3.0.20 broke things for us as well which led me here 😄.

If it's any help, this is the code we have:

...
export default async function handler(req: Request, res: NextApiResponse) {
	if (req.method === "POST") {
...
		try {
			const response = await openai.chat.completions.create({
				...
				stream: true,
			});

			const stream = OpenAIStream(response);
			return new StreamingTextResponse(stream);
		} catch (error) {
			console.error("Error calling OpenAI API:", error);
			return NextResponse.json(
				{ error: "Error calling OpenAI API" },
				{ status: 500 },
			);
		}
	} else {
		// Handle any non-POST requests
		return NextResponse.json(
			{ error: "Error calling OpenAI API" },
			{ status: 405 },
		);
	}
}

It sounds like the right move is to switch to readDataStream so I will try that now. Thanks a lot!

@metawise
Copy link

I'm having a same problem.

Frontend: `'use client';
import { useChat, type Message } from 'ai/react';
import React, { useRef, useEffect } from 'react';
import { Remark, useRemark } from 'react-remark';
import ScrollToBottom from './ScrollToBottom';
const BACKEND = process.env.NEXT_PUBLIC_BACKEND;
export default function ChatComponent() {
const options = {
api: BACKEND + '/api/chat'
};
const { input, handleInputChange, handleSubmit, isLoading, messages, setMessages, setInput } = useChat(options);

const inputRef = useRef<HTMLInputElement>(null);

return (
    <>
        {/* {JSON.stringify(messages)} */}
        {messages.length > 0 && (
            <div className="h-[600px] overflow-y-scroll p-2">
                {messages?.map((message: Message, OutIndex) => {
                    return (
                        <ScrollToBottom key={OutIndex}>
                            {message.role === 'assistant' ? <h3 className="text-lg font-semibold mt-2">GPT-4</h3> : <h3 className="text-lg font-semibold mt-2">User</h3>}
                            {message?.content?.split('\n').map((currentTextBlock: string, index: number) => {
                                if (currentTextBlock === '') {
                                    return;
                                } else {
                                    return <Remark key={index}>{currentTextBlock}</Remark>;
                                }
                            })}
                        </ScrollToBottom>
                    );
                })}
            </div>
        )}

        <form className="mt-4 px-4" onSubmit={handleSubmit}>
            <p>User Message</p>
            <input
                className="w-full bg-slate-100 p-2 outline-none"
                type="url"
                ref={inputRef}
                name="url"
                aria-autocomplete="list"
                required
                disabled={isLoading}
                placeholder="Enter URL to scrape*"
            />
            <textarea
                className="w-full bg-slate-100 p-2  outline-none"
                placeholder={'Enter your additional prompt...'}
                value={input}
                onChange={handleInputChange}
                disabled={isLoading}
            />
            <button disabled={isLoading} className="rounded-md bg-blue-600 text-white p-2 mt-2">
                {isLoading ? 'Loading...' : 'Send'}
            </button>
        </form>
    </>
);

}
`

Backend:
`import OpenAI from 'openai';
import { OpenAIStream, StreamingTextResponse } from 'ai';

// Create an OpenAI API client (that's edge friendly!)
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY
});

// IMPORTANT! Set the runtime to edge
export const runtime = 'edge';

export async function POST(req) {
const { messages } = await req.json();

messages.push({ role: 'system', content: 'tell me joke.' });

console.log(messages);

// Ask OpenAI for a streaming chat completion given the prompt
const response = await openai.chat.completions.create({
    model: 'gpt-4',
    stream: true,
    messages
});

// Convert the response into a friendly text-stream
const stream = OpenAIStream(response);
// Respond with the stream
return new StreamingTextResponse(stream);

}
`

Response:
Screenshot 2024-04-10 at 3 31 43 PM

@lgrammel
Copy link
Collaborator

Update on solutions:

  • if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh
  • If you use your own client-side code, you can

@martgra
Copy link

martgra commented Apr 11, 2024

Hi @lgrammel Would be great to have some updated documentation on this. I struggle with the AIStream - as it stopped working when moving to 3.0.21. I use useChat.

@lgrammel
Copy link
Collaborator

Hi @lgrammel Would be great to have some updated documentation on this. I struggle with the AIStream - as it stopped working when moving to 3.0.21. I use useChat.

The documentation is up-to-date. Can you elaborate what stopped working? Are you using the same version of the AI SDK for client & server and have rebuilt/refreshed?

@martgra
Copy link

martgra commented Apr 11, 2024

@lgrammel

For me the stream is no longer displayed in the UI. I use useChat, the exact same implementation as an earlier version of AI Chatbot template by vercel.

  1. I think the same version of SDK for client and server. Im on a nextjs stack and the package is referenced only once
  2. Yes I have rebuilt and refreshed.

I can elaborate:

I have a custom parser that looks like:

function parseMessageStream(): AIStreamParser {
  return (data: string): string => {
    const json = JSON.parse(data)
    let delta: string
    if (json.data && json.data.message && json.data.message.content) {
      delta = json.data.message.content
    } else {
      delta = ''
    }
    return delta
  }
}

When I look at tokens with the callbacks it parses my stream. However useChat is unable to display the tokens or completion client side.

I suspect there is something in the new data protocol I am missing. So instead of head scratching for an extended period of time I was hoping you had some magic for me 🪄

@lgrammel
Copy link
Collaborator

@martgra If I understand correctly, you are returning custom text chunks. In the new protocol, text chunks are lines that look like this 0:my-chunk\n. You can use formatStreamPart to achieve this, e.g. formatStreamPart("text", delta) (see https://github.com/vercel/ai/blob/main/packages/core/shared/stream-parts.ts#L363 ). Please be aware that this is an internal API though.

@martgra
Copy link

martgra commented Apr 11, 2024

@martgra If I understand correctly, you are returning custom text chunks. In the new protocol, text chunks are lines that look like this 0:my-chunk\n. You can use formatStreamPart to achieve this, e.g. formatStreamPart("text", delta) (see https://github.com/vercel/ai/blob/main/packages/core/shared/stream-parts.ts#L363 ). Please be aware that this is an internal API though.

Suspected something like this :) Now it streams!

@metawise
Copy link

metawise commented Apr 11, 2024

Update on solutions:

  • if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh

  • If you use your own client-side code, you can

Matching the versions to '3.0.21' on both client and server SDK did fix the formatting issue. Thanks!

@guseggert
Copy link

IIUC it's not possible to use readDataStream because it's not exported, is this correct?

@martgra
Copy link

martgra commented Apr 12, 2024

@lgrammel may I ask if useChat sometime will use the SSE protocol directly? Seems like this text stream is less flexible than actually including more metadata in the stream like providers do?

@lgrammel
Copy link
Collaborator

lgrammel commented Apr 12, 2024

IIUC it's not possible to use readDataStream because it's not exported, is this correct?

Thanks - I'll work on getting it exported.

Update: #1334 (published in v3.0.22)

@lgrammel
Copy link
Collaborator

@lgrammel may I ask if useChat sometime will use the SSE protocol directly? Seems like this text stream is less flexible than actually including more metadata in the stream like providers do?

Possibly. For the time being we have no such plans though.

@martgra
Copy link

martgra commented Apr 12, 2024

@lgrammel may I ask if useChat sometime will use the SSE protocol directly? Seems like this text stream is less flexible than actually including more metadata in the stream like providers do?

Possibly. For the time being we have no such plans though.

Allright - thanks for answer. I do actually like this useChat better than the GenAI aiState and UIState framework. Its so lightweight and easy to use.

@flexchar
Copy link

flexchar commented Apr 14, 2024

Hi Lars, I came across this. It still works but not streaming - the response is not shown until it's done. Versions matched on 3.0.22.

First of all, I am frustrated that a breaking change did not cause a major version change. I really believe that this is not a good practice. Also, I cannot seem to find any upgrade guide. This leaves me wondering whether it's my fault or a fault at the library.

Nevertheless, thank you for working on this.

@lgrammel

UPDATE:

I end up discovering that Wrangler broke the streaming so it is not this library, even though the timing did not workout to the benefit.

cloudflare/workers-sdk#5614

I would still be interested in an upgrade guide. It seems to me that either I am confused (could be) or the official docs don't reflect the changes - I still see experimental_streamData in many places.

@lgrammel
Copy link
Collaborator

lgrammel commented Apr 14, 2024

@flexchar I understand that this is frustrating, and I agree it's unfortunate. It was not an intentional breakage, and unfortunately it happened during a split into multiple packages, which made the immediate revert impossible. By now, reverting would break the code of people who have already upgraded.

Here is something close to an upgrade guide: #1316 (comment)

The docs should be updated, e.g. https://sdk.vercel.ai/docs/api-reference/stream-data - where did you still see experimental_streamData?

srizvi added a commit to yocxo/studio that referenced this issue Apr 14, 2024
- Replaced generated summary with last working static due to unexpected breaking change
  - vercel/ai#1316

Link #231
@flexchar
Copy link

Ahh, I may have mixed up with another experimental_stream* functions.

Thanks for fast reply.

Shit happens. Good thing, we don't have to revert since it works on the latest versions if both are the same. But in the future, a greater communication - perhaps a pinned issue or README block would be super nice. Ideally on the release part on Github. :)

@lgrammel
Copy link
Collaborator

lgrammel commented Apr 22, 2024

Update on solutions:

  • if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh
  • If you use your own server-side code, you can
    • pin the version of the AI SDK to 3.0.19
    • use streamMode: "text" in useChat / useCompletion (requires v3.0.23)
  • If you use your own client-side code, you can

@bernaferrari
Copy link

Vercel AI SDK demo is suffering from this
image

@liuhuapiaoyuan
Copy link

Difficult to understand, even if it should be used in useChat, I also think this is a bug because from the literal meaning, we would think that StreamingTextResponse is to convert Stream into text... rather than adding 0: 1:.

@GorvGoyl
Copy link

GorvGoyl commented May 10, 2024

@lgrammel, my API endpoint (with AI SDK) is consumed by my Next.js web app via useChat, as well as by my Chrome extension via SSE. How can I use the same API endpoint for both applications without downgrading the AI SDK?

@flexchar
Copy link

Could you upgrade both parts to the latest version? @GorvGoyl

@lgrammel
Copy link
Collaborator

@GorvGoyl if your endpoint send a text stream (not an ai stream), then you could use streamMode: "text" in useChat

@NZainchkovskiy
Copy link

Update on solutions:

  • if you use useChat or useCompletion, it will continue to work. If you see issues, it's most likely caused by a mismatch of the client and server version of the AI SDK, or you need to rebuild/refresh

  • If you use your own server-side code, you can

    • pin the version of the AI SDK to 3.0.19
    • use streamMode: "text" in useChat / useCompletion (requires v3.0.23)
  • If you use your own client-side code, you can

    • pin the version of the AI SDK to 3.0.19
    • or use readDataStream (see 3.0.20 create a breaking change for StreamingTextResponse #1316 (comment) )
    • or switch to AI Core on the server side and use streamText with toTextStreamResponse (v3.0.21 and newer)
      export async function POST(req: Request) {
        const { prompt } = await req.json();
      
        const result = await experimental_streamText({
          model: openai.completion('gpt-3.5-turbo-instruct'),
          maxTokens: 2000,
          prompt,
        });
      
        return result.toTextStreamResponse();
      }

toTextStreamResponse() lacks callback. Is any option to use it with cache like described in the docs (https://sdk.vercel.ai/docs/advanced/caching)?
Generally speaking according the docs I should cache text response and return it then from cache. But I can't return text stream because useCompletion expects AIStream instead.
I tried to switch to result.toTextStreamResponse, but can't figure how to call cache setting without onFinal callback.
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests