docs/pages/docs/api-reference/streaming-react-response.md

@@ -0,0 +1,156 @@
+---
+title: experimental_StreamingReactResponse
+layout:
+  toc: false
+---
+
+# `experimental_StreamingReactResponse`
+
+The `experimental_StreamingReactResponse` class allows you to stream React component responses.
+
+<Callout>
+  The `experimental_` prefix indicates that the API is not yet stable and may
+  change in the future without a major version bump.
+
+It is currently only implemented from `ai/react`'s `useChat` hook.
+
+</Callout>
+
+## `experimental_StreamingReactResponse(res: ReadableStream, options?: ResponseOptions): Response` [#streamingreactresponse]
+
+The `experimental_StreamingReactResponse` class is designed to facilitate streaming React responses in a server action environment. It can handle and stream both raw content and data payloads, including special UI payloads, through nested promises.
+
+## Parameters
+
+### `res: ReadableStream`
+
+This parameter should be a `ReadableStream`, which encapsulates the HTTP response's content. It represents the stream from which the response is read and processed.
+
+### `options?: {ui?: Function, data?: experimental_StreamData}`
+
+This optional parameter allows additional configurations for rendering React components and handling streamed data.
+
+The options object can include:
+
+- `ui?: (message: {content: string, data?: JSONValue[] | undefined}) => UINode | Promise<UINode>`: A function that receives a message object with `content` and optional `data` fields. This function should return a React component (as `UINode`) for each chunk in the stream. The `data` attribute in the message is available when the `data` option is configured to include stream data.
+- `data?: experimental_StreamData`: An instance of `experimental_StreamData` used to process and stream data along with the response.
+
+## Returns
+
+The method returns a `Promise<ReactResponseRow>`, which resolves to the next row of the React response. Each row contains a payload with UI components (`ui`), raw content (`content`), and a `next` property pointing to the subsequent row or `null` if it's the last row. This setup allows for continuous streaming and rendering of data in a React-based UI.
+
+## Example
+
+### Server-Side Implementation
+
+This example highlights the usage of `experimental_StreamingReactResponse` specifically within a server action context in a Next.js environment. It is crucial to note that this functionality is only available when used in server actions. For more details on server actions, visit [Next.js Documentation on Server Actions](https://nextjs.org/docs/app/api-reference/functions/server-actions).
+
+In this server-side script, the `handler` function interacts with the OpenAI API to process and stream chat completions. The responses are streamed using `OpenAIStream` and subsequently managed by `experimental_StreamingReactResponse` for dynamic React component rendering.
+
+Server:
+
+```tsx
+// app/stream-react-response/action.tsx
+'use server';
+
+import OpenAI from 'openai';
+import { OpenAIStream, experimental_StreamingReactResponse, Message } from 'ai';
+import { Counter } from './counter';
+
+const openai = new OpenAI({
+  apiKey: process.env.OPENAI_API_KEY!,
+});
+
+export async function handler({ messages }: { messages: Message[] }) {
+  // Request the OpenAI API for the response based on the prompt
+  const response = await openai.chat.completions.create({
+    model: 'gpt-3.5-turbo',
+    stream: true,
+    messages: messages.map(m => ({
+      role: m.role,
+      content: m.content,
+    })),
+  });
+
+  // Convert the response into a friendly text-stream
+  const stream = OpenAIStream(response);
+
+  // Respond with the stream
+  return new experimental_StreamingReactResponse(stream, {
+    ui({ content }) {
+      return (
+        <div className="italic text-red-800">
+          Visit Next.js docs at{' '}
+          <a
+            href="https://nextjs.org/docs"
+            target="_blank"
+            className="underline"
+          >
+            https://nextjs.org/docs
+          </a>
+        </div>
+      );
+    },
+  });
+}
+```
+
+### Client-Side Setup
+
+On the client side, the useChat hook from ai/react is utilized to manage the chat interaction. The `Chat` component below renders the chat interface, including input forms and message displays. It dynamically integrates the server-side stream, displaying messages and UI elements as they are received.
+
+```tsx
+// app/stream-react-response/page.tsx
+
+import { handler } from './action';
+import { Chat } from './chat';
+
+export const runtime = 'edge';
+
+export default function Page() {
+  return <Chat handler={handler} />;
+}
+```
+
+```tsx
+// app/stream-react-response/chat.tsx
+
+'use client';
+
+import { useChat } from 'ai/react';
+
+export function Chat({ handler }: { handler: any }) {
+  const { messages, input, handleInputChange, handleSubmit } = useChat({
+    api: handler,
+  });
+
+  return (
+    <div className="container mx-auto p-4">
+      <ul>
+        {messages.map((m, index) => (
+          <li key={index}>
+            {m.role === 'user' ? 'User: ' : 'AI: '}
+            {m.role === 'user' ? m.content : m.ui}
+          </li>
+        ))}
+      </ul>
+
+      <form
+        className="flex gap-2 fixed bottom-0 left-0 w-full p-4 border-t"
+        onSubmit={handleSubmit}
+      >
+        <input
+          className="border border-gray-500 rounded p-2 w-full"
+          placeholder="what is Next.js..."
+          value={input}
+          onChange={handleInputChange}
+          autoFocus
+        />
+        <button type="submit" className="bg-black text-white rounded px-4">
+          Send
+        </button>
+      </form>
+    </div>
+  );
+}
+```

docs/pages/docs/api-reference/use-chat.mdx

@@ -289,8 +289,11 @@ The `useChat` function returns an object containing several helper methods and v
     ],
     [
       'handleSubmit',
-      '(e: React.FormEvent<HTMLFormElement>) => void',
-      'Form submission handler that automatically resets the input field and appends a user message.',
+      '(e: React.FormEvent<HTMLFormElement>, options: ChatRequestOptions) => void',
+      'Form submission handler that automatically resets the input field and appends a user message.' +
+        'The chat request options can have a `data` property ' +
+        'that you can use to send an extra body object to the API endpoint in addition to the `messages` array.',
+      ,
     ],
     [
       'isLoading',

docs/pages/docs/guides/providers/openai.mdx

@@ -311,3 +311,85 @@ const openai = new OpenAI({
   The OpenAI SDK now supports Edge Runtime out of the box, so we recommend using
   the official `openai` library instead.
 </Callout>
+
+## Guide: Using Images with GPT 4 Vision and useChat
+
+You can use the extra `data` property that is part of `handleSubmit` to send additional data
+such as an image URL or a base64 encoded image to the server
+
+```tsx filename="app/page.tsx" showLineNumbers
+'use client';
+
+import { useChat } from 'ai/react';
+
+export default function Chat() {
+  const { messages, input, handleInputChange, handleSubmit } = useChat();
+  return (
+    <div className="flex flex-col w-full max-w-md py-24 mx-auto stretch">
+      {messages.length > 0
+        ? messages.map(m => (
+            <div key={m.id} className="whitespace-pre-wrap">
+              {m.role === 'user' ? 'User: ' : 'AI: '}
+              {m.content}
+            </div>
+          ))
+        : null}
+
+      <form
+        onSubmit={e => {
+          handleSubmit(e, {
+            // data could contain the URL of an image that was uploaded to your service, or a base64 encoded image:
+            data: {
+              imageUrl:
+                'https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Field_sparrow_in_CP_%2841484%29_%28cropped%29.jpg/733px-Field_sparrow_in_CP_%2841484%29_%28cropped%29.jpg',
+            },
+          });
+        }}
+      >
+        <input
+          className="fixed bottom-0 w-full max-w-md p-2 mb-8 border border-gray-300 rounded shadow-xl"
+          value={input}
+          placeholder="What does the image show..."
+          onChange={handleInputChange}
+        />
+      </form>
+    </div>
+  );
+}
+```
+
+On the server, you can pass that information to GPT-4 Vision.
+
+```tsx filename="app/api/chat/route.ts" showLineNumbers
+export async function POST(req: Request) {
+  // 'data' contains the additional data that you have sent:
+  const { messages, data } = await req.json();
+
+  const initialMessages = messages.slice(0, -1);
+  const currentMessage = messages[messages.length - 1];
+
+  const response = await openai.chat.completions.create({
+    model: 'gpt-4-vision-preview', // use a GPT-4 vision model
+    stream: true,
+    max_tokens: 150,
+    messages: [
+      ...initialMessages,
+      {
+        ...currentMessage,
+        content: [
+          { type: 'text', text: currentMessage.content },
+
+          // forward the image information to OpenAI:
+          {
+            type: 'image_url',
+            image_url: data.imageUrl,
+          },
+        ],
+      },
+    ],
+  });
+
+  const stream = OpenAIStream(response);
+  return new StreamingTextResponse(stream);
+}
+```

examples/next-anthropic/package.json

@@ -10,7 +10,7 @@
   },
   "dependencies": {
     "@anthropic-ai/sdk": "^0.6.2",
-    "ai": "2.2.23",
+    "ai": "2.2.24",
     "next": "13.4.12",
     "react": "18.2.0",
     "react-dom": "^18.2.0"

examples/next-fireworks/package.json

@@ -9,7 +9,7 @@
     "lint": "next lint"
   },
   "dependencies": {
-    "ai": "2.2.23",
+    "ai": "2.2.24",
     "next": "13.4.12",
     "openai": "4.16.1",
     "react": "18.2.0",

examples/next-huggingface/package.json

@@ -11,7 +11,7 @@
   "dependencies": {
     "@huggingface/inference": "^2.5.1",
     "next": "13.4.12",
-    "ai": "2.2.23",
+    "ai": "2.2.24",
     "react": "18.2.0",
     "react-dom": "^18.2.0"
   },

examples/next-langchain/package.json

@@ -9,7 +9,7 @@
     "lint": "next lint"
   },
   "dependencies": {
-    "ai": "2.2.23",
+    "ai": "2.2.24",
     "langchain": "^0.0.129",
     "next": "13.4.12",
     "react": "18.2.0",

examples/next-openai-rate-limits/package.json

@@ -11,7 +11,7 @@
   "dependencies": {
     "@upstash/ratelimit": "^0.4.3",
     "@vercel/kv": "^0.2.2",
-    "ai": "2.2.23",
+    "ai": "2.2.24",
     "next": "13.4.12",
     "openai": "4.16.1",
     "react": "18.2.0",

examples/next-openai/app/api/assistant/assistant-setup.md

@@ -0,0 +1,63 @@
+# Home Automation Assistant Example
+
+## Setup
+
+### Create OpenAI Assistant
+
+[OpenAI Assistant Website](https://platform.openai.com/assistants)
+
+Create a new assistant. Enable Code interpreter. Add the following functions and instructions to the assistant.
+
+Then add the assistant id to the `.env.local` file as `ASSISTANT_ID=your-assistant-id`.
+
+### Instructions
+
+```
+You are an assistant with access to a home automation system. You can get and set the temperature in the bedroom, home office, living room, kitchen and bathroom.
+
+The system uses temperature in Celsius. If the user requests Fahrenheit, you should convert the temperature to Fahrenheit.
+```
+
+### getRoomTemperature function
+
+```json
+{
+  "name": "getRoomTemperature",
+  "description": "Get the temperature in a room",
+  "parameters": {
+    "type": "object",
+    "properties": {
+      "room": {
+        "type": "string",
+        "enum": ["bedroom", "home office", "living room", "kitchen", "bathroom"]
+      }
+    },
+    "required": ["room"]
+  }
+}
+```
+
+### setRoomTemperature function
+
+```json
+{
+  "name": "setRoomTemperature",
+  "description": "Set the temperature in a room",
+  "parameters": {
+    "type": "object",
+    "properties": {
+      "room": {
+        "type": "string",
+        "enum": ["bedroom", "home office", "living room", "kitchen", "bathroom"]
+      },
+      "temperature": { "type": "number" }
+    },
+    "required": ["room", "temperature"]
+  }
+}
+```
+
+## Run
+
+1. Run `pnpm run dev` in `examples/next-openai`
+2. Go to http://localhost:3000/assistant