Skip to content

Commit eb150a6

Browse files
lgrammellgrammel
authoredApr 23, 2024··
ai/core: remove scaling of setting values. (#1418)
·
@ai-sdk/xai@2.0.0-canary.10ai@3.0.27
1 parent 202f9ca commit eb150a6

16 files changed

+46
-240
lines changed
 

‎.changeset/fair-avocados-itch.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@ai-sdk/provider-utils': patch
3+
'@ai-sdk/anthropic': patch
4+
'@ai-sdk/provider': patch
5+
'@ai-sdk/mistral': patch
6+
'@ai-sdk/openai': patch
7+
'ai': patch
8+
---
9+
10+
ai/core: remove scaling of setting values (breaking change). If you were using the temperature, frequency penalty, or presence penalty settings, you need to update the providers and adjust the setting values.

‎docs/pages/docs/ai-core/settings.mdx

+5-7
Original file line numberDiff line numberDiff line change
@@ -10,19 +10,17 @@ All AI functions (`generateText`, `streamText`, `generateObject`, `streamObject`
1010

1111
- **maxTokens** - Maximum number of tokens to generate.
1212
- **temperature** - Temperature setting.
13-
This is a number between 0 (almost no randomness) and 1 (very random).
13+
The value is passed through to the provider. The range depends on the provider and model.
1414
It is recommended to set either `temperature` or `topP`, but not both.
15-
- **topP** - Nucleus sampling. This is a number between 0 and 1.
16-
E.g. 0.1 would mean that only tokens with the top 10% probability mass are considered.
15+
- **topP** - Nucleus sampling.
16+
The value is passed through to the provider. The range depends on the provider and model.
1717
It is recommended to set either `temperature` or `topP`, but not both.
1818
- **presencePenalty** - Presence penalty setting.
1919
It affects the likelihood of the model to repeat information that is already in the prompt.
20-
The presence penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
21-
0 means no penalty.
20+
The value is passed through to the provider. The range depends on the provider and model.
2221
- **frequencyPenalty** - Frequency penalty setting.
2322
It affects the likelihood of the model to repeatedly use the same words or phrases.
24-
The frequency penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
25-
0 means no penalty.
23+
The value is passed through to the provider. The range depends on the provider and model.
2624
- **seed** - The seed (integer) to use for random sampling.
2725
If set and supported by the model, calls will generate deterministic results.
2826
- **maxRetries** - Maximum number of retries. Set to 0 to disable retries. Default: 2.

‎packages/anthropic/src/anthropic-messages-language-model.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ export class AnthropicMessagesLanguageModel implements LanguageModelV1 {
9696

9797
// standardized settings:
9898
max_tokens: maxTokens ?? 4096, // 4096: max model output tokens
99-
temperature, // uses 0..1 scale
99+
temperature,
100100
top_p: topP,
101101

102102
// prompt:

‎packages/core/core/generate-object/generate-object.ts

+5-7
Original file line numberDiff line numberDiff line change
@@ -32,19 +32,17 @@ This function does not stream the output. If you want to stream the output, use
3232
3333
@param maxTokens - Maximum number of tokens to generate.
3434
@param temperature - Temperature setting.
35-
This is a number between 0 (almost no randomness) and 1 (very random).
35+
The value is passed through to the provider. The range depends on the provider and model.
3636
It is recommended to set either `temperature` or `topP`, but not both.
37-
@param topP - Nucleus sampling. This is a number between 0 and 1.
38-
E.g. 0.1 would mean that only tokens with the top 10% probability mass are considered.
37+
@param topP - Nucleus sampling.
38+
The value is passed through to the provider. The range depends on the provider and model.
3939
It is recommended to set either `temperature` or `topP`, but not both.
4040
@param presencePenalty - Presence penalty setting.
4141
It affects the likelihood of the model to repeat information that is already in the prompt.
42-
The presence penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
43-
0 means no penalty.
42+
The value is passed through to the provider. The range depends on the provider and model.
4443
@param frequencyPenalty - Frequency penalty setting.
4544
It affects the likelihood of the model to repeatedly use the same words or phrases.
46-
The frequency penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
47-
0 means no penalty.
45+
The value is passed through to the provider. The range depends on the provider and model.
4846
@param seed - The seed (integer) to use for random sampling.
4947
If set and supported by the model, calls will generate deterministic results.
5048

‎packages/core/core/generate-object/stream-object.ts

+5-7
Original file line numberDiff line numberDiff line change
@@ -37,19 +37,17 @@ This function streams the output. If you do not want to stream the output, use `
3737
3838
@param maxTokens - Maximum number of tokens to generate.
3939
@param temperature - Temperature setting.
40-
This is a number between 0 (almost no randomness) and 1 (very random).
40+
The value is passed through to the provider. The range depends on the provider and model.
4141
It is recommended to set either `temperature` or `topP`, but not both.
42-
@param topP - Nucleus sampling. This is a number between 0 and 1.
43-
E.g. 0.1 would mean that only tokens with the top 10% probability mass are considered.
42+
@param topP - Nucleus sampling.
43+
The value is passed through to the provider. The range depends on the provider and model.
4444
It is recommended to set either `temperature` or `topP`, but not both.
4545
@param presencePenalty - Presence penalty setting.
4646
It affects the likelihood of the model to repeat information that is already in the prompt.
47-
The presence penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
48-
0 means no penalty.
47+
The value is passed through to the provider. The range depends on the provider and model.
4948
@param frequencyPenalty - Frequency penalty setting.
5049
It affects the likelihood of the model to repeatedly use the same words or phrases.
51-
The frequency penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
52-
0 means no penalty.
50+
The value is passed through to the provider. The range depends on the provider and model.
5351
@param seed - The seed (integer) to use for random sampling.
5452
If set and supported by the model, calls will generate deterministic results.
5553

‎packages/core/core/generate-text/generate-text.ts

+5-7
Original file line numberDiff line numberDiff line change
@@ -29,19 +29,17 @@ This function does not stream the output. If you want to stream the output, use
2929
3030
@param maxTokens - Maximum number of tokens to generate.
3131
@param temperature - Temperature setting.
32-
This is a number between 0 (almost no randomness) and 1 (very random).
32+
The value is passed through to the provider. The range depends on the provider and model.
3333
It is recommended to set either `temperature` or `topP`, but not both.
34-
@param topP - Nucleus sampling. This is a number between 0 and 1.
35-
E.g. 0.1 would mean that only tokens with the top 10% probability mass are considered.
34+
@param topP - Nucleus sampling.
35+
The value is passed through to the provider. The range depends on the provider and model.
3636
It is recommended to set either `temperature` or `topP`, but not both.
3737
@param presencePenalty - Presence penalty setting.
3838
It affects the likelihood of the model to repeat information that is already in the prompt.
39-
The presence penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
40-
0 means no penalty.
39+
The value is passed through to the provider. The range depends on the provider and model.
4140
@param frequencyPenalty - Frequency penalty setting.
4241
It affects the likelihood of the model to repeatedly use the same words or phrases.
43-
The frequency penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
44-
0 means no penalty.
42+
The value is passed through to the provider. The range depends on the provider and model.
4543
@param seed - The seed (integer) to use for random sampling.
4644
If set and supported by the model, calls will generate deterministic results.
4745

‎packages/core/core/generate-text/stream-text.ts

+5-7
Original file line numberDiff line numberDiff line change
@@ -39,19 +39,17 @@ This function streams the output. If you do not want to stream the output, use `
3939
4040
@param maxTokens - Maximum number of tokens to generate.
4141
@param temperature - Temperature setting.
42-
This is a number between 0 (almost no randomness) and 1 (very random).
42+
The value is passed through to the provider. The range depends on the provider and model.
4343
It is recommended to set either `temperature` or `topP`, but not both.
44-
@param topP - Nucleus sampling. This is a number between 0 and 1.
45-
E.g. 0.1 would mean that only tokens with the top 10% probability mass are considered.
44+
@param topP - Nucleus sampling.
45+
The value is passed through to the provider. The range depends on the provider and model.
4646
It is recommended to set either `temperature` or `topP`, but not both.
4747
@param presencePenalty - Presence penalty setting.
4848
It affects the likelihood of the model to repeat information that is already in the prompt.
49-
The presence penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
50-
0 means no penalty.
49+
The value is passed through to the provider. The range depends on the provider and model.
5150
@param frequencyPenalty - Frequency penalty setting.
5251
It affects the likelihood of the model to repeatedly use the same words or phrases.
53-
The frequency penalty is a number between -1 (increase repetition) and 1 (maximum penalty, decrease repetition).
54-
0 means no penalty.
52+
The value is passed through to the provider. The range depends on the provider and model.
5553
@param seed - The seed (integer) to use for random sampling.
5654
If set and supported by the model, calls will generate deterministic results.
5755

‎packages/core/core/prompt/prepare-call-settings.ts

-32
Original file line numberDiff line numberDiff line change
@@ -39,14 +39,6 @@ export function prepareCallSettings({
3939
message: 'temperature must be a number',
4040
});
4141
}
42-
43-
if (temperature < 0 || temperature > 1) {
44-
throw new InvalidArgumentError({
45-
parameter: 'temperature',
46-
value: temperature,
47-
message: 'temperature must be between 0 and 1 (inclusive)',
48-
});
49-
}
5042
}
5143

5244
if (topP != null) {
@@ -57,14 +49,6 @@ export function prepareCallSettings({
5749
message: 'topP must be a number',
5850
});
5951
}
60-
61-
if (topP < 0 || topP > 1) {
62-
throw new InvalidArgumentError({
63-
parameter: 'topP',
64-
value: topP,
65-
message: 'topP must be between 0 and 1 (inclusive)',
66-
});
67-
}
6852
}
6953

7054
if (presencePenalty != null) {
@@ -75,14 +59,6 @@ export function prepareCallSettings({
7559
message: 'presencePenalty must be a number',
7660
});
7761
}
78-
79-
if (presencePenalty < -1 || presencePenalty > 1) {
80-
throw new InvalidArgumentError({
81-
parameter: 'presencePenalty',
82-
value: presencePenalty,
83-
message: 'presencePenalty must be between -1 and 1 (inclusive)',
84-
});
85-
}
8662
}
8763

8864
if (frequencyPenalty != null) {
@@ -93,14 +69,6 @@ export function prepareCallSettings({
9369
message: 'frequencyPenalty must be a number',
9470
});
9571
}
96-
97-
if (frequencyPenalty < -1 || frequencyPenalty > 1) {
98-
throw new InvalidArgumentError({
99-
parameter: 'frequencyPenalty',
100-
value: frequencyPenalty,
101-
message: 'frequencyPenalty must be between -1 and 1 (inclusive)',
102-
});
103-
}
10472
}
10573

10674
if (seed != null) {

‎packages/mistral/src/mistral-chat-language-model.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ export class MistralChatLanguageModel implements LanguageModelV1 {
8787

8888
// standardized settings:
8989
max_tokens: maxTokens,
90-
temperature, // uses 0..1 scale
90+
temperature,
9191
top_p: topP,
9292
random_seed: seed,
9393

‎packages/openai/src/openai-chat-language-model.test.ts

-45
Original file line numberDiff line numberDiff line change
@@ -330,51 +330,6 @@ describe('doStream', () => {
330330
});
331331
});
332332

333-
it('should scale the temperature', async () => {
334-
prepareStreamResponse({ content: [] });
335-
336-
await provider.chat('gpt-3.5-turbo').doStream({
337-
inputFormat: 'prompt',
338-
mode: { type: 'regular' },
339-
prompt: TEST_PROMPT,
340-
temperature: 0.5,
341-
});
342-
343-
expect((await server.getRequestBodyJson()).temperature).toBeCloseTo(1, 5);
344-
});
345-
346-
it('should scale the frequency penalty', async () => {
347-
prepareStreamResponse({ content: [] });
348-
349-
await provider.chat('gpt-3.5-turbo').doStream({
350-
inputFormat: 'prompt',
351-
mode: { type: 'regular' },
352-
prompt: TEST_PROMPT,
353-
frequencyPenalty: 0.2,
354-
});
355-
356-
expect((await server.getRequestBodyJson()).frequency_penalty).toBeCloseTo(
357-
0.4,
358-
5,
359-
);
360-
});
361-
362-
it('should scale the presence penalty', async () => {
363-
prepareStreamResponse({ content: [] });
364-
365-
await provider.chat('gpt-3.5-turbo').doStream({
366-
inputFormat: 'prompt',
367-
mode: { type: 'regular' },
368-
prompt: TEST_PROMPT,
369-
presencePenalty: -0.9,
370-
});
371-
372-
expect((await server.getRequestBodyJson()).presence_penalty).toBeCloseTo(
373-
-1.8,
374-
5,
375-
);
376-
});
377-
378333
it('should pass custom headers', async () => {
379334
prepareStreamResponse({ content: [] });
380335

‎packages/openai/src/openai-chat-language-model.ts

+3-20
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ import {
1212
generateId,
1313
isParseableJson,
1414
postJsonToApi,
15-
scale,
1615
} from '@ai-sdk/provider-utils';
1716
import { z } from 'zod';
1817
import { convertToOpenAIChatMessages } from './convert-to-openai-chat-messages';
@@ -71,26 +70,10 @@ export class OpenAIChatLanguageModel implements LanguageModelV1 {
7170

7271
// standardized settings:
7372
max_tokens: maxTokens,
74-
temperature: scale({
75-
value: temperature,
76-
outputMin: 0,
77-
outputMax: 2,
78-
}),
73+
temperature,
7974
top_p: topP,
80-
frequency_penalty: scale({
81-
value: frequencyPenalty,
82-
inputMin: -1,
83-
inputMax: 1,
84-
outputMin: -2,
85-
outputMax: 2,
86-
}),
87-
presence_penalty: scale({
88-
value: presencePenalty,
89-
inputMin: -1,
90-
inputMax: 1,
91-
outputMin: -2,
92-
outputMax: 2,
93-
}),
75+
frequency_penalty: frequencyPenalty,
76+
presence_penalty: presencePenalty,
9477
seed,
9578

9679
// messages:

‎packages/openai/src/openai-completion-language-model.test.ts

-45
Original file line numberDiff line numberDiff line change
@@ -208,51 +208,6 @@ describe('doStream', () => {
208208
});
209209
});
210210

211-
it('should scale the temperature', async () => {
212-
prepareStreamResponse({ content: [] });
213-
214-
await provider.completion('gpt-3.5-turbo-instruct').doStream({
215-
inputFormat: 'prompt',
216-
mode: { type: 'regular' },
217-
prompt: TEST_PROMPT,
218-
temperature: 0.5,
219-
});
220-
221-
expect((await server.getRequestBodyJson()).temperature).toBeCloseTo(1, 5);
222-
});
223-
224-
it('should scale the frequency penalty', async () => {
225-
prepareStreamResponse({ content: [] });
226-
227-
await provider.completion('gpt-3.5-turbo-instruct').doStream({
228-
inputFormat: 'prompt',
229-
mode: { type: 'regular' },
230-
prompt: TEST_PROMPT,
231-
frequencyPenalty: 0.2,
232-
});
233-
234-
expect((await server.getRequestBodyJson()).frequency_penalty).toBeCloseTo(
235-
0.4,
236-
5,
237-
);
238-
});
239-
240-
it('should scale the presence penalty', async () => {
241-
prepareStreamResponse({ content: [] });
242-
243-
await provider.completion('gpt-3.5-turbo-instruct').doStream({
244-
inputFormat: 'prompt',
245-
mode: { type: 'regular' },
246-
prompt: TEST_PROMPT,
247-
presencePenalty: -0.9,
248-
});
249-
250-
expect((await server.getRequestBodyJson()).presence_penalty).toBeCloseTo(
251-
-1.8,
252-
5,
253-
);
254-
});
255-
256211
it('should pass custom headers', async () => {
257212
prepareStreamResponse({ content: [] });
258213

‎packages/openai/src/openai-completion-language-model.ts

+3-20
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ import {
99
createEventSourceResponseHandler,
1010
createJsonResponseHandler,
1111
postJsonToApi,
12-
scale,
1312
} from '@ai-sdk/provider-utils';
1413
import { z } from 'zod';
1514
import { convertToOpenAICompletionPrompt } from './convert-to-openai-completion-prompt';
@@ -77,26 +76,10 @@ export class OpenAICompletionLanguageModel implements LanguageModelV1 {
7776

7877
// standardized settings:
7978
max_tokens: maxTokens,
80-
temperature: scale({
81-
value: temperature,
82-
outputMin: 0,
83-
outputMax: 2,
84-
}),
79+
temperature,
8580
top_p: topP,
86-
frequency_penalty: scale({
87-
value: frequencyPenalty,
88-
inputMin: -1,
89-
inputMax: 1,
90-
outputMin: -2,
91-
outputMax: 2,
92-
}),
93-
presence_penalty: scale({
94-
value: presencePenalty,
95-
inputMin: -1,
96-
inputMax: 1,
97-
outputMin: -2,
98-
outputMax: 2,
99-
}),
81+
frequency_penalty: frequencyPenalty,
82+
presence_penalty: presencePenalty,
10083
seed,
10184

10285
// prompt:

‎packages/provider-utils/src/index.ts

-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ export * from './load-api-key';
44
export * from './parse-json';
55
export * from './post-to-api';
66
export * from './response-handler';
7-
export * from './scale';
87
export * from './uint8-utils';
98
export * from './validate-types';
109
export * from './without-trailing-slash';

‎packages/provider-utils/src/scale.ts

-21
This file was deleted.

‎packages/provider/src/language-model/v1/language-model-v1-call-settings.ts

+3-19
Original file line numberDiff line numberDiff line change
@@ -5,24 +5,14 @@ export type LanguageModelV1CallSettings = {
55
maxTokens?: number;
66

77
/**
8-
* Temperature setting. This is a number between 0 (almost no randomness) and
9-
* 1 (very random).
8+
* Temperature setting.
109
*
11-
* Different LLM providers have different temperature
12-
* scales, so they'd need to map it (without mapping, the same temperature has
13-
* different effects on different models). The provider can also chose to map
14-
* this to topP, potentially even using a custom setting on their model.
15-
*
16-
* Note: This is an example of a setting that requires a clear specification of
17-
* the semantics.
10+
* It is recommended to set either `temperature` or `topP`, but not both.
1811
*/
1912
temperature?: number;
2013

2114
/**
22-
* Nucleus sampling. This is a number between 0 and 1.
23-
*
24-
* E.g. 0.1 would mean that only tokens with the top 10% probability mass
25-
* are considered.
15+
* Nucleus sampling.
2616
*
2717
* It is recommended to set either `temperature` or `topP`, but not both.
2818
*/
@@ -31,18 +21,12 @@ export type LanguageModelV1CallSettings = {
3121
/**
3222
* Presence penalty setting. It affects the likelihood of the model to
3323
* repeat information that is already in the prompt.
34-
*
35-
* The presence penalty is a number between -1 (increase repetition)
36-
* and 1 (maximum penalty, decrease repetition). 0 means no penalty.
3724
*/
3825
presencePenalty?: number;
3926

4027
/**
4128
* Frequency penalty setting. It affects the likelihood of the model
4229
* to repeatedly use the same words or phrases.
43-
*
44-
* The frequency penalty is a number between -1 (increase repetition)
45-
* and 1 (maximum penalty, decrease repetition). 0 means no penalty.
4630
*/
4731
frequencyPenalty?: number;
4832

0 commit comments

Comments
 (0)
Please sign in to comment.