Input tokens somewhat surprising (471k for 89 requests)

[MarkHelsinki]

I’m running a very simply C# method (in Unity) calling the API (GPT4-turbo) to create tags based on two strings, a name and a path, which together would constitute about 10-15 input tokens per item max, e.g. {{name: bone}}, {{path: Icons/Health/Hospital/bone}}

A system message is sent once during initialization, with instructions for how to handle inputs and outputs, with token calculator saying it's about 400 token.

I ran about 10 test calls (so probably about 5k of input tokens), and then ran the main method on the longer list, but it started returning TPM error already after about 2 minutes of running.

I would have expected the entire routine to be a maximum of 75,000 input tokens, spread over probably half an hour of processing time.

private void InitializeGPT()
        {
            ApiKeyManager.LoadOpenAIKey(out string openApiKey);
            OpenAIAPI api = new (openApiKey);
            chat = api.Chat.CreateConversation();
            chat.Model = Model.GPT4_Turbo;
            chat.AppendSystemMessage(chatSystemMessage);
            Debug.Log($"<color=#FEC90B><b><size=14>GPT Icon Tags initialized</size></b></color>");
        }

private async void CreateTags()
        {
            int count = 0;

            foreach ((string iconName, string iconPath) in spriteList)
            {
                chat.AppendUserInput($"{{name: {iconName}}}, {{path: {iconPath}}}");
                try
                {
                    string response = await chat.GetResponseFromChatbotAsync();
                    (string iconLabel, List<string> tags) = response.ParseIconTags();
                    iconDatabase.AddTagsWithLabel(iconLabel, tags);
                    Debug.Log($"Tags processed | Item: <color=#FEC90B><b><size=14>{iconLabel}</size></b></color>");
                    count++;
                }
                catch (Exception ex)
                {
                    Debug.LogError($"Error processing tags for {iconName}: {ex.Message}");
                }
            }

            Debug.Log($"<size=18><color=#40FE0B><b>All items processed: {count}</b></color></size>");
        }

Maybe I'm not understanding this correctly, and each request is adding the system message and full message history each time, and that means with something that requires many calls, it accumulates tokens very quickly, or something just doesn't add up.

I'm going to run the token counter from inside the method and see if this throws any more light on how many tokens are being generated.

[MarkHelsinki]

well, I changed strategy and consolidated my API calls, since this isn't a 'chat' bot inside the game but just leveraging AI to produce game data. But I think more than anything, switching back to 3.5 seems to have dropped the count. Even the previous method when I tested the token count it was only incrementing by 100 each time, so 100 bigger calls each time because it was accumulating the chat history when it didn't really need to. So, I created bigger batch calls and inputs, modified the parsing for the outputs and that solved my particular use case.

OpenAI really should enable single use calls that don't generate a history. I guess simply reinitializing everything does the same thing, but doing this at runtime adds some memory overhead, no?

Anyhow, problem solved for me, though I'm still scratching my head about what turned out to be 580k of input tokens for less than 100 simple calls.