Ollama tool calls / structured output via LiteLLM are unreliable

[jackmpcollins]

Follow-on from issue #194

Ollama models (via LiteLLM) are returning incorrect function name in tool call output, which leads to magentic failing to parse this.

from magentic import prompt
from magentic.chat_model.litellm_chat_model import LitellmChatModel
from pydantic import BaseModel, Field


class Superhero(BaseModel):
    name: str
    age: int = Field(description="The age of the hero, could be very old.")
    power: str = Field(examples=["Runs really fast"])
    enemies: list[str]


@prompt(
    "Create a Superhero named {name}. Use the return_superhero function. Make sure to use the correct function name.",
    model=LitellmChatModel("ollama_chat/llama3", api_base="http://localhost:11434")
)
def create_superhero(name: str) -> Superhero: ...

create_superhero("Garden Man")

ValueError: Unknown tool call: {"id":"call_4ca84210-3b30-4cd6-a109-05044d703923","function":{"arguments":"{\"Garden Man\": {\"Name\": \"Garden Man\", \"Age\": 35, \"Power\": \"Can control plants and make them grow at an incredible rate\", \"Enemies\": [\"Pest Control\", \"Weed Killer\"]}}","name":"return_ super hero"},"type":"function","index":0}

I've tried a few variations of the prompt using llama3 to get it to use the correct function name but it basically never gets this right.

In this simple case (one return type, no functions) we could patch over this by ignoring the name / assuming the output is for the return_superhero function, but that would not work for the more general case of multiple return types or functions.

The ultimate solution will require better support for tool calls from ollama and litellm. llama.cpp supports tool calls in their python client https://github.com/abetlen/llama-cpp-python#function-calling but this is not currently exposed in ollama's OpenAI-compatible API https://github.com/ollama/ollama/blob/main/docs/openai.md . I have opened a new github issue with Ollama for this ollama/ollama#4386 . After that, LiteLLM will also require an update to make use of this.