anyOf causing problems with Swagger/ReDoc on primitive types

[Kludex]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

If we have an object, and a primite type, we are still creating an anyOf. That unfortunately doesn't work well with neither ReDoc nor Swagger.

From the code below, the following OpenAPI spec is generated:

Details

{
    "components": {
        "schemas": {
            "HTTPValidationError": {
                "properties": {
                    "detail": {
                        "items": {
                            "$ref": "#/components/schemas/ValidationError"
                        },
                        "title": "Detail",
                        "type": "array"
                    }
                },
                "title": "HTTPValidationError",
                "type": "object"
            },
            "ValidationError": {
                "properties": {
                    "loc": {
                        "items": {
                            "type": ["string", "integer"]
                        },
                        "title": "Location",
                        "type": "array"
                    },
                    "msg": {
                        "title": "Message",
                        "type": "string"
                    },
                    "type": {
                        "title": "Error Type",
                        "type": "string"
                    }
                },
                "required": [
                    "loc",
                    "msg",
                    "type"
                ],
                "title": "ValidationError",
                "type": "object"
            }
        }
    },
    "info": {
        "title": "FastAPI",
        "version": "0.1.0"
    },
    "openapi": "3.1.0",
    "paths": {
        "/": {
            "get": {
                "operationId": "root__get",
                "parameters": [
                    {
                        "description": "Date to query",
                        "in": "query",
                        "name": "at",
                        "required": false,
                        "schema": {
                            "anyOf": [
                                {
                                    "enum": [
                                        "today",
                                        "tomorrow",
                                        "yesterday"
                                    ],
                                    "type": "string"
                                },
                                {
                                    "type": "null"
                                }
                            ],
                            "description": "Date to query",
                            "title": "At"
                        }
                    }
                ],
                "responses": {
                    "200": {
                        "content": {
                            "application/json": {
                                "schema": {}
                            }
                        },
                        "description": "Successful Response"
                    },
                    "422": {
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/HTTPValidationError"
                                }
                            }
                        },
                        "description": "Validation Error"
                    }
                },
                "summary": "Root"
            }
        }
    }
}

Ideally, we'd have the following schema:

Details

{
    "components": {
        "schemas": {
            "HTTPValidationError": {
                "properties": {
                    "detail": {
                        "items": {
                            "$ref": "#/components/schemas/ValidationError"
                        },
                        "title": "Detail",
                        "type": "array"
                    }
                },
                "title": "HTTPValidationError",
                "type": "object"
            },
            "ValidationError": {
                "properties": {
                    "loc": {
                        "items": {
                            "anyOf": [
                                {
                                    "type": "string"
                                },
                                {
                                    "type": "integer"
                                }
                            ]
                        },
                        "title": "Location",
                        "type": "array"
                    },
                    "msg": {
                        "title": "Message",
                        "type": "string"
                    },
                    "type": {
                        "title": "Error Type",
                        "type": "string"
                    }
                },
                "required": [
                    "loc",
                    "msg",
                    "type"
                ],
                "title": "ValidationError",
                "type": "object"
            }
        }
    },
    "info": {
        "title": "FastAPI",
        "version": "0.1.0"
    },
    "openapi": "3.1.0",
    "paths": {
        "/": {
            "get": {
                "operationId": "root__get",
                "parameters": [
                    {
                        "description": "Date to query",
                        "in": "query",
                        "name": "at",
                        "required": false,
                        "schema": {
                            "enum": [
                                 "today",
                                 "tomorrow",
                                 "yesterday"
                            ],
                            "type": ["string", "null"]
                            "description": "Date to query",
                            "title": "At"
                        }
                    }
                ],
                "responses": {
                    "200": {
                        "content": {
                            "application/json": {
                                "schema": {}
                            }
                        },
                        "description": "Successful Response"
                    },
                    "422": {
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/HTTPValidationError"
                                }
                            }
                        },
                        "description": "Validation Error"
                    }
                },
                "summary": "Root"
            }
        }
    }
}

As you can see above, the type becomes an array of primitive types. This is mentioned in the Assertions and Instance Primitive Types on the JSON Schema Draft 2020-12.

Example Code

from typing import Literal, Annotated

from fastapi import Query, FastAPI

app = FastAPI()


@app.get("/")
def root(
    at: Annotated[
        Literal["today", "tomorrow", "yesterday"] | None,
        Query(
            description="Date to query",
        ),
    ] = None
):
    return at

Python, Pydantic & OS Version

pydantic version: 2.0.2
        pydantic-core version: 2.2.0 release build profile
                 install path: /Users/marcelotryle/dev/pydantic/pydantic/pydantic
               python version: 3.11.1 (main, Apr 20 2023, 11:08:52) [Clang 14.0.3 (clang-1403.0.22.14.1)]
                     platform: macOS-13.4-arm64-arm-64bit
     optional deps. installed: ['devtools', 'email-validator', 'typing-extensions']

Ref.:

• FastAPI with Pydantic v2 tiangolo/fastapi#9709 (comment)
• FastAPI with Pydantic v2 tiangolo/fastapi#9709 (reply in thread)

Selected Assignee: @dmontagu

[commonism]

You missed the anyOf in ValidationError.
And I'd expect null to be part of the enum due to https://json-schema.org/draft/2020-12/json-schema-validation.html#name-enum

[Kludex]

You missed the anyOf in ValidationError.

I've fixed it. Thanks 🙏

And I'd expect null to be part of the enum due to json-schema.org/draft/2020-12/json-schema-validation.html#name-enum

Ah, ok. Then 2 issues here. 😅

[commonism]

Currently Literal[x]|None is evaluated via anyOf as well, via const instead of enum

data:Literal["dog"] | None

        schema:
          anyOf:
          - const: dog
          - type: 'null'

Due to the known problems I'd unfold to enum instead of const

schema:
  type: [string, null]
  enum: ["dog", null]

The type can be skipped.

schema:
  enum: ["dog", null]

Skipping the type would even work for mixed enums

data: Literal["a",1,True]|None

        schema:
          - enum:
            - a
            - 1
            - true
            - null

[Kludex]

We have a solution using Pydantic only: #6653.

We are also working on a solution for FastAPI without the need of SkipJsonSchema: tiangolo/fastapi#9873.

[rossmacarthur]

Similar issue where previously null was shown by removing from the required list. Instead there is now an anyOf with null as one of the fields.

from typing import Optional
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class RequestModel(BaseModel):
    name: str
    age: Optional[int]


@app.get("/")
def index(req: RequestModel):
    return {"message": "Hello World"}

When using pydantic v1 it returns this OpenAPI schema for RequestModel

"RequestModel": {
    "properties": {
        "name": {
            "type": "string",
            "title": "Name"
        },
        "age": {
            "type": "integer",
            "title": "Age"
        }
    },
    "type": "object",
    "required": [
        "name"
    ],
    "title": "RequestModel"
},

When using pydantic v2 it returns this OpenAPI schema for RequestModel

"RequestModel": {
    "properties": {
        "name": {
            "type": "string",
            "title": "Name"
        },
        "age": {
            "anyOf": [
                {
                    "type": "integer"
                },
                {
                    "type": "null"
                }
            ],
            "title": "Age"
        }
    },
    "type": "object",
    "required": [
        "name",
        "age"
    ],
    "title": "RequestModel"
},

I understand that both are basically the same meaing for my purposes however I have code generation that runs off the OpenAPI schema and the former generates better TypeScript code. Is there a way to generate the former?

[commonism]

Refer to https://docs.pydantic.dev/latest/migration/#required-optional-and-nullable-fields to adjust your model according to your expectation. If you avoid nullable, you won't get the anyOf.

[rossmacarthur]

@commonism are you suggesting the way to do it would be age: int = None? That seems to work, although I'm not sure how this will affect other code.

[commonism]

v2: age: int = None equals v1:age: Optional[int] - so I expect it'll behave all the same.

[dmontagu]

I think the solution of modifying FastAPI to use a custom GenerateJsonSchema (the PR that @Kludex opened) is going to be preferable for fixing swagger, as it won't require anything special in user code, and I think it's not reasonable to change what we are currently generating as the JSON schema for Optional[int] since it's technically more correct for the type annotation, is definitely what you want for Body parameters, and FastAPI currently generates the openapi spec using functions from pydantic that couldn't be aware of this distinction.

However, once #6798 is merged, it will be possible to do:

import json
from typing import TYPE_CHECKING, Annotated, Union

from pydantic import BaseModel, Field
from pydantic.json_schema import SkipJsonSchema

if TYPE_CHECKING:
    from typing import Optional as OptionalParam
else:
    class OptionalParam:
        def __class_getitem__(cls, item):
            return Annotated[
                Union[item, SkipJsonSchema[None]],
                Field(json_schema_extra=lambda x: x.pop('default', None))
            ]

which will then let you do:

class MyModel(BaseModel):
    x: OptionalParam[int] = None
    y: OptionalParam[int]


print(json.dumps(MyModel.model_json_schema(), indent=2))
"""
{
  "properties": {
    "x": {
      "title": "X",
      "type": "integer"
    },
    "y": {
      "title": "Y",
      "type": "integer"
    }
  },
  "required": [
    "y"
  ],
  "title": "MyModel",
  "type": "object"
}
"""

# output from mypy:
reveal_type(MyModel.x)
# note: Revealed type is "Union[builtins.int, None]"
reveal_type(MyModel.y)
# note: Revealed type is "Union[builtins.int, None]"

which will get you a JSON schema / OpenAPI schema that looks the way it did with Pydantic V1. (And as you can see, still type-checks properly.)

Given this, I think after merging that PR it will make sense to close this issue in Pydantic, and instead open a FastAPI issue for the bug (and/or just wait/hope for tiangolo/fastapi#9873 to be merged).

[commonism]

I consider the quality of the description document the key selling point of pydantic, and as I said - this breaks it.
A description documents consumers model for MyModel behavior is different to the original model, changing the semantics of the model.
Have the library create low quality description documents will decrease developers trust in the library itself and all software using the library.

So let's call it CompatibleOptional, document the shortcomings, and emit "x-nullable: True" as Schema Extension in the dd to document the skipped constraints.
It'll allow updating the CompatibleOptional to be a regular Optional once Swagger/* can do better wrt. to anyOf or maybe pydantic switch to type: […, "null"].

[Kludex]

type: […, "null"]

Did you try to generate a Swagger with a schema that contains array as type? Can you show me a schema that have a Swagger UI working as you'd expect?

[commonism]

Use https://editor-next.swagger.io/
Import https://raw.githubusercontent.com/commonism/aiopenapi3/pydanticv2/tests/fixtures/schema-discriminated-union-deep.yaml

For type: […, none], I'm with the reproduce ability of the description document, which I consider a strong quality aspect.

description document -> model -> description document

Splitting

a: str|None

into

Astr:
  type: string
Anull:
  type: "null"
a: Astr|Anull

requires some kind of folding for the models to get back to the original form of a: str|None.

Having the description document compiler emit structural … difficult to deal with … model definitions does not make it a good choice due to the complexity in the models on the consumers side.
It is valid, it validates properly, but it's horrible to work with due to the nesting of models instead of using the intended v3.1 features.

[Kludex]

Can you add some endpoints that use those? I've asked because there were UI issues with it. Your example only has the components.

[commonism]

Rendering is borked for type:[…,"null"] in path/query … most likely everything else as well.
I'll create a demo and report. swagger-api/swagger-ui#9056

[dmontagu]

I'll just note that if we merge #6951, it will require a subtle change to my previously-suggested code for this.

from pydantic import BaseModel, Field
from pydantic.json_schema import SkipJsonSchema


class Model(BaseModel):
    x: int | SkipJsonSchema[None] = Field(default=None, json_schema_extra=lambda x: x.pop('default'))

print(Model.model_json_schema())
#> {'properties': {'x': {'title': 'X', 'type': 'integer'}}, 'title': 'Model', 'type': 'object'}

In particular, what used to be:

x: Annotated[int | SkipJsonSchema[None], Field(json_schema_extra=lambda x: x.pop('default'))] = None

should now be

x: int | SkipJsonSchema[None] = Field(default=None, json_schema_extra=lambda x: x.pop('default'))

I think this is actually more sensible anyway since = None is like = Field(default=None), and Annotated[<type>, <Field(...1)>] = Field(<...2>) is like Annotated[<type>, <Field(...1)>, Field(<...2>)], and Annotated[int | SkipJsonSchema[None], Field(json_schema_extra=lambda x: x.pop('default'))] on its own doesn't make sense, since there won't be a default to pop.

I have updated the PR body for #6798 to reflect this in case anyone goes looking for it in the future.