JSON Schema is wrong when mode='serialization' and fields have a default

[samuelcolvin]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

In the example below, bar will always have a value, so should be included in required, but it's not.

Example Code

from pydantic import BaseModel

class Foobar(BaseModel):
    foo: int
    bar: int = 1

print(Foobar.model_json_schema(mode='serialization'))
"""
{
    'properties': {
        'foo': {
            'title': 'Foo',
            'type': 'integer',
        },
        'bar': {
            'default': 1,
            'title': 'Bar',
            'type': 'integer',
        },
    },
    'required': ['foo'],
    'title': 'Foobar',
    'type': 'object',
}
"""

Python, Pydantic & OS Version

pydantic version: 2.5.2
        pydantic-core version: 2.14.5
          pydantic-core build: profile=release pgo=true
                 install path: /Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic
               python version: 3.11.5 (main, Aug 26 2023, 13:55:09) [Clang 14.0.3 (clang-1403.0.22.14.1)]
                     platform: macOS-14.1-arm64-arm-64bit
             related packages: typing_extensions-4.9.0 pyright-1.1.335 fastapi-0.104.1 mypy-1.7.0 email-validator-2.1.0.post1

[dmontagu]

If I recall correctly, we actually did things this way at first, and reverted the change. The reasons for this were (in roughly descending order of importance, imo):

• it results in much less complicated JSON schemas because this was by far the most common way that you'd get different schemas for validation and serialization, which results in needing to distinguish in the generated schema, and results in having different types for inputs and outputs, etc., so was very annoying for many users relying on generating an OpenAPI spec.
• it was closer to v1 behavior, so easier for existing code to work with
• you might do exclude_defaults=True in the model dump

In particular, see this issue: #7209
And this PR: #7275

However, I explicitly added a config setting for this: json_schema_serialization_defaults_required. In general I would advise against using it unless you know you are always going to be emitting serialization schemas, but I think that might be the case for FastUI which is where I assume this is coming from.

[samuelcolvin]

Thanks, sorry I didn't see json_schema_serialization_defaults_required, turns out that doesn't work for me as I use excldue_none=None and json_schema_serialization_defaults_required adds stuff to defaults even when the default is None.

It sounds like this definitely shouldn't change in a minor release, but we should perhaps change the default or make it more powerful in V3.

The only think we could do now is make this configurable on GenerateJsonSchema which helps power users.

Another limitation of this being on config is you can't set it on the root model and have it apply to children.

[dmontagu]

You can override the logic in pydantic.json_schema.GenerateJsonSchema.field_is_required, e.g., if you wanted the field to be considered required precisely when the default value is not None, I think you could do that today.

In the future, if we want to make this method easier to use, we might modify the signature to receive the source type and field name as inputs. Passing the field name would be easy as it is already accessible everywhere we currently call field_is_required, but to get a reference to the source type, we'd need to add that to typed_dict_schema (I think it's present in the model_schema and dataclass_schema, though we'd need to thread it a little farther than the field name).

We definitely could make this change in a backwards compatible way by doing some form of introspection of the signature of the field_is_required function (to continue to support overrides with the original signature), but I'd also be okay waiting for new version where we are okay making breaking changes (especially considering migrating would be as easy as just adding the arguments to the signature and ignoring their values).