The "preserve the type of the input whenever possible" union handling is inconsistent

[stevegoss-komodo]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

Code that relies on the strict left-to-right behavior of union validation/parsing is broken in V2, with no available workaround. Not only that, but the subtle change creates inconsistencies where identical data in python or JSON form parses differently. In the example code, all of those would have parsed as UUIDs in V1 (behavior that our code relies on), but now, it parses as string in certain scenarios, but not in others.

It is difficult to migrate with this highly inconsistent and frankly baffling behavior. Using unions like the one in the example with FastAPI as path parameters or query parameters used to be a useful way to detect UUIDs, but now, FastAPI always returns a string.

Example Code

from uuid import UUID

from pydantic import TypeAdapter, BaseModel

IdOrSlug = UUID | str

print(type(TypeAdapter(IdOrSlug).validate_json("\"f4fe10b4-e0c8-4232-ba26-4acd491c2414\"")))
# <class 'uuid.UUID'>

print(type(TypeAdapter(IdOrSlug).validate_python("f4fe10b4-e0c8-4232-ba26-4acd491c2414")))
# <class 'str'>

class AModel(BaseModel):
    my_id: IdOrSlug

model = AModel.model_validate_json("""{"my_id":"f4fe10b4-e0c8-4232-ba26-4acd491c2414"}""")
print(type(model.my_id))
# <class 'uuid.UUID'>

model1 = AModel.model_validate({"my_id":"f4fe10b4-e0c8-4232-ba26-4acd491c2414"})
print(type(model1.my_id))
# <class 'str'>

Python, Pydantic & OS Version

pydantic version: 2.1.1
        pydantic-core version: 2.4.0
          pydantic-core build: profile=release pgo=false mimalloc=true
                 install path: /Users/redacted/projects/platform-orchestration/.venv/lib/python3.10/site-packages/pydantic
               python version: 3.10.10 (main, May  8 2023, 10:19:34) [Clang 14.0.3 (clang-1403.0.22.14.1)]
                     platform: macOS-13.4.1-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

Selected Assignee: @dmontagu

[davidhewitt]

Possibly related: my work in pydantic/pydantic-core#867 will improve the smart unions a bit further. I'd propose that a better behaviour here with "smart unions" would be that this validates as a str in both cases , because the input is a str. I think this should be the case once my PR merges.

I would support adding a way to get back to "dumb" left-to-right validation, even if the smart behaviour remains the default. If there isn't a known way already (@samuelcolvin @dmontagu @adriangb ?), I'd be tempted to add smart_union back to Field, but give it a default of True this time.

[samuelcolvin]

I agree with @davidhewitt, but I would suggest adding dumb_union (default False) since:

1. Better to have False as the default
2. The behavior will inevitably be somewhat different to v1, so a new name makes more sense.

[davidhewitt]

Pydantic 2.2 (releasing very soon) will include Field(union_mode='left_to_right') to opt out of the smart validation.