Elements of Union that have the same structure aren't deserialized correctly

[karlicoss]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

I atttached a snippet of code that illustrates the issue.

Basically, a Union has dataclasses that have exactly the same field names. When dumping to json, pydantic just dumps it assentially as dictionary, erasing any type information. As a result, during deserializing, it just picks the first compatible element of the union, so the type Country gets deserialized as City:

original object   : Country(name='UK')
as json           : {'name': 'UK'}
restored from json: City(name='UK')
Traceback (most recent call last):
  File "/project/pyd.py", line 24, in <module>
    assert obj == restored
AssertionError

I've read the documentation on discriminated Unions, but the issue in my case is that I don't control the types I need to serialize, so I can't add a discriminator field.

Ideally I'd like to have something that discriminates based on class name, e.g. it would dump Country(name='UK') as {'__type__': '__main__.Country', name='UK'} or something like that.
E.g. similar to what cattrs doesn, it allows 'configuring' to discriminate a Union, although it's a bit clunky if you don't know all types in advance, I'd rather have it as an option for TypeAdapter or something like that so it would apply to all Unions.

Is that possible with Pydantic?

Possibly relevant issues:

• replace ultra_strict with new union implementation pydantic-core#867
• Unexpected evaluation of Union #7085
• Add support for externally tagged unions #5244

Example Code

from dataclasses import dataclass
from pydantic import TypeAdapter


@dataclass
class City:
    name: str

@dataclass
class Country:
    name: str

Type = City | Country
obj: Type = Country(name='UK')

Schema = TypeAdapter(Type)
json = Schema.dump_python(obj)
restored = Schema.validate_python(json)

print("original object   :", obj)
print("as json           :", json)
print("restored from json:", restored)

assert obj == restored

Python, Pydantic & OS Version

pydantic version: 2.3.0
        pydantic-core version: 2.6.3
          pydantic-core build: profile=release pgo=true
                 install path: /home/adhoc/.local/lib/python3.10/site-packages/pydantic
               python version: 3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]
                     platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
     optional deps. installed: ['typing-extensions']

[samuelcolvin]

Well the fundamental problem here is that Schema.dump_python when dumping a dataclass converts it to a dictionary, and the dictionaries created for City and Country are the same. You might be able to use a custom serializer to change how City and Country are serialized.

---

While your issue isn't that clear about what you want, I think this case or similar cases could be supported by a discriminator function - this is supported in pydantic-core, but doesn't seem to be well supported in Pydantic V2. I create an issue for that now - see #7462.