Add eval_type_backport to handle union operator in older Pythons

[alexmojaki]

Change Summary

Replaces all calls (outside v1) to typing._eval_type with the new typing_extra.eval_type_backport which uses AST modification to convert X | Y to Union[X, Y] when needed. This is only used when an appropriate TypeError is raised so it should usually only kick in on older Python versions. It might also help with cases like #7875 where | is simply unimplemented but probably shouldn't be treated as a complete solution for that kind of thing.

Related issue number

Closes #7873

Checklist

• The pull request title is a good summary of the changes - it will be used in the changelog
• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @hramezani

[alexmojaki]

please review

[sydney-runkle]

@alexmojaki,

Thanks for the contributions. Let's wait until tomorrow until @dmontagu can give some feedback to merge this. I think this would be fine to include in a patch release, effectively as a bug fix, after we release v2.5

[samuelcolvin]

@PrettyWood tried this long ago, I personally think it's too complicated and confusing to be worth while.

[PrettyWood]

@samuelcolvin We couldn't do it before because of cython. Without it it's doable.
But I also think it's very hacky as it's more an extension of python itself than pydantic.

[sydney-runkle]

I personally think it's too complicated and confusing to be worth while.

But I also think it's very hacky as it's more an extension of python itself than pydantic.

Closing this 👍

[alexmojaki]

Well that's disappointing. I questioned whether it was worth it in the original issue and I explained that I was going to use AST modification. @PrettyWood your last comment there seemed OK with this approach, did I misunderstand or was this not what you were expecting?

I personally think it's too complicated and confusing to be worth while.

@samuelcolvin was this referring to @PrettyWood's approach that manipulated tokens, or the AST approach here? I'm honestly surprised to hear that it's considered complicated/confusing/hacky, it's a small amount of pretty straightforward code that's working well and has no issues that I know of.

What if we make it optional and off by default, so you have to e.g. turn on an environment variable to enable it? Everything would still go through eval_type_backport but when it sees the right kind of exception it checks whether the feature is enabled, and if it isn't then it tells the user what they can do to enable it?

[PrettyWood]

I didn't say I'm against it. I see the added value. I just don't know if it's pydantic job to add that kind of logic.

[samuelcolvin]

Hi @alexmojaki sorry to disappoint.

I'm just a bit scared that:

1. It ends up introducing more and more issues, and since we're started to support it, we then have to fix those issues.
2. It's conceptually confusing to both experienced and inexperienced developers - the whole issue of what you can use in what version of python is already confusing, as is runtime use of type hints, adding "but you can actually use newer type hints in older code if you do this special "future" import that isn't actually going to be the future behaviour then reply on this library to backport some, but not all new syntax" definitely adds significant extra complexity.

Still, I have wanted this a number of times, so here's my proposal:

@alexmojaki would you consider packaging this as a new library, pydantic can add it as an optional depedency and try to import it and fall back to the current behaviour when it's not installed, or not required. We can add some explanation and a link to the package's docs to our docs.

That way it's isolated, off by default and the code doesn't need to live in pydantic or have the same quality/maintenance guarentees that pydantic does. WDYT?

[alexmojaki]

Thanks, that's a fair and helpful explanation. I was also thinking that a separate library was a likely solution, I'll do that.

[alexmojaki]

PR with separate library: #8209