python==3.9, pydantic>=2.1.0 regression in support for ParamSpec, RecursionError in model definition

[ringohoffman]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

This bug does not appear in pydantic==2.1.0 in python 3.8. This seems to be related to the introduction of __instancecheck__ in python 3.9.

In pydantic<=2.0.3 in python==3.9, there was no problem using a field that was a Generic w.r.t. a ParamSpec, but after pydantic==2.1.0, a RecursionError is now raised when defining a BaseModel with a field that uses a ParamSpec (or a Generic TypeAlias that uses a ParamSpec).

│ /Users/ringo/opt/anaconda3/envs/glass39/lib/python3.9/site-packages/pydantic/_internal/_generics │
│ .py:363 in <genexpr>                                                                             │
│                                                                                                  │
│   360 │   # Handle special case for typehints that can have lists as arguments.                  │
│   361 │   # `typing.Callable[[int, str], int]` is an example for this.                           │
│   362 │   if isinstance(type_, (List, list)):                                                    │
│ ❱ 363 │   │   if any(has_instance_in_type(element, isinstance_target) for element in type_):     │
│   364 │   │   │   return True                                                                    │
│   365 │                                                                                          │
│   366 │   return False                                                                           │
│                                                                                                  │
│ /Users/ringo/opt/anaconda3/envs/glass39/lib/python3.9/site-packages/pydantic/_internal/_generics │
│ .py:363 in has_instance_in_type                                                                  │
│                                                                                                  │
│   360 │   # Handle special case for typehints that can have lists as arguments.                  │
│   361 │   # `typing.Callable[[int, str], int]` is an example for this.                           │
│   362 │   if isinstance(type_, (List, list)):                                                    │
│ ❱ 363 │   │   if any(has_instance_in_type(element, isinstance_target) for element in type_):     │
│   364 │   │   │   return True                                                                    │
│   365 │                                                                                          │
│   366 │   return False                                                                           │
│                                                                                                  │
│ /Users/ringo/opt/anaconda3/envs/glass39/lib/python3.9/site-packages/pydantic/_internal/_generics │
│ .py:363 in <genexpr>                                                                             │
│                                                                                                  │
│   360 │   # Handle special case for typehints that can have lists as arguments.                  │
│   361 │   # `typing.Callable[[int, str], int]` is an example for this.                           │
│   362 │   if isinstance(type_, (List, list)):                                                    │
│ ❱ 363 │   │   if any(has_instance_in_type(element, isinstance_target) for element in type_):     │
│   364 │   │   │   return True                                                                    │
│   365 │                                                                                          │
│   366 │   return False                                                                           │
│                                                                                                  │
│ /Users/ringo/opt/anaconda3/envs/glass39/lib/python3.9/site-packages/pydantic/_internal/_generics │
│ .py:362 in has_instance_in_type                                                                  │
│                                                                                                  │
│   359 │                                                                                          │
│   360 │   # Handle special case for typehints that can have lists as arguments.                  │
│   361 │   # `typing.Callable[[int, str], int]` is an example for this.                           │
│ ❱ 362 │   if isinstance(type_, (List, list)):                                                    │
│   363 │   │   if any(has_instance_in_type(element, isinstance_target) for element in type_):     │
│   364 │   │   │   return True                                                                    │
│   365                                                                                            │
│                                                                                                  │
│ /Users/ringo/opt/anaconda3/envs/glass39/lib/python3.9/typing.py:719 in __instancecheck__         │
│                                                                                                  │
│    716 │   │   else:                                                                             │
│    717 │   │   │   setattr(self.__origin__, attr, val)                                           │
│    718 │                                                                                         │
│ ❱  719 │   def __instancecheck__(self, obj):                                                     │
│    720 │   │   return self.__subclasscheck__(type(obj))                                          │
│    721 │                                                                                         │
│    722 │   def __subclasscheck__(self, cls):                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RecursionError: maximum recursion depth exceeded

Example Code

from typing import Callable, Generic, TypeVar

import pydantic
from typing_extensions import ParamSpec

T = TypeVar("T")
P = ParamSpec("P")


class MyGenericTypeVarClass(Generic[T]):
    def __init__(self, item: T) -> None:
        super().__init__()


class MyGenericParamSpecClass(Generic[P]):
    def __init__(self, func: Callable[P, None], *args: P.args, **kwargs: P.kwargs) -> None:
        super().__init__()


class TypeVarGenericModel(pydantic.BaseModel, Generic[T]):  # OK
    my_generic: MyGenericTypeVarClass[T]

    model_config = pydantic.ConfigDict(arbitrary_types_allowed=True)


class ParamSpecGenericModel(pydantic.BaseModel, Generic[P]):  # RecursionError
    my_generic: MyGenericParamSpecClass[P]

    model_config = pydantic.ConfigDict(arbitrary_types_allowed=True)

Python, Pydantic & OS Version

pydantic version: 2.1.0
        pydantic-core version: 2.4.0
          pydantic-core build: profile=release pgo=false mimalloc=true
                 install path: /Users/ringo/opt/anaconda3/envs/glass39/lib/python3.9/site-packages/pydantic
               python version: 3.9.16 (main, Mar  8 2023, 04:29:44)  [Clang 14.0.6 ]
                     platform: macOS-10.16-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

Selected Assignee: @adriangb

[dmontagu]

We haven't implemented support for ParamSpec, TypeVarTuple, or variadic generics at all yet; if it was working before, that was a happy accident.

If you would like to provide some unit tests that would be appreciated, I think it will probably be time to add proper support for this stuff soon.

[dmontagu]

Also, I get an error in your code above in python<=3.9; python 3.10 and 3.11 work fine.

[ringohoffman]

Hmm... seems like its just 3.9 then... 😅

[dmontagu]

Okay, I see the problem is that the ParamSpec P satisfies P in P 🙄. I think we can at least fix the recursion error without too much issue.

[dmontagu]

@ringohoffman could you check if #6923 resolves your issue? It at least doesn't cause a recursion error but I'm not actually running any code with the relevant model.

[ringohoffman]

Yes! It fixes my problem! Thanks for the lightning fast turnaround!