Can't pickle generics in certain cases

[strangemonad]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

The reified generic classes only seem to be registered as a value in their modules in some cases. The test case below illustrates the bug.

This may be a dupe of the root cause of #7503.

The core issue seems to be that pickle (at least in python 3.11) first tries to find a for the class to pickle using __qualname__
https://github.com/python/cpython/blob/978fba58aef347de4a1376e525df2dacc7b2fff3/Lib/pickle.py#L1062. This seems to be the case since at least 3.8

But Pydantic only seems to be registering a symbol for the reified generic classes when they're created at a global level

pydantic/pydantic/_internal/_generics.py

Line 152 in 4d7bef6

if called_globally: # create global reference and therefore allow pickling

I'm not close enough to the details of the original implementation by @dmontagu in 53fcbec but is there a reason to only restrict the updating of module references to when things are run at the top-level? What would be the downside of updating the origin.__module__ to always have the reified generic instances?

Some off the cuff thoughts around options:

• eliminating the type var args in the generic subclass __qualname__ will probably have a bunch of other, unintended side-effects. It also means that when you're un-pickling, it would use the raw generic base class
• Whichever way the specialized generic subclass is registered, there needs to be some portion of the un-pickling code path that specifies that same generic type and args e.g. if you're starting a new process and my.module.MyGeneric[T] exists on the import path but you haven't ever defined a MyGeneric[str] un-pickling with fail with a similar error to how pickling currently fails in the test case below

Example Code

import pickle
from typing import Generic, TypeVar

from pydantic import BaseModel

T = TypeVar("T")


class MyGeneric(BaseModel, Generic[T]):
    prop: T


def create_and_pickle():
    m = MyGeneric[str](prop="test")
    print(m.__class__.__qualname__)
    print(pickle.dumps(m))


# If you uncomment this next line, it has the effect of registering a __qualname__ of
# "MyGeneric[str]" in this module and make this test case work.
# MyGeneric[str]

create_and_pickle()

Python, Pydantic & OS Version

pydantic version: 2.6.4
        pydantic-core version: 2.16.3
          pydantic-core build: profile=release pgo=true
                 install path: /Users/shawn/Code/instance-bio/instance/services/web/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.4 (main, Aug 14 2023, 09:41:08) [Clang 14.0.3 (clang-1403.0.22.14.1)]
                     platform: macOS-14.4.1-arm64-arm-64bit
             related packages: typing_extensions-4.11.0 pyright-1.1.321 pydantic-extra-types-2.7.0 fastapi-0.110.2 pydantic-settings-2.2.1
                       commit: unknown

[sydney-runkle]

Hi @strangemonad,

Thanks for your questions. I think this is a duplicate of #8913. Let me know if that explanation is helpful enough for your use case!