🚧 Pydantic plugins experimental implementation (#6820)

Co-authored-by: Samuel Colvin <s@muelcolvin.com>
Co-authored-by: Serge Matveenko <lig@pydantic.dev>

.github/workflows/ci.yml

@@ -178,6 +178,29 @@ jobs:
       - name: test
         run: make test-fastapi
 
+  test-plugin:
+    name: test pydantic plugin
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - uses: pdm-project/setup-pdm@v3
+        with:
+          python-version: '3.11'
+          cache: true
+
+      - name: install deps
+        run: |
+          pdm use -f $PYTHON
+          pdm install -G testing
+
+      - name: install example plugin
+        run: pdm add ./tests/plugin
+
+      - run: pdm run pytest ./tests/plugin
+        env:
+          TEST_PLUGIN: 1
+
   test-mypy:
     name: mypy ${{ matrix.mypy-version }} / ${{ matrix.python-version }}
     runs-on: ubuntu-latest
@@ -340,6 +363,7 @@ jobs:
       - test-memray
       - test-mypy
       - test-fastapi
+      - test-plugin
 
     runs-on: ubuntu-latest
 

docs/api/plugin.md

@@ -0,0 +1,5 @@
+!!! warning "Experimental feature"
+    Plugins support is experimental and is subject to change in minor releases.
+    Developing plugins is not recommended until the feature becomes stable.
+
+::: pydantic.plugin

docs/usage/plugins.md

@@ -0,0 +1,113 @@
+!!! warning "Experimental feature"
+    Plugins support is experimental and is subject to change in minor releases.
+    Developing plugins is not recommended until the feature becomes stable.
+
+Pydantic allows users to create plugins that can be used to extend the functionality of the library.
+
+Plugins are installed via Python entry points. You can read more about entry points in the
+[Entry points specification](https://packaging.python.org/specifications/entry-points/) from the
+Python Packaging Authority.
+
+In case you have a project called `my-pydantic-plugin`, you can create a plugin by adding the following
+to your `pyproject.toml`:
+
+```toml
+[project.entry-points.pydantic]
+my_plugin = "my_pydantic_plugin:plugin"
+```
+
+The entry point group is `pydantic`, `my_plugin` is the name of the plugin, `my_pydantic_plugin` is the module to load plugin object from, and `plugin` is the object name to load.
+
+Plugins are loaded in the order they are found, and the order they are found is not guaranteed.
+
+As a user, you can modify the behavior of the plugin in a `BaseModel` using the `plugin_settings` [Model Config](../usage/model_config.md) argument or
+class keyword argument. This argument takes a dictionary of settings that will be passed to all plugins as is.
+The plugin can then use these settings to modify its behavior. It is recommended for plugins to separate their settings
+into their own dedicates keys in a plugin specific key in the `plugin_settings` dictionary.
+
+```py test="skip"
+from pydantic import BaseModel
+
+
+class Foo(BaseModel, plugin_settings={'my-plugin': {'observe': 'all'}}):
+    ...
+```
+
+## Build a plugin
+
+??? api "API Documentation"
+    [`pydantic.plugin`][pydantic.plugin]<br>
+
+Pydantic provides an API for creating plugins. The API is exposed via the `pydantic.plugin` module.
+
+On your plugin you can _wrap_ the following methods:
+
+* [`validate_python`][pydantic_core.SchemaValidator.validate_python]: Used to validate the data from a Python object.
+* [`validate_json`][pydantic_core.SchemaValidator.validate_json]: Used to validate the data from a JSON string.
+* [`validate_strings`][pydantic_core.SchemaValidator.validate_strings]: Used to validate the data from strings.
+
+For each method, you can implement the following callbacks:
+
+* `on_enter`: Called before the validation of a field starts.
+* `on_success`: Called when the validation of a field succeeds.
+* `on_error`: Called when the validation of a field fails.
+
+Let's see an example of a plugin that _wraps_ the `validate_python` method of the [`SchemaValidator`][pydantic_core.SchemaValidator].
+
+```py
+from typing import Any, Dict, Optional, Union
+
+from pydantic_core import CoreConfig, CoreSchema, ValidationError
+
+from pydantic.plugin import (
+    NewSchemaReturns,
+    PydanticPluginProtocol,
+    ValidatePythonHandlerProtocol,
+)
+
+
+class OnValidatePython(ValidatePythonHandlerProtocol):
+    def on_enter(
+        self,
+        input: Any,
+        *,
+        strict: Optional[bool] = None,
+        from_attributes: Optional[bool] = None,
+        context: Optional[Dict[str, Any]] = None,
+        self_instance: Optional[Any] = None,
+    ) -> None:
+        print(input)
+
+    def on_success(self, result: Any) -> None:
+        print(result)
+
+    def on_error(self, error: ValidationError) -> None:
+        print(error.json())
+
+
+class Plugin(PydanticPluginProtocol):
+    def new_schema_validator(
+        self,
+        schema: CoreSchema,
+        config: Union[CoreConfig, None],
+        plugin_settings: Dict[str, object],
+    ) -> NewSchemaReturns:
+        return OnValidatePython(), None, None
+
+
+plugin = Plugin()
+```
+
+## Using Plugin Settings
+
+Consider that you have a plugin called setting called "observer", then you can use it like this:
+
+```py
+from pydantic import BaseModel
+
+
+class Foo(BaseModel, plugin_settings={'observer': 'all'}):
+    ...
+```
+
+On each validation call, the `plugin_settings` will be passed to a callable registered for the events.

mkdocs.yml

@@ -96,6 +96,7 @@ nav:
   - 'Strict Mode': usage/strict_mode.md
   - 'Conversion Table': usage/conversion_table.md
   - 'Settings Management': usage/pydantic_settings.md
+  - 'Pydantic Plugins': usage/plugins.md
 - Examples:
   - Secrets: examples/secrets.md
 - Error Messages:
@@ -131,6 +132,7 @@ nav:
     - 'Network Types': api/networks.md
     - 'Annotated Handlers': api/annotated_handlers.md
     - 'Version Information': api/version.md
+    - 'Pydantic Plugins': api/plugin.md
   - Pydantic Core:
     - 'pydantic_core': api/pydantic_core.md
     - 'pydantic_core.core_schema': api/pydantic_core_schema.md

pydantic/_internal/_config.py

@@ -75,6 +75,7 @@ class ConfigWrapper:
     protected_namespaces: tuple[str, ...]
     hide_input_in_errors: bool
     defer_build: bool
+    plugin_settings: dict[str, object] | None
     schema_generator: type[GenerateSchema] | None
     json_schema_serialization_defaults_required: bool
     json_schema_mode_override: Literal['validation', 'serialization', None]
@@ -242,6 +243,7 @@ def _context_manager() -> Iterator[None]:
     hide_input_in_errors=False,
     json_encoders=None,
     defer_build=False,
+    plugin_settings=None,
     schema_generator=None,
     json_schema_serialization_defaults_required=False,
     json_schema_mode_override=None,

pydantic/_internal/_dataclasses.py

@@ -20,6 +20,7 @@
 
 from ..errors import PydanticUndefinedAnnotation
 from ..fields import FieldInfo
+from ..plugin._schema_validator import create_schema_validator
 from ..warnings import PydanticDeprecatedSince20
 from . import _config, _decorators, _discriminated_union, _typing_extra
 from ._core_utils import collect_invalid_schemas, simplify_schema_references, validate_core_schema
@@ -168,8 +169,11 @@ def __init__(__dataclass_self__: PydanticDataclass, *args: Any, **kwargs: Any) -
     # __pydantic_decorators__ and __pydantic_fields__ should already be set
     cls = typing.cast('type[PydanticDataclass]', cls)
     # debug(schema)
+
     cls.__pydantic_core_schema__ = schema = validate_core_schema(schema)
-    cls.__pydantic_validator__ = validator = SchemaValidator(schema, core_config)
+    cls.__pydantic_validator__ = validator = create_schema_validator(
+        schema, core_config, config_wrapper.plugin_settings
+    )
     cls.__pydantic_serializer__ = SchemaSerializer(schema, core_config)
 
     if config_wrapper.validate_assignment:

pydantic/_internal/_model_construction.py

@@ -9,11 +9,12 @@
 from types import FunctionType
 from typing import Any, Callable, Generic, Mapping
 
-from pydantic_core import PydanticUndefined, SchemaSerializer, SchemaValidator
+from pydantic_core import PydanticUndefined, SchemaSerializer
 from typing_extensions import dataclass_transform, deprecated
 
 from ..errors import PydanticUndefinedAnnotation, PydanticUserError
 from ..fields import Field, FieldInfo, ModelPrivateAttr, PrivateAttr
+from ..plugin._schema_validator import create_schema_validator
 from ..warnings import PydanticDeprecatedSince20
 from ._config import ConfigWrapper
 from ._core_utils import collect_invalid_schemas, simplify_schema_references, validate_core_schema
@@ -495,7 +496,7 @@ def complete_model_class(
 
     # debug(schema)
     cls.__pydantic_core_schema__ = schema = validate_core_schema(schema)
-    cls.__pydantic_validator__ = SchemaValidator(schema, core_config)
+    cls.__pydantic_validator__ = create_schema_validator(schema, core_config, config_wrapper.plugin_settings)
     cls.__pydantic_serializer__ = SchemaSerializer(schema, core_config)
     cls.__pydantic_complete__ = True
 

pydantic/_internal/_validate_call.py

@@ -8,6 +8,7 @@
 import pydantic_core
 
 from ..config import ConfigDict
+from ..plugin._schema_validator import create_schema_validator
 from . import _discriminated_union, _generate_schema, _typing_extra
 from ._config import ConfigWrapper
 from ._core_utils import simplify_schema_references, validate_core_schema
@@ -66,7 +67,7 @@ def __init__(self, function: Callable[..., Any], config: ConfigDict | None, vali
         self.__pydantic_core_schema__ = schema = schema
         core_config = config_wrapper.core_config(self)
         schema = _discriminated_union.apply_discriminators(schema)
-        self.__pydantic_validator__ = pydantic_core.SchemaValidator(schema, core_config)
+        self.__pydantic_validator__ = create_schema_validator(schema, core_config, config_wrapper.plugin_settings)
 
         if self._validate_return:
             return_type = (

pydantic/config.py

@@ -192,6 +192,12 @@ class without an annotation and has a type that is not in this tuple (or otherwi
     [`Model.model_rebuild(_types_namespace=...)`][pydantic.BaseModel.model_rebuild]. Defaults to False.
     """
 
+    plugin_settings: dict[str, object] | None
+    """A `dict` of settings for plugins. Defaults to `None`.
+
+    See [Pydantic Plugins](../usage/plugins.md) for details.
+    """
+
     schema_generator: type[_GenerateSchema] | None
     """
     A custom core schema generator class to use when generating JSON schemas.

pydantic/main.py

@@ -740,7 +740,13 @@ def __getattr__(self, item: str) -> Any:
                 except KeyError as exc:
                     raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}') from exc
             else:
-                pydantic_extra = object.__getattribute__(self, '__pydantic_extra__')
+                # `__pydantic_extra__` can fail to be set if the model is not yet fully initialized.
+                # See `BaseModel.__repr_args__` for more details
+                try:
+                    pydantic_extra = object.__getattribute__(self, '__pydantic_extra__')
+                except AttributeError:
+                    pydantic_extra = None
+
                 if pydantic_extra is not None:
                     try:
                         return pydantic_extra[item]
@@ -898,11 +904,16 @@ def __repr_args__(self) -> _repr.ReprArgs:
             field = self.model_fields.get(k)
             if field and field.repr:
                 yield k, v
+
         # `__pydantic_extra__` can fail to be set if the model is not yet fully initialized.
         # This can happen if a `ValidationError` is raised during initialization and the instance's
         # repr is generated as part of the exception handling. Therefore, we use `getattr` here
         # with a fallback, even though the type hints indicate the attribute will always be present.
-        pydantic_extra = getattr(self, '__pydantic_extra__', None)
+        try:
+            pydantic_extra = object.__getattribute__(self, '__pydantic_extra__')
+        except AttributeError:
+            pydantic_extra = None
+
         if pydantic_extra is not None:
             yield from ((k, v) for k, v in pydantic_extra.items())
         yield from ((k, getattr(self, k)) for k, v in self.model_computed_fields.items() if v.repr)