Spec: Define "type expression", "annotation expression", "type qualif…

…ier", "special form" (#1693)

As discussed in https://discuss.python.org/t/basic-terminology-for-types-and-type-forms/46741

Co-authored-by: Sebastian Rittau <srittau@rittau.biz>
Co-authored-by: Eric Traut <eric@traut.com>
Co-authored-by: Carl Meyer <carl@oddbird.net>

docs/spec/annotations.rst

@@ -55,42 +55,166 @@ decorators ``@property``, ``@staticmethod`` and ``@classmethod``.
 
 .. _valid-types:
 
-Valid type expression forms
----------------------------
-
-Type hints may be built-in classes (including those defined in
-standard library or third-party extension modules), abstract base
-classes, types available in the ``types`` module, and user-defined
-classes (including those defined in the standard library or
-third-party modules).
-
-While annotations are normally the best format for type hints,
-there are times when it is more appropriate to represent them
-by a special comment, or in a separately distributed stub
-file.  (See below for examples.)
+Type and annotation expressions
+-------------------------------
+
+The terms *type expression* and *annotation expression* denote specific
+subsets of Python expressions that are used in the type system.  All
+type expressions are also annotation expressions, but not all annotation
+expressions are type expressions.
+
+.. _`type-expression`:
+
+A *type expression* is any expression that validly expresses a type. Type
+expressions are always acceptable in annotations and also in various other
+places. Specifically, type expressions are used in the following locations:
+
+* In a type annotation (always as part of an annotation expression)
+* The first argument to :ref:`cast() <cast>`
+* The second argument to :ref:`assert_type() <assert-type>`
+* The bounds and constraints of a ``TypeVar`` (whether created through the
+  old syntax or the native syntax in Python 3.12)
+* The definition of a type alias (whether created through the ``type`` statement,
+  the old assignment syntax, or the ``TypeAliasType`` constructor)
+* The type arguments of a generic class (which may appear in a base class
+  or in a constructor call)
+* The definitions of fields in the functional forms for creating
+  :ref:`TypedDict <typeddict>` and :ref:`NamedTuple <namedtuple>` types
+* The base type in the definition of a :ref:`NewType <newtype>`
+
+.. _`annotation-expression`:
+
+An *annotation expression* is an expression that is acceptable to use in
+an annotation context (a function parameter annotation, function return
+annotation, or variable annotation). Generally, an annotation expression
+is a type expression, optionally surrounded by one or more :term:`type qualifiers <type qualifier>`
+or by `Annotated`. Each type qualifier is valid only in some contexts. Note
+that while annotation expressions are the only expressions valid as type
+annotations in the type system, the Python language itself makes no such
+restriction: any expression is allowed.
 
 Annotations must be valid expressions that evaluate without raising
-exceptions at the time the function is defined (but see below for
-forward references).
-
-Annotations should be kept simple or static analysis tools may not be
-able to interpret the values. For example, dynamically computed types
-are unlikely to be understood.  (This is an
-intentionally somewhat vague requirement; specific inclusions and
-exclusions may be added in the future as warranted by the discussion.)
-
-In addition to the above, the following special constructs defined
-below may be used: ``None``, ``Any``, ``Union``, ``Tuple``,
-``Callable``, all ABCs and stand-ins for concrete classes exported
-from ``typing`` (e.g. ``Sequence`` and ``Dict``), type variables, and
-type aliases.
+exceptions at the time the function is defined (but see :ref:`forward-references`).
+
+.. _`expression-grammar`:
+
+The following grammar describes the allowed elements of type and annotation expressions:
+
+.. productionlist:: expression-grammar
+    annotation_expression: <Required> '[' `annotation_expression` ']'
+                         : | <NotRequired> '[' `annotation_expression` ']'
+                         : | <ReadOnly> '[' `annotation_expression`']'
+                         : | <ClassVar> '[' `annotation_expression`']'
+                         : | <Final> '[' `annotation_expression`']'
+                         : | <InitVar> '[' `annotation_expression` ']'
+                         : | <Annotated> '[' `annotation_expression` ','
+                         :               expression (',' expression)* ']'
+                         : | <TypeAlias>
+                         :       (valid only in variable annotations)
+                         : | `unpacked`
+                         :       (valid only for *args annotations)
+                         : | <Unpack> '[' name ']'
+                         :       (where name refers to an in-scope TypedDict;
+                         :        valid only in **kwargs annotations)
+                         : | `string_annotation`
+                         :       (must evaluate to a valid `annotation_expression`)
+                         : | name '.' 'args'
+                         :      (where name must be an in-scope ParamSpec;
+                         :       valid only in *args annotations)
+                         : | name '.' 'kwargs'
+                         :       (where name must be an in-scope ParamSpec;
+                         :        valid only in **kwargs annotations)
+                         : | `type_expression`
+    type_expression: <Any>
+                   : | <Self>
+                   :       (valid only in some contexts)
+                   : | <LiteralString>
+                   : | <NoReturn>
+                   : | <Never>
+                   : | <None>
+                   : | name
+                   :       (where name must refer to a valid in-scope class,
+                   :        type alias, or TypeVar)
+                   : | name '[' (`maybe_unpacked` | `type_expression_list`)
+                   :        (',' (`maybe_unpacked` | `type_expression_list`))* ']'
+                   :       (the `type_expression_list` form is valid only when
+                   :        specializing a ParamSpec)
+                   : | name '[' '(' ')' ']'
+                   :       (denoting specialization with an empty TypeVarTuple)
+                   : | <Literal> '[' expression (',' expression) ']'
+                   :       (see documentation for Literal for restrictions)
+                   : | `type_expression` '|' `type_expression`
+                   : | <Optional> '[' `type_expression` ']'
+                   : | <Union> '[' `type_expression` (',' `type_expression`)* ']'
+                   : | <type> '[' <Any> ']'
+                   : | <type> '[' name ']'
+                   :       (where name must refer to a valid in-scope class
+                   :        or TypeVar)
+                   : | <Callable> '[' '...' ',' `type_expression` ']'
+                   : | <Callable> '[' name ',' `type_expression` ']'
+                   :       (where name must be a valid in-scope ParamSpec)
+                   : | <Callable> '[' <Concatenate> '[' (`type_expression` ',')+
+                   :              (name | '...') ']' ',' `type_expression` ']'
+                   :       (where name must be a valid in-scope ParamSpec)
+                   : | <Callable> '[' '[' `maybe_unpacked` (',' `maybe_unpacked`)*
+                   :              ']' ',' `type_expression` ']'
+                   : | `tuple_type_expression`
+                   : | <Annotated> '[' `type_expression` ','
+                   :               expression (',' expression)* ']'
+                   : | <TypeGuard> '[' `type_expression` ']'
+                   :       (valid only in some contexts)
+                   : | <TypeIs> '[' `type_expression` ']'
+                   :       (valid only in some contexts)
+                   : | `string_annotation`
+                   :       (must evaluate to a valid `type_expression`)
+    maybe_unpacked: `type_expression` | `unpacked`
+    unpacked: '*' `unpackable`
+            : | <Unpack> '[' `unpackable` ']'
+    unpackable: `tuple_type_expression``
+              : | name
+              :       (where name must refer to an in-scope TypeVarTuple)
+    tuple_type_expression: <tuple> '[' '(' ')' ']'
+                         :      (representing an empty tuple)
+                         : | <tuple> '[' `type_expression` ',' '...' ']'
+                         :       (representing an arbitrary-length tuple)
+                         : | <tuple> '[' `maybe_unpacked` (',' `maybe_unpacked`)* ']'
+    string_annotation: string
+                     :     (must be a string literal that is parsable
+                     :      as Python code; see "String annotations")
+    type_expression_list: '[' `type_expression` (',' `type_expression`)* ']'
+                        : | '[' ']'
+
+Notes:
+
+* The grammar assumes the code has already been parsed as Python code, and
+  loosely follows the structure of the AST. Syntactic details like comments
+  and whitespace are ignored.
+
+* ``<Name>`` refers to a :term:`special form`. Most special forms must be imported
+  from :py:mod:`typing` or ``typing_extensions``, except for ``None``,  ``InitVar``,
+  ``type``, and ``tuple``. The latter two have aliases in :py:mod:`typing`: :py:class:`typing.Type`
+  and :py:class:`typing.Tuple`.  ``InitVar`` must be imported from :py:mod:`dataclasses`.
+  ``Callable`` may be imported from either :py:mod:`typing` or :py:mod:`collections.abc`.
+  Special forms may be aliased
+  (e.g., ``from typing import Literal as L``), and they may be referred to by a
+  qualified name (e.g., ``typing.Literal``). There are other special forms that are not
+  acceptable in any annotation or type expression, including ``Generic``, ``Protocol``,
+  and ``TypedDict``.
+
+* Any leaf denoted as ``name`` may also be a qualified name (i.e., ``module '.' name``
+  or ``package '.' module '.' name``, with any level of nesting).
+
+* Comments in parentheses denote additional restrictions not expressed in the
+  grammar, or brief descriptions of the meaning of a construct.
+
+.. _ `string-annotations`:
 
 .. _`forward-references`:
 
-Forward references
+String annotations
 ------------------
 
-When a type hint contains names that have not been defined yet, that
+When a type hint cannot be evaluated at runtime, that
 definition may be expressed as a string literal, to be resolved later.
 
 A situation where this occurs commonly is the definition of a
@@ -118,7 +242,7 @@ same namespaces in which default arguments to the same function would
 be evaluated.
 
 Moreover, the expression should be parseable as a valid type hint, i.e.,
-it is constrained by the rules from the section on :ref:`valid-types`.
+it is constrained by the rules from :ref:`the expression grammar <expression-grammar>`.
 
 If a triple quote is used, the string should be parsed as though it is
 implicitly surrounded by parentheses. This allows newline characters to be

docs/spec/class-compat.rst

@@ -10,10 +10,10 @@ Class type compatibility
 
 (Originally specified in :pep:`526`.)
 
-A covariant type ``ClassVar[T_co]`` exists in the ``typing``
+A :term:`type qualifier` ``ClassVar[T]`` exists in the :py:mod:`typing`
 module. It accepts only a single argument that should be a valid type,
 and is used to annotate class variables that should not be set on class
-instances. This restriction is ensured by static checkers,
+instances. This restriction is enforced by static checkers,
 but not at runtime.
 
 Type annotations can be used to annotate class and instance variables
@@ -26,7 +26,7 @@ in ``__init__`` or ``__new__``. The syntax is as follows::
       damage: int                           # instance variable without default
       stats: ClassVar[dict[str, int]] = {}  # class variable
 
-Here ``ClassVar`` is a special class defined by the typing module that
+Here ``ClassVar`` is a :term:`special form` defined by the :py:mod:`typing` module that
 indicates to the static type checker that this variable should not be
 set on instances.
 

docs/spec/directives.rst

@@ -18,6 +18,8 @@ should emit an error if the value is not of the specified type::
         assert_type(name, str)  # OK, inferred type of `name` is `str`
         assert_type(name, int)  # type checker error
 
+The second argument must be a valid :term:`type expression`.
+
 .. _`reveal-type`:
 
 ``reveal_type()``
@@ -79,7 +81,8 @@ Some type checkers may not be able to infer that the type of
 ``a[index]`` is ``str`` and only infer ``object`` or ``Any``, but we
 know that (if the code gets to that point) it must be a string.  The
 ``cast(t, x)`` call tells the type checker that we are confident that
-the type of ``x`` is ``t``.  At runtime a cast always returns the
+the type of ``x`` is ``t``. ``t`` must be a valid :term:`type expression`.
+At runtime a cast always returns the
 expression unchanged -- it does not check the type, and it does not
 convert or coerce the value.
 

docs/spec/glossary.rst

@@ -7,6 +7,11 @@ This section defines a few terms that may be used elsewhere in the specification
 
 .. glossary::
 
+   annotation expression
+      An expression that is valid to use within an annotation. This is usually a
+      :term:`type expression`, sometimes with additional :term:`type qualifiers <type qualifier>`.
+      See :ref:`"Type and annotation expression" <annotation-expression>` for details.
+
    distribution
       The packaged file which is used to publish and distribute
       a release. (:pep:`426`)
@@ -25,6 +30,26 @@ This section defines a few terms that may be used elsewhere in the specification
       While most distributions are named after the one package they install, some
       distributions install multiple packages.)
 
+   special form
+      A special form is an object that has a special meaning within the type system,
+      comparable to a keyword in the language grammar. Examples include ``Any``,
+      ``Generic``,  ``Literal``, and ``TypedDict``. Special forms can often but not always be used
+      within :ref:`type expressions <type-expression>`. Special forms can usually
+      be imported from the :py:mod:`typing` module or equivalently from ``typing_extensions``,
+      but some special forms are placed in other modules.
+
    stub
       A file containing only type information, empty of runtime code
       (the filename ends in ``.pyi``). See :ref:`stub-files`.
+
+   type expression
+      An expression that represents a type. The type system requires the use of type
+      expressions within :term:`annotation expression` and also in several other contexts.
+      See :ref:`"Type and annotation expression" <type-expression>` for details.
+
+   type qualifier
+      A type qualifier is a :term:`special form` that qualifies a :term:`type expression` to
+      form an :term:`annotation expression`. For example, the type qualifier :ref:`Final <uppercase-final>`
+      can be used around a type to indicate that the annotated value may not be overridden or modified.
+      This term is also used for other special forms that modify a type, but using a different
+      syntactic context, such as the `@final <at-final>` decorator.

docs/spec/historical.rst

@@ -280,10 +280,10 @@ and type checkers may warn if they are used.
 --------------------------
 
 Before Python 3.10 (:pep:`604`), Python did not support the ``|`` operator
-for creating unions of types. Therefore, the ``typing.Union`` special form can also
+for creating unions of types. Therefore, the ``typing.Union`` :term:`special form` can also
 be used to create union types. Type checkers should treat the two forms as equivalent.
 
-In addition, the ``Optional`` special form provides a shortcut for a union with ``None``.
+In addition, the ``Optional`` :term:`special form` is equivalent to a union with ``None``.
 
 Examples:
 

docs/spec/namedtuples.rst

@@ -1,3 +1,5 @@
+.. _`namedtuple`:
+
 Named Tuples
 ============
 

docs/spec/narrowing.rst

@@ -13,14 +13,14 @@ TypeGuard
 
 (Originally specified in :pep:`647`.)
 
-The symbol ``TypeGuard``, exported from the ``typing`` module, is a special form
+The symbol ``TypeGuard``, exported from the ``typing`` module, is a :term:`special form`
 that accepts a single type argument. It is used to annotate the return type of a
 user-defined type guard function. Return statements within a type guard function
 should return bool values, and type checkers should verify that all return paths
 return a bool.
 
 ``TypeGuard`` is also valid as the return type of a callable, for example
-in callback protocols and in the ``Callable`` special form. In these
+in callback protocols and in the ``Callable`` :term:`special form`. In these
 contexts, it is treated as a subtype of bool. For example, ``Callable[..., TypeGuard[int]]``
 is assignable to ``Callable[..., bool]``.
 
@@ -115,7 +115,7 @@ TypeIs
 
 (Originally specified in :pep:`742`.)
 
-The special form ``TypeIs`` is similar in usage, behavior, and runtime
+The :term:`special form` ``TypeIs`` is similar in usage, behavior, and runtime
 implementation as ``TypeGuard``.
 
 ``TypeIs`` accepts a single type argument and can be used as the return type
@@ -201,7 +201,7 @@ It is an error to narrow to a type that is not consistent with the input type::
         ...
 
 ``TypeIs`` is also valid as the return type of a callable, for example
-in callback protocols and in the ``Callable`` special form. In these
+in callback protocols and in the ``Callable`` :term:`special form`. In these
 contexts, it is treated as a subtype of bool. For example, ``Callable[..., TypeIs[int]]``
 is assignable to ``Callable[..., bool]``.
 

docs/spec/protocol.rst

@@ -41,7 +41,7 @@ protocol members.
 Defining a protocol
 ^^^^^^^^^^^^^^^^^^^
 
-Protocols are defined by including a special new class ``typing.Protocol``
+Protocols are defined by including a :term:`special form` ``typing.Protocol``
 (an instance of ``abc.ABCMeta``) in the base classes list, typically
 at the end of the list. Here is a simple example::
 

docs/spec/qualifiers.rst

@@ -3,6 +3,14 @@
 Type qualifiers
 ===============
 
+This chapter describes the behavior of some :term:`type qualifiers <type qualifier>`.
+Additional type qualifiers are covered in other chapters:
+
+* :ref:`ClassVar <classvar>`
+* :ref:`NotRequired <notrequired>`
+* :ref:`ReadOnly <readonly>`
+* :ref:`Required <required>`
+
 .. _`at-final`:
 
 ``@final``
@@ -67,7 +75,7 @@ It is an error to use ``@final`` on a non-method function.
 
 (Originally specified in :pep:`591`.)
 
-The ``typing.Final`` type qualifier is used to indicate that a
+The ``typing.Final`` :term:`type qualifier` is used to indicate that a
 variable or attribute should not be reassigned, redefined, or overridden.
 
 Syntax
@@ -247,7 +255,7 @@ details of the syntax:
 
     V == Annotated[list[tuple[int, int]], MaxLen(10)]
 
-* As with most special forms, ``Annotated`` is not type compatible with
+* As with most :term:`special forms <special form>`, ``Annotated`` is not type compatible with
   ``type`` or ``type[T]``::
 
     v1: type[int] = Annotated[int, ""]  # Type error