Make datetime generic over tzinfo

[srittau]

No description provided.

[srittau]

This is a trial to judge the impact of such a change that could prevent runtime errors due to mixing timezone-aware and unaware datetimes.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

steam.py (https://github.com/Gobot1234/steam.py)
- steam/app.py:461: error: Incompatible types in assignment (expression has type "None", variable has type "datetime")  [assignment]
+ steam/app.py:461: error: Incompatible types in assignment (expression has type "None", variable has type "datetime[Any]")  [assignment]
- steam/abc.py:668: error: Unsupported operand types for > ("datetime" and "None")  [operator]
+ steam/abc.py:668: error: No overload variant of "__gt__" of "datetime" matches argument type "None"  [operator]
+ steam/abc.py:668: note: Possible overload variants:
+ steam/abc.py:668: note:     def __gt__(self, datetime[tzinfo] | datetime[tzinfo | None], /) -> bool
+ steam/abc.py:668: note:     def __gt__(self, datetime[None] | datetime[tzinfo | None], /) -> bool
+ steam/abc.py:668: note:     def __gt__(self, datetime[Any], /) -> NoReturn
- steam/abc.py:668: note: Left operand is of type "datetime | None"
+ steam/abc.py:668: note: Left operand is of type "datetime[Any] | None"
- steam/abc.py:668: error: Unsupported operand types for < ("datetime" and "None")  [operator]
+ steam/abc.py:668: error: No overload variant of "__lt__" of "datetime" matches argument type "None"  [operator]
+ steam/abc.py:668: note:     def __lt__(self, datetime[tzinfo] | datetime[tzinfo | None], /) -> bool
+ steam/abc.py:668: note:     def __lt__(self, datetime[None] | datetime[tzinfo | None], /) -> bool
+ steam/abc.py:668: note:     def __lt__(self, datetime[Any], /) -> NoReturn
- steam/abc.py:668: note: Right operand is of type "datetime | None"
+ steam/abc.py:668: note: Right operand is of type "datetime[Any] | None"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/_libs/tslibs/timestamps.pyi:148: error: Signatures of "__le__" of "Timestamp" and "__ge__" of "datetime[Any]" are unsafely overlapping  [misc]
+ pandas/_libs/tslibs/timestamps.pyi:148: note: Error code "misc" not covered by "type: ignore" comment
+ pandas/_libs/tslibs/timestamps.pyi:149: error: Signatures of "__lt__" of "Timestamp" and "__gt__" of "datetime[Any]" are unsafely overlapping  [misc]
+ pandas/_libs/tslibs/timestamps.pyi:149: note: Error code "misc" not covered by "type: ignore" comment
+ pandas/_libs/tslibs/timestamps.pyi:150: error: Signatures of "__ge__" of "Timestamp" and "__le__" of "datetime[Any]" are unsafely overlapping  [misc]
+ pandas/_libs/tslibs/timestamps.pyi:150: note: Error code "misc" not covered by "type: ignore" comment
+ pandas/_libs/tslibs/timestamps.pyi:151: error: Signatures of "__gt__" of "Timestamp" and "__lt__" of "datetime[Any]" are unsafely overlapping  [misc]
+ pandas/_libs/tslibs/timestamps.pyi:151: note: Error code "misc" not covered by "type: ignore" comment

openlibrary (https://github.com/internetarchive/openlibrary)
+ openlibrary/tests/core/test_unmarshal.py:32: error: Unused "type: ignore" comment  [unused-ignore]
+ openlibrary/tests/core/test_unmarshal.py: note: In function "parse_datetime":
+ openlibrary/tests/core/test_unmarshal.py:32: error: No overload variant of "datetime" matches argument type "Generator[int, None, None]"  [call-overload]
+ openlibrary/tests/core/test_unmarshal.py:32: note: Error code "call-overload" not covered by "type: ignore" comment
+ openlibrary/tests/core/test_unmarshal.py:32: note: Possible overload variants:
+ openlibrary/tests/core/test_unmarshal.py:32: note:     def [_TzInfoT] __new__(cls, year: SupportsIndex, month: SupportsIndex, day: SupportsIndex, hour: SupportsIndex = ..., minute: SupportsIndex = ..., second: SupportsIndex = ..., microsecond: SupportsIndex = ..., tzinfo: None = ..., *, fold: int = ...) -> datetime[None]
+ openlibrary/tests/core/test_unmarshal.py:32: note:     def [_TzInfoT] __new__(cls, year: SupportsIndex, month: SupportsIndex, day: SupportsIndex, hour: SupportsIndex = ..., minute: SupportsIndex = ..., second: SupportsIndex = ..., microsecond: SupportsIndex = ..., *, tzinfo: tzinfo, fold: int = ...) -> datetime[tzinfo]
+ openlibrary/tests/core/test_unmarshal.py:32: note:     def [_TzInfoT] __new__(cls, year: SupportsIndex, month: SupportsIndex, day: SupportsIndex, hour: SupportsIndex, minute: SupportsIndex, second: SupportsIndex, microsecond: SupportsIndex, tzinfo: tzinfo, *, fold: int = ...) -> datetime[tzinfo]

tornado (https://github.com/tornadoweb/tornado)
+ tornado/test/locale_test.py:100: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:100: error: Argument 1 to "format_date" of "Locale" has incompatible type "timedelta"; expected "Union[int, float, datetime[Any]]"  [arg-type]
+ tornado/test/locale_test.py:106: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:106: error: Argument 1 to "format_date" of "Locale" has incompatible type "timedelta"; expected "Union[int, float, datetime[Any]]"  [arg-type]
+ tornado/test/locale_test.py:112: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:112: error: Argument 1 to "format_date" of "Locale" has incompatible type "timedelta"; expected "Union[int, float, datetime[Any]]"  [arg-type]
+ tornado/test/locale_test.py:119: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:119: error: Argument 1 to "format_date" of "Locale" has incompatible type "timedelta"; expected "Union[int, float, datetime[Any]]"  [arg-type]
+ tornado/test/locale_test.py:126: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:126: error: Incompatible types in assignment (expression has type "timedelta", variable has type "datetime[None]")  [assignment]
+ tornado/test/locale_test.py:132: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:132: error: Incompatible types in assignment (expression has type "timedelta", variable has type "datetime[None]")  [assignment]
+ tornado/test/locale_test.py:138: error: Unsupported operand types for - ("object" and "timedelta")  [operator]
+ tornado/test/locale_test.py:138: error: Incompatible types in assignment (expression has type "timedelta", variable has type "datetime[None]")  [assignment]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/types/datetime.py:589: error: Incompatible types in assignment (expression has type "datetime[None]", variable has type "datetime[tzinfo]")  [assignment]
+ tests/types/test_datetime.py:807: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/types/test_datetime.py:807: error: No overload variant of "datetime" matches argument type "map[int]"  [call-overload]
+ tests/types/test_datetime.py:807: note: Error code "call-overload" not covered by "type: ignore" comment
+ tests/types/test_datetime.py:807: note: Possible overload variants:
+ tests/types/test_datetime.py:807: note:     def [_TzInfoT] __new__(cls, year: SupportsIndex, month: SupportsIndex, day: SupportsIndex, hour: SupportsIndex = ..., minute: SupportsIndex = ..., second: SupportsIndex = ..., microsecond: SupportsIndex = ..., tzinfo: None = ..., *, fold: int = ...) -> datetime[None]
+ tests/types/test_datetime.py:807: note:     def [_TzInfoT] __new__(cls, year: SupportsIndex, month: SupportsIndex, day: SupportsIndex, hour: SupportsIndex = ..., minute: SupportsIndex = ..., second: SupportsIndex = ..., microsecond: SupportsIndex = ..., *, tzinfo: tzinfo, fold: int = ...) -> datetime[tzinfo]
+ tests/types/test_datetime.py:807: note:     def [_TzInfoT] __new__(cls, year: SupportsIndex, month: SupportsIndex, day: SupportsIndex, hour: SupportsIndex, minute: SupportsIndex, second: SupportsIndex, microsecond: SupportsIndex, tzinfo: tzinfo, *, fold: int = ...) -> datetime[tzinfo]

werkzeug (https://github.com/pallets/werkzeug)
- tests/test_wrappers.py:543: error: Incompatible types in assignment (expression has type "int", variable has type "datetime | None")  [assignment]
+ tests/test_wrappers.py:543: error: Incompatible types in assignment (expression has type "int", variable has type "datetime[Any] | None")  [assignment]
- tests/test_wrappers.py:587: error: Incompatible types in assignment (expression has type "int", variable has type "datetime | None")  [assignment]
+ tests/test_wrappers.py:587: error: Incompatible types in assignment (expression has type "int", variable has type "datetime[Any] | None")  [assignment]

spark (https://github.com/apache/spark)
- python/pyspark/sql/session.py:1254: note:          def [AtomicValue in (datetime, date, Decimal, bool, str, int, float)] createDataFrame(self, data: RDD[AtomicValue], schema: AtomicType | str, verifySchema: bool = ...) -> DataFrame
+ python/pyspark/sql/session.py:1254: note:          def [AtomicValue in (datetime[Any], date, Decimal, bool, str, int, float)] createDataFrame(self, data: RDD[AtomicValue], schema: AtomicType | str, verifySchema: bool = ...) -> DataFrame
- python/pyspark/sql/session.py:1254: note:          def [AtomicValue in (datetime, date, Decimal, bool, str, int, float)] createDataFrame(self, data: Iterable[AtomicValue], schema: AtomicType | str, verifySchema: bool = ...) -> DataFrame
+ python/pyspark/sql/session.py:1254: note:          def [AtomicValue in (datetime[Any], date, Decimal, bool, str, int, float)] createDataFrame(self, data: Iterable[AtomicValue], schema: AtomicType | str, verifySchema: bool = ...) -> DataFrame

Tanjun (https://github.com/FasterSpeeding/Tanjun)
+ tanjun/schedules.py:1077: error: Need type annotation for "result"  [var-annotated]
+ tanjun/schedules.py:1078: error: "NoReturn" has no attribute "total_seconds"  [attr-defined]

SinbadCogs (https://github.com/mikeshardmind/SinbadCogs)
+ scheduler/tasks.py:104: error: Returning Any from function declared to return "float"  [no-any-return]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/test_scalars.py:564: error: Expression is of type "datetime[None]", not "datetime[Any]"  [assert-type]
+ tests/test_scalars.py:623: error: Expression is of type "datetime[None]", not "datetime[Any]"  [assert-type]

ibis (https://github.com/ibis-project/ibis)
- ibis/expr/types/temporal.py:800: error: Argument "right" to "TimestampDelta" has incompatible type "datetime | ibis.expr.types.generic.Value"; expected "ibis.expr.operations.core.Value[Timestamp, Any]"  [arg-type]
+ ibis/expr/types/temporal.py:800: error: Argument "right" to "TimestampDelta" has incompatible type "datetime[Any] | ibis.expr.types.generic.Value"; expected "ibis.expr.operations.core.Value[Timestamp, Any]"  [arg-type]
- ibis/backends/exasol/__init__.py:269: error: "datetime" has no attribute "tz_convert"  [attr-defined]

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ bson/json_util.py:925: error: Unused "type: ignore" comment  [unused-ignore]

discord.py (https://github.com/Rapptz/discord.py)
- discord/embeds.py:338: error: Incompatible types in assignment (expression has type "None", variable has type "datetime")  [assignment]
+ discord/embeds.py:338: error: Incompatible types in assignment (expression has type "None", variable has type "datetime[tzinfo]")  [assignment]
- discord/scheduled_event.py:155: note:     def parse_time(timestamp: str) -> datetime
+ discord/scheduled_event.py:155: note:     def parse_time(timestamp: str) -> datetime[Any]
- discord/scheduled_event.py:155: note:     def parse_time(timestamp: str | None) -> datetime | None
+ discord/scheduled_event.py:155: note:     def parse_time(timestamp: str | None) -> datetime[Any] | None
- discord/member.py:155: note:     def parse_time(timestamp: str) -> datetime
+ discord/member.py:155: note:     def parse_time(timestamp: str) -> datetime[Any]
- discord/member.py:155: note:     def parse_time(timestamp: str | None) -> datetime | None
+ discord/member.py:155: note:     def parse_time(timestamp: str | None) -> datetime[Any] | None
- discord/ext/tasks/__init__.py:207: error: Incompatible types in assignment (expression has type "datetime", variable has type "None")  [assignment]
+ discord/ext/tasks/__init__.py:207: error: Incompatible types in assignment (expression has type "datetime[Any]", variable has type "None")  [assignment]
- discord/ext/tasks/__init__.py:209: error: Incompatible types in assignment (expression has type "datetime", variable has type "None")  [assignment]
+ discord/ext/tasks/__init__.py:209: error: Incompatible types in assignment (expression has type "datetime[tzinfo]", variable has type "None")  [assignment]
- discord/ext/tasks/__init__.py:217: error: Argument 1 to "_try_sleep_until" of "Loop" has incompatible type "None"; expected "datetime"  [arg-type]
+ discord/ext/tasks/__init__.py:217: error: Argument 1 to "_try_sleep_until" of "Loop" has incompatible type "None"; expected "datetime[Any]"  [arg-type]
- discord/ext/tasks/__init__.py:219: error: Incompatible types in assignment (expression has type "None", variable has type "datetime")  [assignment]
+ discord/ext/tasks/__init__.py:219: error: Incompatible types in assignment (expression has type "None", variable has type "datetime[Any]")  [assignment]
- discord/ext/tasks/__init__.py:220: error: Incompatible types in assignment (expression has type "datetime", variable has type "None")  [assignment]
+ discord/ext/tasks/__init__.py:220: error: Incompatible types in assignment (expression has type "datetime[Any]", variable has type "None")  [assignment]
- discord/ext/tasks/__init__.py:226: error: Unsupported operand types for >= ("datetime" and "None")  [operator]
+ discord/ext/tasks/__init__.py:226: error: No overload variant of "__ge__" of "datetime" matches argument type "None"  [operator]
+ discord/ext/tasks/__init__.py:226: note: Possible overload variants:
+ discord/ext/tasks/__init__.py:226: note:     def __ge__(self, datetime[tzinfo] | datetime[tzinfo | None], /) -> bool
+ discord/ext/tasks/__init__.py:226: note:     def __ge__(self, datetime[None] | datetime[tzinfo | None], /) -> bool
+ discord/ext/tasks/__init__.py:226: note:     def __ge__(self, datetime[Any], /) -> NoReturn
- discord/ext/tasks/__init__.py:237: error: Argument 1 to "_try_sleep_until" of "Loop" has incompatible type "None"; expected "datetime"  [arg-type]
+ discord/ext/tasks/__init__.py:237: error: Argument 1 to "_try_sleep_until" of "Loop" has incompatible type "None"; expected "datetime[Any]"  [arg-type]
- discord/ext/tasks/__init__.py:238: error: Incompatible types in assignment (expression has type "datetime", variable has type "None")  [assignment]
+ discord/ext/tasks/__init__.py:238: error: Incompatible types in assignment (expression has type "datetime[Any]", variable has type "None")  [assignment]
- discord/ext/tasks/__init__.py:254: error: Argument 1 to "_try_sleep_until" of "Loop" has incompatible type "None"; expected "datetime"  [arg-type]
+ discord/ext/tasks/__init__.py:254: error: Argument 1 to "_try_sleep_until" of "Loop" has incompatible type "None"; expected "datetime[Any]"  [arg-type]
- discord/ext/tasks/__init__.py:762: error: Incompatible types in assignment (expression has type "datetime", variable has type "None")  [assignment]
+ discord/ext/tasks/__init__.py:762: error: Incompatible types in assignment (expression has type "datetime[Any]", variable has type "None")  [assignment]
- discord/ext/tasks/__init__.py:765: error: Argument 1 to "recalculate" of "SleepHandle" has incompatible type "None"; expected "datetime"  [arg-type]
+ discord/ext/tasks/__init__.py:765: error: Argument 1 to "recalculate" of "SleepHandle" has incompatible type "None"; expected "datetime[Any]"  [arg-type]

koda-validate (https://github.com/keithasaurus/koda-validate)
+ koda_validate/generic.py:93: error: Returning Any from function declared to return "bool"  [no-any-return]
+ koda_validate/generic.py:95: error: Returning Any from function declared to return "bool"  [no-any-return]
+ koda_validate/generic.py:105: error: Returning Any from function declared to return "bool"  [no-any-return]
+ koda_validate/generic.py:107: error: Returning Any from function declared to return "bool"  [no-any-return]

alerta (https://github.com/alerta/alerta)
- alerta/models/key.py:24: error: Incompatible default for argument "expire_time" (default has type "None", argument has type "datetime")  [assignment]
+ alerta/models/key.py:24: error: Incompatible default for argument "expire_time" (default has type "None", argument has type "datetime[Any]")  [assignment]
- alerta/models/key.py:54: error: Argument "expire_time" to "ApiKey" has incompatible type "datetime | None"; expected "datetime"  [arg-type]
+ alerta/models/key.py:54: error: Argument "expire_time" to "ApiKey" has incompatible type "datetime[Any] | None"; expected "datetime[Any]"  [arg-type]
- alerta/models/heartbeat.py:28: error: Incompatible default for argument "create_time" (default has type "None", argument has type "datetime")  [assignment]
+ alerta/models/heartbeat.py:28: error: Incompatible default for argument "create_time" (default has type "None", argument has type "datetime[Any]")  [assignment]
- alerta/models/heartbeat.py:88: error: Argument "create_time" to "Heartbeat" has incompatible type "datetime | None"; expected "datetime"  [arg-type]
+ alerta/models/heartbeat.py:88: error: Argument "create_time" to "Heartbeat" has incompatible type "datetime[Any] | None"; expected "datetime[Any]"  [arg-type]

bokeh (https://github.com/bokeh/bokeh)
- src/bokeh/util/serialization.py:167:1: error: Argument 1 to "convert_datetime_type" becomes "Any | Any | Any | datetime | date | time | datetime64" due to an unfollowed import  [no-any-unimported]
+ src/bokeh/util/serialization.py:167:1: error: Argument 1 to "convert_datetime_type" becomes "Any | Any | Any | datetime[Any] | date | time | datetime64" due to an unfollowed import  [no-any-unimported]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/config_option.py: note: In member "is_expired" of class "ConfigOption":
+ lib/streamlit/config_option.py:305:9: error: Returning Any from function declared to return "bool"  [no-any-return]
- lib/tests/streamlit/elements/time_input_test.py:107:13: note:     def time_input(self, label: str, value: Union[time, datetime, Literal['now']] = ..., key: Optional[Union[str, int]] = ..., help: Optional[str] = ..., on_change: Optional[Callable[..., None]] = ..., args: Optional[Tuple[Any, ...]] = ..., kwargs: Optional[Dict[str, Any]] = ..., *, disabled: bool = ..., label_visibility: Literal['visible', 'hidden', 'collapsed'] = ..., step: Union[int, timedelta] = ...) -> time
+ lib/tests/streamlit/elements/time_input_test.py:107:13: note:     def time_input(self, label: str, value: Union[time, datetime[Any], Literal['now']] = ..., key: Optional[Union[str, int]] = ..., help: Optional[str] = ..., on_change: Optional[Callable[..., None]] = ..., args: Optional[Tuple[Any, ...]] = ..., kwargs: Optional[Dict[str, Any]] = ..., *, disabled: bool = ..., label_visibility: Literal['visible', 'hidden', 'collapsed'] = ..., step: Union[int, timedelta] = ...) -> time
- lib/tests/streamlit/elements/time_input_test.py:138:13: note:     def time_input(self, label: str, value: Union[time, datetime, Literal['now']] = ..., key: Optional[Union[str, int]] = ..., help: Optional[str] = ..., on_change: Optional[Callable[..., None]] = ..., args: Optional[Tuple[Any, ...]] = ..., kwargs: Optional[Dict[str, Any]] = ..., *, disabled: bool = ..., label_visibility: Literal['visible', 'hidden', 'collapsed'] = ..., step: Union[int, timedelta] = ...) -> time
+ lib/tests/streamlit/elements/time_input_test.py:138:13: note:     def time_input(self, label: str, value: Union[time, datetime[Any], Literal['now']] = ..., key: Optional[Union[str, int]] = ..., help: Optional[str] = ..., on_change: Optional[Callable[..., None]] = ..., args: Optional[Tuple[Any, ...]] = ..., kwargs: Optional[Dict[str, Any]] = ..., *, disabled: bool = ..., label_visibility: Literal['visible', 'hidden', 'collapsed'] = ..., step: Union[int, timedelta] = ...) -> time

[srittau]

While I think there's some promise in this approach, I struggle with Any annotated datetimes. The __sub__ overloads (for example), derive a return type of Any, probably because mypy is merging the timedelta and NoReturn annotations of multiple annotations. Does anyone have an idea how to work around this?

[bluetech]

I don't have much to add except to say that having this will be truly great, my code is littered with "aware"/"not aware" comments which really should be in the type, so I hope you can get it to work. (I haven't been bothered with it enough to use something like DataType yet).

I am curious about two things:

• Why default TzInfoT to Any instead of tzinfo | None?
• Why are the overloads needed? I guess typecheckers don't know the type of default values so they don't see that fromtimestamp(123) is the same as fromtimestamp(123, tz=None) and so can't infer TzInfoT=None?

[srittau]

@bluetech

• Why default TzInfoT to Any instead of tzinfo | None?

It tried both, but both didn't work. tzinfo | None is also my preferred default, but I hoped that Any would have less problems when it didn't.

• Why are the overloads needed? I guess typecheckers don't know the type of default values so they don't see that fromtimestamp(123) is the same as fromtimestamp(123, tz=None) and so can't infer TzInfoT=None?

Exactly.