Typing issues

[eykamp]

It appears that this version of psycopg is developed using typing throughout, which is fantastic. Unfortunately, something is not working for me using VSCode and pylance extension (see screenshot below). Is this a known issue, and if so, do you know if it is a pylance problem or a psycopg issue?

This code sample is from the Basic Usage documentation page, and it runs until it fails to connect, which proves the basic install is functional.

[dlax]

It might help if you could provide details about the actual problem (by clicking on the "View Problem" link, I suppose).

[eykamp]

A typical example is shown in the UR of the screen capture: "Type of conn is unknown". This is what pylance shows when a type is missing, incomplete, or undetermined.

Looking at the psycopg source, it seems that the type should be Connection, which inherits from BaseConnection[Row]. I know that pylance has difficulty with the type Row, and I wonder if its use in the definition of Connection might be the problem.

[dvarrazzo]

@eykamp we use mypy as the reference type checker, which has no problem with the definition of the Connection.

We even have tests that prove that typing code using psycopg works correctly. In you case, conn should be understood as a psycopg.Connection[Tuple[Any, ...]].

I assume from what I see that pylance is just a frontend to pyright, which does the real checking. If any you could try to run pyright on psycopg code to see if it understand everything.

We are open to improvements for other checking tool, provided that they are not too invasive and don't break mypy. Closing this, because for sure the type returned by connect() is advertised and mypy can use it alright.

[eykamp]

Thanks -- I suspect it is a pylance issue (I believe pyright was rolled into pylance and is no longer a going concern), but I was hoping someone here had some insight. If I can get it to work with mypy, which I'm sure I can, I'll report this to the pylance folks to see if it suggests a problem there.

[eykamp]

I reported this over at the pylance project, and their response was interesting and hopefully useful:

This library is marked as "py.typed", which means that it is claiming to be fully typed. While it does contain some type information, some of the symbols are lacking type annotations.

Pyright (the type checker that pylance is built upon) has a command-line option "--verifytypes" that you can run on a "py.typed" library to determine its "type completeness". When I run pyright --verifytypes psycopg on the latest published version of the library, I get the following report:

Symbols exported by "psycopg": 1473
  With known type: 1121
  With partially unknown type: 352
  Functions without docstring: 456
  Functions without default param: 82
  Classes without docstring: 201

Other symbols referenced but not exported by "psycopg": 1038
  With known type: 919
  With partially unknown type: 119

Type completeness score: 76.1%

For "py.typed" libraries, pyright (and pylance) rely on type annotations. Symbols that have no type annotations are assumed to be "Any" / "Unknown". Guidance for library maintainers can be found here. Pyright fully implements this guidance, whereas mypy has not yet been updated to comply with these rules. That explains why you are seeing the difference between the two type checkers.

If this library is important to you, please reach out to the maintainer and recommend that they add the missing type annotations. And if they have any questions, we're happy to assist.

In the meantime, consumers of psycopg have a couple of options:

1. Disable strict mode type checking in pyright / pylance so you don't receive error messages for unknown types.
2. Manually delete the "py.typed" file from the installed library so pyright / pylance falls back on type inference for symbols that are missing type annotations.

microsoft/pylance-release#2267 (comment)

[dlax]

https://github.com/python/typing/blob/master/docs/source/libraries.rst#type-completeness

So apparently, it's "required" to annotate instance/class attributes even if they are assign a value:

class BaseCursor(Generic[ConnectionType, Row]):
    ...
    def __init__(self, connection: ConnectionType):                             
        self._conn = connection
        self.format = Format.TEXT  # Missing type annotation for public attibute here

Some pyright complaints on psycopg come from there.

[dvarrazzo]

Psycopg codebase is checked with mypy --strict, which gives error if any single variable or function parameter has no type.

We have 100% type coverage according to mypy rules, which infers correctly the type of every single variable or attribute we use. Pyright is broken.

[dvarrazzo]

@dlax

So apparently, it's "required" to annotate instance/class attributes even if they are assign a value:

This is a pretty big backward incompatible change. I wonder if it has been discussed somewhere.

[dlax]

So apparently, it's "required" to annotate instance/class attributes even if they are assign a value:

This is a pretty big backward incompatible change. I wonder if it has been discussed somewhere.

This comes from python/typing#889 and there is a follow-up discussion at python/typing#913 (haven't read it completely, yet).

I agree this is debatable.

Not sure it should be discussed in terms of backwards compatibility because type checkers actually become more strict as they improve over time, so they're kind of always backward incompatible...

[dvarrazzo]

I am pretty angry right now at that. I think I should add some notes to the conversation in python/typing#913 but now I'm too upset to be constructive.