# type: str between attributes breaks attribute documentation

[Querela]

Describe the bug

The comment # type: str between docstring-commented (dataclass) attributes breaks the autoclass documentation.

To Reproduce
To reproduce I have linked my project. A more brief example may come later.

Steps to reproduce the behavior:

$ git clone https://github.com/Querela/python-krakenexapi
$ cd python-krakenexapi
$ # change lines as described below
$ # NOTE: maybe create a virtual enviroment first?
$ pip install -r docs/requirements.txt
$ sphinx-build -b html docs dist/docs
$ cd dist/docs
$ # open dist/docs/reference/krakenexapi.wallet.html and compare outputs

Failing:

@dataclass(frozen=True)
class TradingTransaction:
    #: Currency pair, of base and quote currency.
    #: `quote` currency determines the price. `base` is the currency being traded.
    currency_pair: CurrencyPair
    # #: Type of transaction
    # type: str
    #: Price of (crypto) currency.
    price: float
    #: Amount of currency.
    amount: float
    #: Cost of `base` currency
    #: (:attr:`~krakenexapi.wallet.TradingTransaction.currency_pair`)
    #: in `quote` currency.
    cost: float
    #: Fees for transactions, in `quote` currency.
    fees: float

Working:

@dataclass(frozen=True)
class TradingTransaction:
    #: Currency pair, of base and quote currency.
    #: `quote` currency determines the price. `base` is the currency being traded.
    currency_pair: CurrencyPair
    #: Price of (crypto) currency.
    price: float
    #: Amount of currency.
    amount: float
    #: Cost of `base` currency
    #: (:attr:`~krakenexapi.wallet.TradingTransaction.currency_pair`)
    #: in `quote` currency.
    cost: float
    #: Fees for transactions, in `quote` currency.
    fees: float

Note that the only difference is in the middle line with the problematic comment:

    #: Price of (crypto) currency.
    # type: str                                        <-------------- HERE
    price: float

If I change type to stype or anything else, it won't fail. I assume it somehow parses type and tries to interpret it as part of the documentation or some python code? The curious thing is that it is not even a docstring comment #:, so my only other guess is that it tries to interpret it as typehint (even if it is on separate line)?

Expected behavior
It should show the dataclass attributes in the generated documentation.

Your project

• https://github.com/Querela/python-krakenexapi

Screenshots
Maybe this will show it visually, but in-short, all attributes in dataclasses will not appear if the problematic comment is in the code.

If comment is inserted (only properties, no attributes):

Without the comment (you can see the attributes):

Environment info

• OS: Linux Ubuntu 20.04 (WSL2)
• Python version: 3.9.0+
• Sphinx version: 3.4.2
• Sphinx extensions: sphinx.ext.autodoc, sphinx.ext.autosummary, sphinx.ext.coverage, sphinx.ext.extlinks, sphinx.ext.ifconfig, sphinx.ext.napoleon, sphinx.ext.todo, sphinx.ext.viewcode
• Extra tools: Browser
• conf.py
• TOX env: alabaster==0.7.12,Babel==2.9.0,certifi==2020.12.5,chardet==4.0.0,docutils==0.16,idna==2.10,imagesize==1.2.0,Jinja2==2.11.2,-e git+git@github.com:Querela/python-krakenexapi.git@f29ec8b41ac57e81317ab4e7ee0761f07e4bd3db#egg=krakenexapi,MarkupSafe==1.1.1,packaging==20.8,Pygments==2.7.3,pyparsing==2.4.7,pytz==2020.5,requests==2.25.1,snowballstemmer==2.0.0,Sphinx==3.4.2,sphinx-rtd-theme==0.5.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,urllib3==1.26.2,wrapt==1.12.1

NOTE Update
The order of comments was slightly different (but with same end results)

    # #: Type of transaction
    # type: str
    #: Price of (crypto) currency.

[tk0miya]

Yes, autodoc considers that comment starts with type: is a kind of type annotation.
In PEP-484, they are called type-comments: https://www.python.org/dev/peps/pep-0484/#type-comments

Hence, there are two kind of type annotations on failed code:

class TradingTransaction:
    #: Currency pair, of base and quote currency.
    #: `quote` currency determines the price. `base` is the currency being traded.
    currency_pair: CurrencyPair
    #: Price of (crypto) currency.
    # type: str
    price: float

We can't determine the correct type of TradingTransaction.currency_pair is CurrencyPair of str (or str might be a type comment for price!).

Expected behavior
It should show the dataclass attributes in the generated documentation.

Are you saying autodoc should build code having both variable annotation and type comments? I don't understand why do you create a module. Please tell me why do you want to annotate types in two ways.

[Querela]

No, that is more or less my issue.
I only wanted code type annotations (varname: typename), not type hints in comments (varname = 0 # type: typename). Can I disable this? Because sphinx did not output any error and disabled all instance types in the whole file, not just the single class. Somewhat unexpected. (I had also assumed that type hints in comments should occur in the same line, as more comments were above and below the critical line. Or did it consider a type hint for an empty line and broke because of this?)

I was just trying to comment out my variable type: str. (probably a really bad choice on my end, since it overloads standard functions, but in my defense, it was work-in-progress and not really the final name)

Note, that I posted an excerpt that was slightly different (I updated it in the issue description but in the end with the same issue/result)

    # #: Type of transaction            <--- commented out
    # type: str                         <--- commented out (but interpreted as type hint?)
    #: Price of (crypto) currency.
    price: float

# i wanted to use it like this (in a dataclass, so default values are not critical)
varname: typename = defaultvalue

# i know that it is also to possible to use it like this
varname = defaultvalue  # type:  typename

# but I thought this was not a type annotation
varname = defaultvalue
# type: typename

[tk0miya]

I reproduced what happened in this case. I found ast.parse() raises SyntaxError when type: comment is located at wrong place. As a result, autodoc can't detect any comments for attributes, and all of them are filtered (they will be shown if :undoc-members: option given).

FROM python:3.8-slim

RUN apt update; apt install -y build-essential curl git unzip vim
RUN git clone https://github.com/Querela/python-krakenexapi
WORKDIR /python-krakenexapi
RUN sed -i -e 's/ttype:/type:/' src/krakenexapi/wallet.py
RUN python -c 'import ast; code = open("src/krakenexapi/wallet.py").read(); ast.parse(code, type_comments=True)'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 455
    # type: str
            ^
SyntaxError: invalid syntax

[tk0miya]

I'm debating which is better 1) retry parsing without type_comments, or 2) emit a warning for SyntaxError.

[Querela]

Yes, but with :undoc-members: the docstring content is not preserved/found (if I remember correctly), they are essentially like undocumented variables, so no docstring.

Is it possible to explicitly set sphinx to ignore type hints in comments? So the parsing with ast can use this and not parse them if not required. Default can be the current behavior.
Users that mix their type hints probably will not make this error?

And might this be an error in ast (unlikely) or was it just a really unfortunate usage from my side?
The following throws an error, too. Modifying it to use ## or type : works, just not the exact form below. So, I guess it is intentional, and I have to be careful how I comment out my code...

import ast
ast.parse("# type: str", type_comments=True)

---

I think 2) a warning makes it more transparent for the user. Just having all comments disappear in the generated doc is disconcerting. And silently reparsing it and then having other type hints in comments being ignored, might look ok but is not immediately visible to the user and can then be overlooked for some time. Researching when it happened might take more time, so a visible warning or error might be best.

[tk0miya]

In my understanding, what you really want is autodoc can build a document even if the code contains invalid type_comments. So such option is not needed, right? I think your goal is 1) to make our parser more robust even if the code contains invalid type_comments, or 2) to warn on comment parsing failure.

[Querela]

Yes, you are right.
A warning would be best in my opinion.

[tk0miya]

I posted a PR #8667 to fix this. Finally, I choose the 1st way because robustness is worthy for our parse. And it is a bit hard to emit a warning on the error.

[Querela]

Ok. Thank you.
If it might occur for other users, they can probably find the issue here?
To make it visible, I would have liked a warning/error but I'm not that familiar with the code. And as a quick glance at the module shows that any error is wrapped in a PycodeError and this did not result in a visual clue for me.
So the only easy option might be a warn.warning but here I'm even more unfamiliar how the usage convention is (often only used for depreciation notices?, not completely valid input in numpy) and whether it is visible on normal sphinx usage...

[tk0miya]

If it might occur for other users, they can probably find the issue here?

What do you mean? After #8667 merged, this error will not happen. If you'd like to find the invalid type comment like # type: str at wrong place, please use mypy or other type checkers. Sphinx simply ignores it now.

[Querela]

Yes, you are correct.
I guess, I was thinking of some unusual circumstances but will probably not happen.

[tk0miya]

Thank you for comment. Now I merged #8667. So I'm closing this. Thank you for reporting :-)

[Querela]

Great. More like thank you for fixing ;-)