Code coverage appears much lower on python 3.8

[jbarnoud]

The drop in code coverage reported by codecov seems to be directly related to python 3.8. The travis builds with any other version of python yield a code coverage of ~70% while the python 3.8 build yields a coverage of ~50%. I can reproduce the same numbers locally.

On tonight master (053e486), both python 3.5 and 3.8 report the same number of tests, and the success status is the same. Python 3.8 reports a few more warning, though. See #130 and #131 for the warnings.

At least one issue has been raised on the coverage module regarding changes in behaviour with python 3.8: nedbat/coveragepy#707

We probably should not upload coverage report from python 3.8 until the issue is solved.

[davidghiurco]

I just ran into this problem after eliminating all other variables. I've isolated it down to a change in Python version causing our CI to fail because of lower-than-expected test coverage on otherwise unmodified code. Our test coverage drops after Python is upgraded to 3.8.x from 3.7.x

I see this issue was closed 3 years ago, I'm wondering what the resolution is? Was Python 3.7 calculating coverage incorrectly?

[pckroon]

There were some CPython optimization that affected coverage, see also https://github.com/nedbat/coveragepy/. I can't recall how we truly resolved the issue here. It could be we simply bit the bullet and added tests to bump the coverage on all versions over the threshold. Or it was as simple as updating coveragepy. Too long ago for me to remember all the details :)