Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash with -j 2 when linting empty file #7635

Closed
cdce8p opened this issue Oct 17, 2022 · 3 comments · Fixed by #7640
Closed

Crash with -j 2 when linting empty file #7635

cdce8p opened this issue Oct 17, 2022 · 3 comments · Fixed by #7640
Labels
Blocker 🙅 Blocks the next release Crash 💥 A bug that makes pylint crash macOS multiprocessing Needs investigation 🔬 A bug or crash where it's not immediately obvious what is happenning Regression
Milestone
2.15.5

Comments

@cdce8p
Copy link
Member

cdce8p commented Oct 17, 2022

Bug description

Noticed this crash while linting an empty file with -j 2. It might be OS dependent as so far I can only reproduce it on MacOS. Linux seems fine.

Bisected the issue to #7284.
In particular this line:
https://github.com/PyCQA/pylint/blob/77e8ae4f3902697babb186482f577a2c8b1a3725/pylint/lint/pylinter.py#L379

/CC: @daogilvie, @DanielNoord

Configuration

No response

Command used

pylint -j 2 test.py

Pylint output

(venv-310) $ pylint -j 2 test.py 
Traceback (most recent call last):
  File "/.../pylint/venv-310/bin/pylint", line 33, in <module>
    sys.exit(load_entry_point('pylint', 'console_scripts', 'pylint')())
  File "/.../pylint/pylint/__init__.py", line 35, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/.../pylint/pylint/lint/run.py", line 207, in __init__
    linter.check(args)
  File "/.../pylint/pylint/lint/pylinter.py", line 668, in check
    check_parallel(
  File "/.../pylint/pylint/lint/parallel.py", line 141, in check_parallel
    jobs, initializer=initializer, initargs=[dill.dumps(linter)]
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 304, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 276, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 498, in dump
    StockPickler.dump(self, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 1375, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 1375, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 1375, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 1375, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 1375, in save_module
    pickler.save_reduce(_import_module, (obj.__name__,), obj=obj,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 687, in save_reduce
    save(cls)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/.../pylint/venv-310/lib/python3.10/site-packages/dill/_dill.py", line 1439, in save_type
    StockPickler.save_global(pickler, obj, name=name)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pickle.py", line 1076, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <class 'astroid.util.Uninferable'>: it's not the same object as astroid.util.Uninferable

Expected behavior

No crash

Pylint version

pylint 2.16.0-dev
astroid 2.13.0-dev0
Python 3.10.8 (v3.10.8:aaaf517424, Oct 11 2022, 10:14:40) [Clang 13.0.0 (clang-1300.0.29.30)]

OS / Environment

MacOS

Additional dependencies

No response
@cdce8p cdce8p added Blocker 🙅 Blocks the next release Regression Crash 💥 A bug that makes pylint crash Needs investigation 🔬 A bug or crash where it's not immediately obvious what is happenning labels Oct 17, 2022
@cdce8p cdce8p added this to the 2.15.5 milestone Oct 17, 2022
@daogilvie
Copy link
Contributor

Ah, that's a shame — I did not explicitly think to check if the module object could be pickled to synch across processes/threads. I think I know what might be worth trying though — I'll look into this shortly.

@daogilvie
Copy link
Contributor

daogilvie commented Oct 17, 2022 •
edited

Right, my immediate feeling is the core issue is that the contents of the _dynamic_plugins dictionary is not pickleable, hence the dill.dumps call in check_parallel crashes: https://github.com/PyCQA/pylint/blob/77e8ae4f3902697babb186482f577a2c8b1a3725/pylint/lint/parallel.py#L136-L141
I think it would be appropriate to remove the modules/errors from the self._dynamic_plugins dictionary before the call to dill.dumps, as I think that by this point the custom modules have already been registered and had their configuration loaded.
I propose one of two things:

  1. Swap out the dictionary at the end of load_plugin_configuration, where everything has been done and the contents are no longer needed: https://github.com/PyCQA/pylint/blob/77e8ae4f3902697babb186482f577a2c8b1a3725/pylint/lint/pylinter.py#L383-L405 OR
  2. Add an explicit method+call in check_parallel (as above) to swap out the dictionary before the Pool is set up.

If we don't want to edit the dictionary, we have to add extra complexity to the plugin loader itself. I don't know what would be involved to have entirely pickle-able values — this would also exclude the ModuleNotFoundError I think, as it might contains a traceback or frame.

I'll trial option 1 in a branch now, see if that tanks some tests.

@daogilvie daogilvie mentioned this issue Oct 17, 2022
Merged
@daogilvie
Copy link
Contributor

Ok, linked PR #7640 addresses this issue. The failing tests were failing before the commit, and running pylint -j 2 test.py on an empty file no longer crashes.
Happily today was mostly a series of all-hands broadcast meetings at work, so I was able to jump in and fix the mess I made here in the quiet moments 😄
Let me know if this is something that seems suitable as a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Blocker 🙅 Blocks the next release Crash 💥 A bug that makes pylint crash macOS multiprocessing Needs investigation 🔬 A bug or crash where it's not immediately obvious what is happenning Regression
Projects
None yet
Milestone
2.15.5
Development

Successfully merging a pull request may close this issue.

Swap plugin cache to pickle-able values when done
3 participants
@daogilvie @Pierre-Sassoulas @cdce8p