Language server fails to load pylsp_black plugin in macOS app

[mrclary]

Problem Description

pylsp fails to load the pylsp_black plugin in the macOS standalone application. This does not appear to afflict Spyder installed from pip or conda, only the standalone application. I have not verified the Windows application. There are two issues:

• pathspec is not included in the bundle
• black cannot be imported

In the lsp server log:

WARNING - pylsp.config.config - Failed to load pylsp entry point 'pylsp_black': The 'pathspec>=0.9.0' distribution was not found and is required by black

This indicates that the pathspec package is not included in the application bundle and is easily remedied by adding it to INCLUDES. However, upon doing so, a subsequent issue arises:

WARNING - pylsp.config.config - Failed to load pylsp entry point 'pylsp_black': No module named '610faff656c4cfcbb4a3__mypyc'

From the Spyder internal console, one can verify that the pylsp_black plugin is unable to load the black module.

>>> import pkg_resources
>>> [ep for ep in pkg_resources.iter_entry_points('pylsp') if ep.name=='pylsp_black'][0].load()
Traceback (most recent call last):
  File "code.pyc", line 90, in runcode
  File "<console>", line 2, in <module>
  File "/Users/rclary/Documents/Python/spyder/installers/macOS/dist/Spyder.app/Contents/Resources/lib/python3.9/pkg_resources/__init__.py", line 2458, in load
    return self.resolve()
  File "/Users/rclary/Documents/Python/spyder/installers/macOS/dist/Spyder.app/Contents/Resources/lib/python3.9/pkg_resources/__init__.py", line 2464, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/rclary/Documents/Python/spyder/installers/macOS/dist/Spyder.app/Contents/Resources/lib/python3.9/pylsp_black/plugin.py", line 7, in <module>
    import black
ModuleNotFoundError: No module named '610faff656c4cfcbb4a3__mypyc'
>>> import black
Traceback (most recent call last):
  File "code.pyc", line 90, in runcode
  File "<console>", line 2, in <module>
ModuleNotFoundError: No module named '610faff656c4cfcbb4a3__mypyc'

Further investigation is required to remedy this last issue.

Versions

• Spyder version: 5.3.0 None
• Python version: 3.9.5 64-bit
• Qt version: 5.15.2
• PyQt5 version: 5.15.6
• Operating System: Darwin 20.6.0

Dependencies

# Mandatory:
applaunchservices >=0.1.7            :  0.2.1 (OK)
atomicwrites >=1.2.0                 :  1.4.0 (OK)
chardet >=2.0.0                      :  4.0.0 (OK)
cloudpickle >=0.5.0                  :  2.0.0 (OK)
cookiecutter >=1.6.0                 :  1.7.3 (OK)
diff_match_patch >=20181111          :  20200713 (OK)
intervaltree >=3.0.2                 :  3.1.0 (OK)
IPython >=7.31.1;<8.0.0              :  7.32.0 (OK)
jedi >=0.17.2;<0.19.0                :  0.18.1 (OK)
jellyfish >=0.7                      :  0.9.0 (OK)
jsonschema >=3.2.0                   :  4.4.0 (OK)
keyring >=17.0.0                     :  23.5.0 (OK)
nbconvert >=4.0                      :  6.4.5 (OK)
numpydoc >=0.6.0                     :  1.2.1 (OK)
parso >=0.7.0;<0.9.0                 :  0.8.3 (OK)
pexpect >=4.4.0                      :  4.8.0 (OK)
pickleshare >=0.4                    :  0.7.5 (OK)
psutil >=5.3                         :  5.9.0 (OK)
pygments >=2.0                       :  2.11.2 (OK)
pylint >=2.5.0                       :  2.13.3 (OK)
pyls_spyder >=0.4.0                  :  0.4.0 (OK)
pylsp >=1.4.1;<1.5.0                 :  1.4.1 (OK)
pylsp_black >=1.2.0                  :  1.2.0 (OK)
qdarkstyle >=3.0.2;<3.1.0            :  3.0.3 (OK)
qstylizer >=0.1.10                   :  0.2.1 (OK)
qtawesome >=1.0.2                    :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0             :  5.3.0 (OK)
qtpy >=2.0.1                         :  2.0.1 (OK)
rtree >=0.9.7                        :  0.9.7 (OK)
setuptools >=49.6.0                  :  60.10.0 (OK)
sphinx >=0.6.6                       :  4.5.0 (OK)
spyder_kernels >=2.3.0;<2.4.0        :  2.3.0 (OK)
textdistance >=4.2.0                 :  4.2.2 (OK)
three_merge >=0.1.1                  :  0.1.1 (OK)
watchdog >=0.10.3                    :  2.1.7 (OK)
zmq >=17                             :  22.3.0 (OK)

# Optional:
cython >=0.21                        :  0.29.28 (OK)
matplotlib >=3.0.0                   :  3.5.1 (OK)
numpy >=1.7                          :  1.22.3 (OK)
pandas >=1.1.1                       :  1.4.1 (OK)
scipy >=0.17.0                       :  1.8.0 (OK)
sympy >=0.7.3                        :  1.10.1 (OK)

# Spyder plugins:
spyder_terminal.terminalplugin 1.2.2 :  1.2.2 (OK)

[mrclary]

Nor is this remedied by including black in PACKAGES (i.e. outside the zip directory).
I do not know what is causing this problem.

[ccordoba12]

What if you add Mypy to the the list of packages included in the app?

[mrclary]

Mypy is not installed in the build environment and is not required by any packages in the environment.
Nevertheless, including all black dependencies in the list of packages does not work either (click, mypy_extensions, pathspec, platformdirs, tomli, typing_extensions).
However, I have discovered the issue. The black distribution includes black, _black_versions, blackd, blib2to3, and 610faff656c4cfcbb4a3__mypyc packages, which are missed by py2app. These are simple enough to include in the INCLUDES list, but I have my reservations about 610faff656c4cfcbb4a3__mypyc. How is this package named? It appears to be the result of some kind of hash; will it change? Does it depend on platform, operating system version, Python version, black version, ... ?

[ccordoba12]

You could ask on the Black repo about it or add the file that ends in __mypyc instead.

[mrclary]

So including blib2to3 in PACKAGES, and 610faff656c4cfcbb4a3__mypyc and pathspec in INCLUDES, resolves the issues. I've submitted a new recipe for black to py2app ronaldoussoren/py2app#418. It should be resolved by #17612. And, yes, I'll inquire on the psf/black repo.