with --files, coala executes every file in separate process, so that mypy checks run very slowly #6130
Comments
That is how slow mypy works if you feed it with files one-by-one like Coala does. |
remort
changed the title
|
Hi can I be assigned this issue ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Steps to reproduce:
Run Coala, with mypy bear enabled only, over catalog with code running this command:
coala --ci -c .coarc --files $(find . -iname '*.py')It will run several minutes (~4m) on my 66 files found.
If you run mypy itself, passing it the same file list (
mypy $(find . -iname '*.py')) - it will finish all the checks in several (~12) seconds (see example below).Looking at top (
top -bto see full command line for every process) I realized that coala run mypy over every file.So 66 files leads to consequent 66 runs of mypy passing it a particular filename each time.
You can emulate exactly this slow behaviour by running following code:
for line in $(find . -iname '*.py'); do /usr/local/bin/python -m mypy --python-version 3.6 "$line"; doneThe for-loop will be finished in several minutes. Being run file-by-file, mypy works very slow.
The exact command with
python -m mypy...taken from top batch output. That's how Coala run mypy,pylint, etcetera python linters.The following code runs mypy over all the same 66 files in ~12 seconds:
That's how we should run linter executable over bunch of files.
Simultaneous multiprocessing jobs counted by '--jobs' option, as you run linters inside coala, shouldn't be an issue.
You can divide files bu number of jobs and pass every chunkof files to it's job, for example.
Job should run linter executable just once, passing an entire chunk of files to the linter's command line.
The text was updated successfully, but these errors were encountered: