Ignore clause not ignoring directories

[geajack]

This is a different issue to issues/908.

Steps to reproduce

1. Create a directory test and within that a directory stuff.
2. Create files test/a.py and test/stuff/b.py. Put syntax errors in both.
3. From test, run pylint *.py **/*.py --ignore stuff.

Current behavior

Pylint does not ignore stuff/b.py, producing the message

a.py:1:0: E0001: invalid syntax (<unknown>, line 1) (syntax-error)
************* Module b
stuff/b.py:1:0: E0001: invalid syntax (<unknown>, line 1) (syntax-error)

Expected behavior

Pylint ignores the file stuff/b.py.

pylint --version output

astroid 2.1.0
Python 3.7.1 (default, Dec 14 2018, 19:28:38) 
[GCC 7.3.0]```

[geajack]

The problem seems to be in utils.expand_modules() in which we find the code that actually performs the ignoring:

if os.path.basename(something) in black_list:
    continue
if _basename_in_blacklist_re(os.path.basename(something), black_list_re):
    continue

Here something will be of the form "stuff/b.py" and os.path.basename(something) will be of the form b.py. In other words, before we do the ignoring, we specifically remove from the filepath all of the information about what directory it's in, so that it's impossible to have any way of ignoring a directory. Is this the intended behavior?

[PCManticore]

Hi @geajack Thanks for reporting an issue. That behaviour it's probably not intended, I think we definitely need to fix this to allow ignoring directories as well.

[bgehman]

@geajack @PCManticore Is there a work around to force pylint to ignore directories? I've tried ignore, ignored-modules, and ignore-patterns and not getting to a working solution. Background is I want to pylint scan source repositories (during our TravisCI PR builds), but want to exclude any python source found in certain directories: specifically directories brought in using git-submodules (as those submodules are already covered by their own TravisCI builds). Would like to set something in the project's .pylintrc that would configure pylint to ignore those directories...

[Michionlion]

Has there been any progress on this issue? It's still apparent in pylint 2.3.1.

[PCManticore]

@bgehman Right now ignoring directories is not supported, as per this issue suggests. We should add support for ignoring directories to --ignore and --ignore-patterns, while --ignored-modules does something else entirely (ignores modules from being type checked, not completely analysed).

@Michionlion There was no progress on this issue, as you can see there are 400 issues opened, so depending on my time, it's entirely possible that an issue could stay open for months or years. Feel free to tackle a PR if you need this fixed sooner.

[ashb]

Relates to #2541

[shidenggui]

I also meet this problem.
Can we check path directly? I think it's more convenient for usage.

[jw-lilly]

workaround... add this to your .pylintrc:

init-hook=
    sys.path.append(os.getcwd());
    from pylint_ignore import PylintIgnorePaths;
    PylintIgnorePaths('my/thirdparty/subdir', 'my/other/badcode')

then create pylint_ignore.py:

from pylint.utils import utils


class PylintIgnorePaths:
    def __init__(self, *paths):
        self.paths = paths
        self.original_expand_modules = utils.expand_modules
        utils.expand_modules = self.patched_expand

    def patched_expand(self, *args, **kwargs):
        result, errors = self.original_expand_modules(*args, **kwargs)

        def keep_item(item):
            if any(1 for path in self.paths if item['path'].startswith(path)):
                return False

            return True

        result = list(filter(keep_item, result))

        return result, errors

[unho]

When will we get a fix for this issue?

[Hubro]

This is still broken, one and a half year later... The documentation still claims that these parameters can ignore directories.

[digitaldavenyc]

@jw-lilly Thank you! Saved me a ton of time.

But... you have linting errors in your script 😝

[jw-lilly]

@jw-lilly Thank you! Saved me a ton of time.

But... you have linting errors in your script 😝

@digitaldavenyc You're welcome! I don't know whether to be happy or sad that I've recruited others to join me in monkeypatching... :-/

[Zeeshan138063]

Thanks @jw-lilly
I used your solution with little modifications to work like --ignore-patters

[hamdrew]

Here is a find workaround to ignore a particular subfolder when running pylint:

find . -path '**/migrations' -prune -false -o -name '*.py' | xargs pylint

It essentially asks "Give me all .py file except those that have the string /migrations in their path".

More find examples on excluding one or more directories

[fdamken]

It seems to me that the workaround proposed by @jw-lilly broke recently (on 2021-04-03, from version 2.8.0 on) with commit 63ca059 as the expand_modules method does not belong to utils anymore. It was moved to the pylint.lint.expand_modules module, from where it is imported as from pylint.lint.expand_modules import expand_modules in various files. However, this style of import makes it impossible to overwrite the expand_modules function as before.

@Pierre-Sassoulas Do you think it is a possibility to revert that commit to make this workaround work again or to merge #3266 for supporting this natively?

[Pierre-Sassoulas]

Hello, thank you for bringing this to my attention. I'd prefer to relaunch #3266, it seems it i's blocked because the original implementer does not have time to rebase it. I'd be glad to review and merge it. Basically all mainteners participated in it at some point so this is not a controversial change.

[Pierre-Sassoulas]

As an aside it's possible to use pylint through pre-commit (which has its own exclude system) .

[fdamken]

Hello, thank you for bringing this to my attention. I'd prefer to relaunch #3266, it seems it i's blocked because the original implementer does not have time to rebase it. I'd be glad to review and merge it. Basically all mainteners participated in it at some point so this is not a controversial change.

I relaunched the PR as #4516, rebased the corresponding branch on top of master, adjusted the changes to match the ones introduced with 63ca059 and the renaming from black- to ignore-list.

[jamesbraza]

@fdamken how do we use that PR? I used to have this in my pyproject.toml:

# SEE: https://github.com/PyCQA/pylint/issues/2686#issuecomment-621927895
# Python code to execute, usually for sys.path manipulation such as pygtk.require().
init-hook = """
sys.path.append(os.getcwd());
from pylint_ignore import PylintIgnorePaths;
PylintIgnorePaths("relative/path/to/directory/to/ignore");
"""

How do I integrate the new ignore-paths option? Can you provide an example here?

Also, thank you for fixing this, it's awesome.

[fdamken]

@jamesbraza yes, of course: The ignore-paths option accepts a list of regular expressions that the path of the file to check (or, so to say, the path of the time to check where or is to be checked) is matched against. If there is any match, the file is ignored.

So too ignore any directory tests, for example, add ignore-paths = ^(.*/|)tests/.*$. I think (!) that currently this property only checks against full paths, so excluding a finger relative to the PyLint config is not possible… You can anyhow add something like ignore-paths = ^.*/relative/path/to/directory/to/ignore/.*$, which ignores the directory. Of course this causes problems of this exact path already anywhere else, too. I'll take a look at this again.

[jamesbraza]

Thank you for the response @fdamken, I have tried a lot of stuff and am still having trouble. Are you sure ignore-paths will integrate with pyproject.toml?

Check out this GitHub search and the example config in examples/pylintrc. I am not sure if the location within pyproject.toml matters, do you know which section of the config ignore-paths should go in? Is it [tool.pylint.master]?

Also, when using in pyproject.toml, should we specify like:

ignore-paths = ["unittest_.*"]
# or
ignore-paths = "unittest_.*"

I have tried many permutations of possible regexes. My current regex checks out on regex101: .*unique_directory_name.*

And I still can't get it to work :( any further help is much appreciated.

[fdamken]

Hi @jamesbraza,
sorry for coming back to this so late, I was really busy and had to test this. Starting from PyLint 2.9 (you can try in version 2.9.0.dev1 which is on PyPi), the following exclusion directive works in the .pylintrc:

ignore-paths=^src/experiments/.*$,
             ^src/ingredients/.*$,
             ^src/scripts/.*$,
             ^src/tests/.*$

So you separate multiple regular expressions using commas and the path is matched starting from the working directory. For pyproject.toml, you have to surround each entry with quotes and, when using multiple expressions, the whole things with brackets:

[tool.pylint.MASTER]
ignore-paths = [ "^src/experiments/.*$",
                 "^src/ingredients/.*$",
                 "^src/scripts/.*$",
                 "^src/tests/.*$" ]

Best regards and sorry again!

[jamesbraza]

Hello @fdamken ! Thank you for detailing out how to use it. It seems https://github.com/PyCQA/pylint/pull/4623/files#diff-81fbdb16099ef5c97297b1a8dbd9a6a629906cab99950094742298d574573de3R24 added this to the example .pylintrc in the master section (reinforces what you've said).

Your comment:

you can try in version 2.9.0.dev1 which is on PyPi

pointed out my error.... I wasn't using the version of pylint your PR was merged to! (facepalm)

Now that pylint 2.9 is released, I am using this in my code, and it works great! Thank you for implementing this fix.