typos: allow configPath to be of type string or path + take excludes from `.typos.toml` into account when `pre-commit run typos --all-files` runs #405

phip1611 · 2024-03-12T11:54:11Z

Unfortunately, the typos story still isn't solved properly at this point. My concerns decsribed here still exist.

The Problem

Hint: The problem is also shortly discussed in the commit messages.

Yes, using pass_filenames = false should only be done if no other variant is possible (pass_filenames = true is fine). My first "fix" merged a few months ago was wrong. And a commit hook should only run on changed files, I agree.

But, people also use pre-commit run --all-files to run all checks on the whole repo. Some people don't even use Git commit hooks at all and just use pre-commit to run all style checks at once. And that's a perfeclty valid usecase.

The problem is that if you run $ pre-commit run typos --all-files, typos behave differently compared to $ typos ..

The Fix

I modified typos.configPath in a way to be either of type string or path. The change is additive and not breaking.

To use the new option, you have to do something like this:

    settings = {
      typos = {
        # configuration = ""  # POSSIBLE
        # configPath = ".typos.toml"; # OLD
        configPath = .typos.toml; # NEW, RECOMMENDED
      };
    };

You can verify that my solution works by checking the typos invocation:

$ strace -s 64 -Tfe trace=execve pre-commit run --all-files typos

Unintended files are not longer passed to typos. 🎉

Ping @totoroot @domenkozar

phip1611 · 2024-03-12T11:55:15Z

FYI ping @blitz

phip1611 · 2024-03-12T12:09:45Z

~~Personally, I'd also like to get rid of configPath of type str, but then we break the API again.~~

configPath is now either string or path.

totoroot · 2024-03-12T13:28:16Z

Sounds good. I'll test it in the coming days.

domenkozar · 2024-03-19T03:52:42Z

Sorry for the mess, we revamed module structure in #397, could you port these changes over?

phip1611 · 2024-03-19T12:01:17Z

@domenkozar done

domenkozar · 2024-03-25T23:25:40Z

ping @totoroot for testing

phip1611 · 2024-04-02T14:02:30Z

I rebased again and improved my solution. My solution is additive and doesn't break anything; I tested all the existing use cases. Can we merge it, @domenkozar ?

modules/hooks.nix

astro

LGTM apart from a few nits

modules/hooks.nix

phip1611 · 2024-04-19T10:54:32Z

Ping @m1-s @domenkozar @totoroot - I hope we can finally get this merged as we need it in multiple of our projects at Cyberus Technology :)

domenkozar · 2024-04-19T11:13:08Z

If there's no comments in next few days I'll merge it.

roberth

Was mostly interested in the usage of path values, but also found a bug.

roberth · 2024-04-19T11:28:12Z

modules/hooks.nix

@@ -3331,6 +3375,7 @@ lib.escapeShellArgs (lib.concatMap (ext: [ "--ghc-opt" "-X${ext}" ]) hooks.ormol
            in
            "${hooks.typos.package}/bin/typos ${cmdArgs}";
          types = [ "text" ];
+          excludes = excludesFromConfig;


excludes is expected to be in regex form, whereas excludesFromConfig has globs.

From my personal testing I can verify that it works when the typos.toml file doesn't use globs.

What can we do do mitigate this issue? This issue really needs to be solved for some of our projects. Any idea?

Maybe look for *s and ?s and if they occur, throw an explanation that it's not implemented?
Not fantastic, but leaps better than making users debug or keeping your colleagues/clients waiting.

Ideally, the globs could be translated. https://github.com/hercules-ci/gitignore.nix has inherited an implementation of that from nix-gitignore, but I don't know if it's the same dialect if any. It translates globs into a regex, using a regex... :D ("so now I have three problems"?)

I updated the branch and updated the commit messages.

I think removing everything that looks like a glob from exclude-list is fine and good-enough. It is clearly a benefit compared to the current situation.

Will do it on Monday. Happy weekend :)

Are you suggesting to silently drop excludes? I would hate that as a user and flip a table ;)

Likewise!

Agree. Yikes, the whole thing is complicated. I just want typos always behave the same, with and without pre-commit run. The current behavior means trouble and manual workarounds for at least one of our internal projects.

See #405 (comment) for a summary of the current discussion.

modules/hooks.nix

This came up after a long discussion in [0]. Using this (as default option) is probably the simplest solution and the most natural behaviour. [0]: cachix#405

Although that is not the solution for the performance problems (in form of huge amounts of resources are consumed) of `typos` discussed in [0] and referenced discussions, this is technically the right thing to do and what users expect. [0]: cachix#405

Enable typos.configPath to be of type path. This way, we can use excludes from .typos.toml when running `pre-commit run typos --all-files`. Otherwise, the execution differs from a normal invocation of typos, which is unexpected and leads to wrong results. To not break anything, and to be compliant with the existing API, I modified configPath to be either a Nix path or a string. The main motivation for this change is a repository on my machine where pre-commit ignores directory foobar (which is excluded by .typos.toml) but passes all 65,000 files of foobar as argument to typos. In these situation, typos goes nuts and often the system freezes. So with this change, I prevent that possibly tens of thousands of files that should not be checked in any case are passed to typos, which results in a smooth experience. [0]: cachix#387 (comment)

Although that is not the solution for the performance problems (in form of huge amounts of resources are consumed) of `typos` discussed in [0] and referenced discussions, this is technically the right thing to do and what users expect. [0]: cachix#405

Although that is not the solution for the performance problems (in form of huge amounts of resources are consumed) of `typos` discussed in [0], referenced discussions, and the previous commit, this is technically the right thing to do and what users expect. [0]: cachix#405

phip1611 · 2024-04-22T10:24:37Z

It's a little unfortunate that the whole topic got so complex.

Intermediate summary

I'm working towards that $ pre-commit run [--all-files] behaves similar to $ typos with the exact same set of excludes as this is the correct thing to do.

However, in this whole long process, we discovered/faced/targeted the following problems:

typos usually is not supposed to be invoked with arguments, but the natural way is typos . (config file is implicitly used from the project root). You can use it with arguments but that is not what I've seen - it is not broadly used
pre-commit passes files directly as arguments to tools such as typos
Some folks didn't want to set pass_filenames = false for typos
When typos is invoked with thousands of files (we have that in one of our projects, and we excluded that directory as it is third party code) as argument it literally explodes and CPU and mem consumption are insane; system freezes happen (even with --force-exclude)
For exlusions, typos uses globs, pre-commit uses regex
If .typos.toml uses "globs without *", my approach presented in this PR works and it's fine.
The transformation from globs to regex is not trivial.

Observation/Problems/Next steps

IMHO this is critical and must be addressed to not annoy and confuse people and to not make them drop typos support via pre-commit or use workarounds
I do not see a pragmatic way forward here. Such a rather simple thing is already a time sink. I want to satisfy everyones needs. But I can't spend multiple days on this, unfortunately.

This needs to be addressed. The alternative is that we at my Company manually specify excludes locally and that I create a blogpost or so for typos in pre-commit-hooks.nix for other people having the same problem.

ping @roberth - Thoughts?

phip1611 changed the title ~~typos: fix behaviour~~ typos: use excludes from .typos.toml when pre-commit run typos runs Mar 12, 2024

phip1611 changed the title ~~typos: use excludes from .typos.toml when pre-commit run typos runs~~ typos: use excludes from .typos.toml when pre-commit run typos --all-files runs Mar 12, 2024

phip1611 force-pushed the typos-config-file branch from 913b2bc to 7756e23 Compare March 12, 2024 12:07

phip1611 closed this Mar 19, 2024

phip1611 force-pushed the typos-config-file branch from 7756e23 to e8dc1b4 Compare March 19, 2024 11:41

phip1611 reopened this Mar 19, 2024

phip1611 marked this pull request as draft March 19, 2024 11:46

phip1611 force-pushed the typos-config-file branch from 1e849ba to 1273fa2 Compare March 19, 2024 11:50

phip1611 marked this pull request as ready for review March 19, 2024 11:50

phip1611 force-pushed the typos-config-file branch from 1273fa2 to 6d71e46 Compare April 2, 2024 13:58

phip1611 changed the title ~~typos: use excludes from .typos.toml when pre-commit run typos --all-files runs~~ typos: allow configPath to be of type string or path + take excludes from .typos.toml into account when pre-commit run typos --all-files runs Apr 2, 2024

phip1611 force-pushed the typos-config-file branch from 6d71e46 to 91aeb2a Compare April 2, 2024 14:08

phip1611 commented Apr 8, 2024

View reviewed changes

modules/hooks.nix Outdated Show resolved Hide resolved

astro suggested changes Apr 8, 2024

View reviewed changes

modules/hooks.nix Outdated Show resolved Hide resolved

modules/hooks.nix Outdated Show resolved Hide resolved

modules/hooks.nix Outdated Show resolved Hide resolved

phip1611 force-pushed the typos-config-file branch from 91aeb2a to 7e6fcae Compare April 8, 2024 13:58

phip1611 requested a review from astro April 8, 2024 13:59

astro approved these changes Apr 8, 2024

View reviewed changes

roberth suggested changes Apr 19, 2024

View reviewed changes

phip1611 force-pushed the typos-config-file branch from 1336ec2 to 5dcc460 Compare April 19, 2024 16:57

phip1611 force-pushed the typos-config-file branch from 5dcc460 to 6b6a426 Compare April 19, 2024 16:57

phip1611 force-pushed the typos-config-file branch from 6b6a426 to 74fdf6d Compare April 19, 2024 17:01

phip1611 force-pushed the typos-config-file branch from 74fdf6d to 46b368e Compare April 19, 2024 17:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

typos: allow configPath to be of type string or path + take excludes from `.typos.toml` into account when `pre-commit run typos --all-files` runs #405

typos: allow configPath to be of type string or path + take excludes from `.typos.toml` into account when `pre-commit run typos --all-files` runs #405

phip1611 commented Mar 12, 2024 •

edited

phip1611 commented Mar 12, 2024

phip1611 commented Mar 12, 2024 •

edited

totoroot commented Mar 12, 2024

domenkozar commented Mar 19, 2024

phip1611 commented Mar 19, 2024

domenkozar commented Mar 25, 2024

phip1611 commented Apr 2, 2024

astro left a comment

phip1611 commented Apr 19, 2024 •

edited

domenkozar commented Apr 19, 2024

roberth left a comment

roberth Apr 19, 2024

phip1611 Apr 19, 2024 •

edited

roberth Apr 19, 2024

phip1611 Apr 19, 2024 •

edited

roberth Apr 19, 2024

phip1611 Apr 22, 2024 •

edited

phip1611 commented Apr 22, 2024 •

edited

typos: allow configPath to be of type string or path + take excludes from .typos.toml into account when pre-commit run typos --all-files runs #405

Are you sure you want to change the base?

typos: allow configPath to be of type string or path + take excludes from .typos.toml into account when pre-commit run typos --all-files runs #405

Conversation

phip1611 commented Mar 12, 2024 • edited

The Problem

The Fix

phip1611 commented Mar 12, 2024

phip1611 commented Mar 12, 2024 • edited

totoroot commented Mar 12, 2024

domenkozar commented Mar 19, 2024

phip1611 commented Mar 19, 2024

domenkozar commented Mar 25, 2024

phip1611 commented Apr 2, 2024

astro left a comment

Choose a reason for hiding this comment

phip1611 commented Apr 19, 2024 • edited

domenkozar commented Apr 19, 2024

roberth left a comment

Choose a reason for hiding this comment

roberth Apr 19, 2024

Choose a reason for hiding this comment

phip1611 Apr 19, 2024 • edited

Choose a reason for hiding this comment

roberth Apr 19, 2024

Choose a reason for hiding this comment

phip1611 Apr 19, 2024 • edited

Choose a reason for hiding this comment

roberth Apr 19, 2024

Choose a reason for hiding this comment

phip1611 Apr 22, 2024 • edited

Choose a reason for hiding this comment

phip1611 commented Apr 22, 2024 • edited

Intermediate summary

Observation/Problems/Next steps

typos: allow configPath to be of type string or path + take excludes from `.typos.toml` into account when `pre-commit run typos --all-files` runs #405

typos: allow configPath to be of type string or path + take excludes from `.typos.toml` into account when `pre-commit run typos --all-files` runs #405

phip1611 commented Mar 12, 2024 •

edited

phip1611 commented Mar 12, 2024 •

edited

phip1611 commented Apr 19, 2024 •

edited

phip1611 Apr 19, 2024 •

edited

phip1611 Apr 19, 2024 •

edited

phip1611 Apr 22, 2024 •

edited

phip1611 commented Apr 22, 2024 •

edited