Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn when adding large local paths #4034

Open
roberth opened this issue Sep 18, 2020 · 3 comments · May be fixed by #10661
Open

Warn when adding large local paths #4034

roberth opened this issue Sep 18, 2020 · 3 comments · May be fixed by #10661
Labels

Comments

@roberth
Copy link
Member

roberth commented Sep 18, 2020

Is your feature request related to a problem? Please describe.

The dumping large paths message has been slightly helpful by providing a hint that something big is getting added to the store each time.
This message is disappearing with #4030 (unless you use an older daemon), so you'll be filling up your store with slight variations of large paths if you aren't careful. Virtually nobody checks the size of their derivation closures.

Describe the solution you'd like

Provide a warning when adding a large local path to the store, when it is

  • not the output of an evaluation time fetcher with a content hash, because these tend to be unavoidable and tend to change less often
  • larger than a threshold that defaults to, say, 250 MB
  • is not a single file (which would seem rather intentional)

Example warning

You're copying to the Nix store from path `/home/user/workspace` which seems to be rather large (> 100MB). In most cases this happens by accident because the path wasn't filtered correctly. Alternatively, you may want to split your build into smaller derivations if possible, or set `warn-large-path-threshold = 250 #MB` to a larger value.

Describe alternatives you've considered

Nix flakes do alleviate the problem by only including (potentially unstaged!) files that are listed in git's staging area.
However, user can still reference paths that aren't in a repo.

Additional context

@cole-h
Copy link
Member

cole-h commented Sep 20, 2020

The "you'll be filling up your store with slight variations of large paths if you aren't careful" makes me think the overhaul should re-implement this warning before it could be considered merge-worthy. Since we currently have a warning for this, I feel it would be a step back to have it removed with no alternative.

I also don't think the "is not a single file" criterion is a great idea to not warn on -- it might be intentional, but it also might not. Maybe there are files a1 and A1: A1 is 100G, while a1 is 100K. Holding shift slightly too long, and now Nix is trying to add 100G to the store. A contrived example, but I hope you get my point.

Sorry for being so negative 😬

@stale
Copy link

stale bot commented Mar 19, 2021

I marked this as stale due to inactivity. → More info

@stale
Copy link

stale bot commented Apr 19, 2022

I closed this issue due to inactivity. → More info

@stale stale bot closed this as completed Apr 19, 2022
@thufschmitt thufschmitt reopened this Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants