Dependency which relies on ~/.cargo/registry/src

[max-sixty]

Thanks for the excellent GH Action!

duckdb seems to rely on files in ~/.cargo/registry/src — full context over at PRQL/prql#2870.

Would it be possible to optionally disable removing this path? Or offer a glob for disabling some paths from being removed?

This is related to #100 — though in this case, it makes the whole build take almost as much time as without a cache (feel free to close as dupe if it is indeed the same though)

[max-sixty]

(duckdb/duckdb-rs#178 for the issue on that repo)

[max-sixty]

Thanks a lot @Swatinem ! We'll test it now

[0o-de-lally]

Thanks @Swatinem I was just writing up a comment here, then I noticed you just shipped a change!

For future reference to others with large projects. If you have dependencies which themselves have large dependency lists (especially c ones it seems) artifacts may be saved in ./cargo/registry. This gh action priori to current version not preserving that directly (intentionally).

There was a fork made by @bmwill https://github.com/bmwill/rust-cache, which allows you to specify a path to save. Though that one is not maintained and seems like some gh warnings in the deployment may soon become errors.

Found it by following the breadcrumbs here:
#100
bmwill/sui@4735504

[max-sixty]

This works! Thank you v much @Swatinem .

FYI — if interesting — we still get one line stating Compiling libduckdb-sys v0.8.1 https://github.com/PRQL/prql/actions/runs/5742050637/job/15564225629#step:9:16, but this takes less than a second, rather than six minutes.

[0o-de-lally]

@max-sixty I'm testing this now too. Slightly off topic, have you encountered an issue like this? #155

[0o-de-lally]

@Swatinem Worked for me! Great stuff!

[max-sixty]

@max-sixty I'm testing this now too. Slightly off topic, have you encountered an issue like this? #155

Not yet! I eagerly await my meeting with it though...

[bmwill]

There was a fork made by @bmwill https://github.com/bmwill/rust-cache, which allows you to specify a path to save. Though that one is not maintained and seems like some gh warnings in the deployment may soon become errors.

Yeah I threw that together a while back in order to fix caching of the rocksdb-sys crate IIRC. I had meant to look into upstreaming a fix but as you can see i never got around to doing so :) I'll need to take the time to see whats changed in the meantime and maybe the fork won't be needed anymore (either that or i'll need to fix those gh warnings)

[0o-de-lally]

@bmwill the last release here likely fixed what you originally were trying to solve.

Off topic, I'm curious to see how you all solved these types of cache issues: #155

[max-sixty]

Ah, unfortunately I spoke too soon, I'm sorry.

FYI — if interesting — we still get one line stating Compiling libduckdb-sys v0.8.1 https://github.com/PRQL/prql/actions/runs/5742050637/job/15564225629#step:9:16, but this takes less than a second, rather than six minutes.

Unfortunately while this is true, a few lines later we get Checking duckdb v0.8.1, and that takes eight minutes.

I can't seem to reopen — @Swatinem let me know if you'd like to reopen, or I should start a new issue (or that this is the issue of the dependency rather than this repo and we should reopen upstream?)

[Swatinem]

I might try myself again at fixing this, I thought keeping the src of X-sys crates around would solve this, but that does not seem to be true, though in my own testing in this repo with jemalloc, it seems to work.

[max-sixty]

This seems like it might be fixed! https://github.com/PRQL/prql/actions/runs/5986134105/job/16238928467 is an example of it not running Compiling libduckdb-sys v0.8.1. The upstream dependency is the same.

I couldn't see a commit which fixed this, but I don't have much context. I guess it's possible it's something non-deterministic. Though it seems fairly consistent recently...

[max-sixty]

We're reverting our workaround for this: #3449, given it seems to be fixed.

I'll close this issue, but feel free to reopen if I'm mistaken (possibly something else changed such that this doesn't affect us any longer...)

Thank you!