Caching of config files in .bloop

[pvid]

Motivation

I use metals with a fairly large monorepo, where bloopInstall takes ~3 minutes. I find that I have to import builds multiple times a day because of switching to different git branch that contains changes to build files.

Proposal

Cache the bloop configuration files and compile artifacts for a build digest so that changing between "hot" branches with different build needs just reindexing.

This could be achieved by changing the bloop output directory for each build digest. It would add complexity to the way bloop files are handled and also the bloop cli would not work out of the box.

There is also another trade-off - bloop caches compile artifacts based on some hash of the build definition. So that means are if you change the build definition just for subset of modules in your project, much of the compilation work that bloop has done in the past is reused. If every build digest would have a separate bloop folder, this everything would need to be recompiled again. That could be solved by keeping bloop configs and bloop compile outputs in separate folders - configs could be addressed by build digest, but compile artifacts could be shared for all build digests.

Do you think this is something worth pursuing and incorporating into metals?

[tgodzik]

Thanks for the raising this topic! I think I had the same issues while working on some bigger features esspecialy (in Bloop for example :S).

This might be useful for some people, but I wonder if we can achieve something like this via low-tech solution. Doing it properly might be hard to achieve and maybe it would be useful to just have duplicated workspaces in some cases?

Maybe we could approach this differently by trying to minimize the time spent in bloopInstall. Ideally it should be very fast if we already imported it once, so a bigger gain might be figuring out what does the plugin do wrong.

[pvid]

Thank you for the response!

Regardless of what the sbt bloop plugin does, it takes a while for sbt to load the project and resolve dependencies. Especially for a large project. I am working in a monorepo with 100+ modules.

I will try to come up a with some prototype sometimes in coming weeks

[pvid]

I have hacked together a a caching mechanism for .bloop config, see commit pvid@fa6fbeb It actually works quite well and made my workflows much more enjoyable. However, it is certainly not ready to be included in metals in this form

[tgodzik]

Looks sensible, it's not a lot of code, so I think it could be possible to include it.

[pvid]

I am not that fond of using some magic files for build digests. I think it should be better integrated with how metals keeps track of build hashes

There is also the issue of expiration/cleanup, otherwise they would stay there forever.

[tgodzik]

We could put that into the H2 database maybe? And perform cleanup based on last access?