Caching kcov

[orenbenkiki]

I'm working on maximizing caching in Travis.

I see that when kcov is build locally it is placed in a local .build version. This means it is rebuilt again for every project (and after clean).

I'm caching $HOME/.cache on Travis, (which covers $HOME/.cache/cargo-make). If kcov was cached there, it would have been reused between builds.

The scheme might be something like this:

• Try the system kcov.
• If that fails, try $HOME/.cache/cargo-make/kcov/...
• If that fails, build $HOME/.cache/cargo-make/kcov.$NEW_UNIQUE/...
• ln -sf $HOME/.cache/cargo-make/kcov.$NEW_UNIQUE $HOME/.cache/cargo-make/kcov (this is atomic and doesn't have a time window where $HOME/.cache/cargo-make/kcov doesn't exist).
• for OLD_KCOV in $HOME/.cache/cargo-make/kcov.*; do test $OLD_KCOV = $HOME/.cache/cargo-make/kcov.$NEW_UNIQUE || rm -rf $OLD_KCOV; done
• Use $HOME/.cache/cargo-make/kcov/... which should work now (and be reused later).

[sagiegurari]

cargo make will only install kcov if its not available or if too old version.
which means for local dev, it will not reinstall.
for travis specific caching solution, i would prefer not to have that logic inside cargo make core tasks. it would be very complex to support and maintain a solution for every ci provider.

[orenbenkiki]

I agree it isn't simple. But it isn't travis-specific. The only requirement from the CI is to be able to cache "something" between builds.

If cargo-make places everything caching-worthy in $HOME/.cache/cargo-make, it is up to each CI (or the CI-specific configuration scripts) to figure out how to cache it using that "something".

There is a specific way to do this in travis, but that's not something cargo-make needs to worry about.

That said, it is a fair decision to not support caching at all, or caching of kcov in particular. This is a feature request, not a bug report ;-)

[orenbenkiki]

FWIW, I implemented this in scmf: https://github.com/orenbenkiki/scmf/blob/master/install_kcov.sh
And am using it, for example, in: https://github.com/orenbenkiki/cargo-todox/blob/master/.travis.yml

[sagiegurari]

that's great. from quick look, it looks like only the unzipped sources are cached, not the compiled binary. right? so you do compile it each time, right?
if you could just build it and add it to the path, you might be able to cache the whole thing

[orenbenkiki]

Actually all the build files are cached in the build directory. Re-running make is just for safety/completeness (and is very fast). In effect the only "real work" done is the install which places kcov in the system path )which is also very fast - just file copy). Since I was only going for Travis, I moved the apt commands to the .travis.yml file, but if I wanted to make this more universal I'd move the apt-get install of the needed dynamic libraries to the (unconditional part of the) script.

[emschwartz]

I'm using CircleCI and would love to be able to cache kcov as well (installing it each time currently takes ~45 seconds per build). Circle has a similar method for adding paths to the cache and it would be very straightforward to add whatever path the build files were saved to if they're put in a consistent location.

[sagiegurari]

i could enable you to decide where to install kcov (see #98) which then you could put in custom location and cache that location in your CI build.
would that work?

[orenbenkiki]

Yes, that would solve the problem, at least for me. TravisCI allows specifying which (not-under-current-build) directories are cached, so installing KCOV using something like ~/.cache/kcov would allow reusing it in following executions.

Thanks!

[sagiegurari]

going to solve it via #98 and this one would be a documentation only basically.

[sagiegurari]

@orenbenkiki @emschwartz if you could please install cargo-make from master and verify its working good for you?
basically you need to define a new env var CARGO_MAKE_KCOV_INSTALLATION_DIRECTORY which will be a directory that kcov will be installed to and should be defined as cached by the CI provider you are using.
you can define the env var in your makefile.toml or in your CI yaml file. as long as cargo-make sees it.

[orenbenkiki]

I tried this on my development PC and it seems to work fine. I didn't try in on Travis but it "should" work the same way...

[sagiegurari]

thanks