Stop copying jar files #1074
Open
Stop copying jar files #1074
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
POM target shouldn't get a http_file target at all, they don't have a a file or a sha256, there's nothing to download
If the file is None then that means we shouldn't pass this to the http_file call, most likely this entries also have a None sha256 sum
manuelnaranjo
requested review from
jin,
Wyverald,
shs96c and
cheister
as code owners
manuelnaranjo
force-pushed
the
mnaranjo/no-copy
branch
from
3c1a089
to
8abcc5f
Compare
In Java 13 -noverify got marked as deprecated and may be dropped in the future
Since we have downloaded_file_path that points to the actual jar name in our http_file invocations we can directly consume from those targets instead of copying, this has an impact not just in IO as we reduce copying, but also reduces the amount of targets bazel is aware as we don't need an extra node, reducing the amount of RAM needed for rules_jvm_external slightly. In large maven repositories we use at Booking.com we saw a decrease in the disk cache from 2.4GB to 300MB, which aligns with a repository_cache of roughly 2.1GB. The time it takes to do a `bazel build @maven//...` went from 120 seconds to 50 seconds on a i9-13900H using 75% of the cores.
manuelnaranjo
force-pushed
the
mnaranjo/no-copy
branch
from
8abcc5f
to
d88ca8e
Compare
|
@manuelnaranjo, the idea seems like a good one, but I note that none of the CI tests are working yet. You can run these locally using |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#1070
Stop making use of a genrule that copies the already downloaded jar handled by http_file file into the target repository and instead make use of an alias for proxying the artifact into the target repository.