Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build cache miss when jars have been hashed with different normalization configuration #16144

Closed
ghale opened this issue Feb 13, 2021 · 3 comments · Fixed by #16145
Closed

Build cache miss when jars have been hashed with different normalization configuration #16144

ghale opened this issue Feb 13, 2021 · 3 comments · Fixed by #16145
Assignees
@ghale
Labels
a:bug affects-version:6.8 in:core DO NOT USE
Milestone
6.8.3

Comments

@ghale
Copy link
Member

ghale commented Feb 13, 2021

We've received multiple reports of cache misses on runtime normalization input properties for jar files that should be identical. At least one reporter confirmed that using a different gradle user home resolved the issue. It turns out some of the changes to runtime classpath normalization in 6.8 don't accurately capture changes to the normalization configuration, which can lead to a sequence of events like the following:

  1. CI server runs a build that hashes a 3rd party library in a project without properties file normalization rules. The hash of this file is cached in the resources cache.
  2. Another build runs that does have properties file normalization rules, but when the hasher gets to the library, it does not properly capture the rules when calculating the key for the resources cache and calculates the same key as the build without normalization rules. Rather than hashing the file again, it gets a hit on the resources cache and incorporates this hash into the task's build cache key.
  3. A build on another system that does not have the library's hash in the resources cache runs the build with the properties file normalization rules, gets a miss for that file in the resources cache, and does the fully normalized hash of the library. The resulting hash is different which results in a different cache key for the task, causing a build cache miss.

Expected Behavior

We should get.a build cache hit on the task.

Current Behavior

The build cache key can be incorrectly calculated on one side or the other, causing a miss.

Context

Poorer build perfomance due to cache misses.

Steps to Reproduce

See above.
@ghale ghale added this to the 6.8.3 milestone Feb 13, 2021
@ghale ghale self-assigned this Feb 13, 2021
@ghale ghale mentioned this issue Feb 13, 2021
Merged
ghale added a commit that referenced this issue Feb 13, 2021
@ghale
Fixes issue with file hashing cache
a4b8590 
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 16, 2021
@ghale
Fixes issue with file hashing cache
5312ef5 
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
@wolfs wolfs modified the milestones: 6.8.3, 7.0 RC1 Feb 17, 2021
@wolfs wolfs mentioned this issue Feb 17, 2021
Closed
@wolfs
Copy link
Member

wolfs commented Feb 17, 2021

I created #16192 to backport the fix to 6.8.3.

@wolfs wolfs mentioned this issue Feb 19, 2021
Merged
@wolfs wolfs modified the milestones: 7.0 RC1, 6.8.3 Feb 19, 2021
github-actions bot pushed a commit to w4bo/github-actions-helloworld that referenced this issue Feb 23, 2021
Update gradle wrapper to 6.8.3
f085ae3 
This is a patch release for Gradle 6.8.

This fixes [a critical bug](gradle/gradle#16144) present in Gradle 6.8, 6.8.1 and 6.8.2.

[All issues fixed in this patch release](https://github.com/gradle/gradle/milestone/171?closed=1)

Please don’t use the original 6.8 release or previous patch releases, and instead upgrade to 6.8.3.

## Upgrade Instructions

Switch your build to use Gradle 6.8.3 by updating your wrapper:

`./gradlew wrapper --gradle-version=6.8.3`

See the [Gradle 6.x upgrade guide](https://docs.gradle.org/6.8.3/userguide/upgrading_version_6.html#changes_6.8) to learn about deprecations, breaking changes and other considerations when upgrading to Gradle 6.8.3.

## Reporting Problems

If you find a problem with this release, please file a bug on [GitHub Issues](https://github.com/gradle/gradle/issues) adhering to our issue guidelines.
If you're not sure you're encountering a bug, please use the [forum](https://discuss.gradle.org/c/help-discuss).
tobiasschaefer added a commit to camunda-community-hub/micronaut-camunda-platform-7 that referenced this issue Feb 23, 2021
@tobiasschaefer
Update to gradle 6.8.3 due to a critical bug, see gradle/gradle#16144
467aa96
martisaw pushed a commit to camunda-community-hub/micronaut-camunda-platform-7 that referenced this issue Feb 23, 2021
@tobiasschaefer @martisaw
Update to gradle 6.8.3 due to a critical bug, see gradle/gradle#16144
51c2f0c
@carstenhag
Copy link

Is it really a "critical bug" as per the 6.8.3 release notes? Seems like it's only performance relevant? :)

@wolfs
Copy link
Member

wolfs commented Feb 23, 2021

This can lead to incorrect builds, which makes it a critical bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ghale ghale

Labels
a:bug affects-version:6.8 in:core DO NOT USE
Projects
None yet
Milestone
6.8.3
Development

Successfully merging a pull request may close this issue.

Fix issue with file hash cache during runtime classpath normalization gradle/gradle
3 participants
@wolfs @ghale @carstenhag