Remote BuildCache calculation is broken due to InstrumentingClasspathFileTransformer generates unstable jars

[AndreasTu]

The Gradle Remote BuildCache is broken for Tasks types defined by Gradle-Plugins which are not part of the Gradle distribution.
Gradle calculates a BuildCache hash for the task implementation via the classloaders.
The classloader-hash calculation is done on the Transformed Jar files from ClasspathTransformerCache of the folder jars-8.
But the InstrumentingClasspathFileTransformer class does not produce the same Jar file content on different machines.
This leads to different hashes for the task implementation, which leads to a remote BuildCache miss.

One of the causes why InstrumentingClasspathFileTransformer is generating unstable output is:

 // Remove the signature from the manifest, as the classes may have been instrumented
Manifest manifest = new Manifest(new ByteArrayInputStream(entry.getContent()));

This uses the java.util.jar.Manifest API, which internally represents the manifest entries with a HashMap, which is not stable in different processes. This leads to different Manifest.MF files on different machines.

Expected Behavior

A @CacheableTask task from an external plugin shall be remote cacheable.

Current Behavior

The task contributed from a external plugins gets different ClassLoader Hashes, so Gradle generates a different hash for the task implementation class.

Context

Steps to Reproduce

Add an external plugin in the buildSrc/build.gradle which supports Caching.
Execute the task on two machines.
The second machine will not get the Cache entry of the first.

Note: This has to be a non-Gradle Task type, because for the Built-in type gradle uses another "constant" hash value, without tranforming the gradle plugin jar files.

Your Environment

Internal BuildScans are available.
Gradle: 6.5.1
Java: 11.0.5
Windows

[AndreasTu]

@wolfs can you please have a look at it, because it is BuildCache related.
@adammurdoch can you also have a look at it, you where the last person editing the InstrumentingClasspathFileTransformer

[wolfs]

I think this is a duplicate of #13874 and has been fixed in 6.6.

[wolfs]

In 6.6 we also normalize manifest, so the manifest generation doesn't need to be reproducible. Though it should be: #13875.

[AndreasTu]

Thanks for the fast information!