Bazel does not populate/sync all caches in a combined cache

[mendozatudares]

Description of the bug:

When attempting to speed up a remote-cache-enabled build (--remote_cache, HTTP) using a local disk cache (--disk_cache, same drive), I discovered Bazel would not populate the disk cache with artifacts it retrieved from the remote cache for faster access. Bazel also populated only the remote cache upon a cache miss/rebuild. This left us with an empty disk cache that was never accessed.

This seems to be undocumented or inconsistent with anecdotes claiming Bazel reads and writes to both caches as a single cache.

Which category does this issue belong to?

Documentation, Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I was able repro this with just disk caches:

bazel build :target1 --disk_cache=~/.cache/disk1
bazel build :target2 --disk_cache=~/.cache/disk2

# disk1 already populated with target1, does not populate disk2
bazel clean && bazel build :target1 --disk_cache=~/.cache/disk1 --disk_cache=~/.cache/disk2

# disk2 already populated with target2, does not populate disk1
bazel clean && bazel build :target2 --disk_cache=~/.cache/disk1 --disk_cache=~/.cache/disk2

# neither disk1 nor disk2 populated with target3, only disk2 populated
bazel clean && bazel build :target3 --disk_cache=~/.cache/disk1 --disk_cache=~/.cache/disk2

In this example, :targetX's do not share any dependencies.

Which operating system are you running Bazel on?

Linux, macOS, Windows

What is the output of bazel info release?

release 7.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

1. BuildBuddy article written sometime around Bazel 5.0-5.1 reports that this "combined cache" behavior should be the default, but can be tuned for different behavior with certain flags.
2. remote_http_cache + experimental_remote_disk_cache from bazel-discuss group, where a suggested approach was to use a proxy to handle the caches.

Any other information, logs, or outputs that you want to share?

No response

[coeuvre]

The repro doesn't make sense to me: you cannot combine multiple disk caches, only the last --disk_cache is used.

Things have changed a lot since Bazel 5. For example we have enabled "Build without the Bytes" by default in Bazel 7, therefore the intermediate outputs are not downloaded by Bazel which explains why some outputs are not downloaded to disk cache.

However, I doubt

I discovered Bazel would not populate the disk cache with artifacts it retrieved from the remote cache for faster access.

and

This left us with an empty disk cache that was never accessed.

Can you provide a proper repro?