Bazel 8: switch to Blake3 as the default hash function

[sluongng]

Currently, we are using SHA256 as the default hash function of Bazel. However, for builds with large binary artifacts, alternative hash functions such as SHA256Tree, Blake3, or PSHA2 perform much better.

Among these newer hash algorithms, Blake3 is quite popular and other build tools have started to adopt it. The tooling is also widely available with b3sum CLI https://crates.io/crates/b3sum being a direct replacement for sha256sum.

So this discussion is to put forward the idea that we should flip the default to Blake3 in a future version of Bazel (8?), and start accounting for the tasks needed to make it happen.

[fmeum]

This discussion prompted me to add a benchmark for hash functions to Bazel: #22017

The preliminary result is that while BLAKE3 is about 8x faster than SHA2-256 on Linux x86_64, I found it to 25% slower on an M3 Mac. That is surprising, but if true, we would of course need to fix that before pursuing making it the default.

Edit: It's not that surprising since NEON only has 128 bit SIMD registers and Apple Silicon provides highly optimized hardware intrinsics for SHA256 (BLAKE3-team/BLAKE3#315).

Edit: Nvm, I forgot -c opt, which makes an enormous difference on macOS. Updated the numbers.

FYI @brentleyjones

[EdSchouten]

@EdSchouten Are you planning to work on adding SHA256Tree support to Bazel?

Let me discuss this internally and get back you on this soon!

[EdSchouten]

The preliminary result is that while BLAKE3 is about 8x faster than SHA2-256 on Linux x86_64, I found it to 25% slower on an M3 Mac.

I'm amazed that the difference on Apple Silicon is so small. Are you sure the SHA-256 implementation you used is actually using the special SHA-256 hardware instructions?

I know that if you use those, you can easily reach 4 GB/s. With BLAKE3 I wasn't able to get anywhere near that.

[EricBurnett]

Re psha2: We'd consider it, yes. We've published the definition already (https://docs.google.com/document/d/1JnrL4iFnkWrZUYXiIAZC4KLzOgj0y0gFNd4jZyZT590/edit?usp=drivesdk), and I think code is on the table if it can be reasonably extracted. That said, we're talking about V3 as well right now, which would likely involve switching to a structural hash of some form rather than anything that requires the whole input to hash upfront. So PSHA2 might be the wrong solution to put effort into.

For similar reasons I'd probably be against a change in the default. SHA-256 works well enough, and in most contexts the hash function isn't the key bottleneck to performance so much as the API requirement to hash whole files upfront, which a drop -in replacement won't solve.

[EdSchouten]

@EdSchouten Are you planning to work on adding SHA256Tree support to Bazel?

Giving it some more thought, I think we most likely will not.

From our perspective we still want to work on designing a new version of the remote execution protocol (an REv3) that eliminates the existence of large CAS objects entirely, where files are modelled as shallow Merkle trees, using techniques such as Content Defined Chunking. Once we have that, there no longer is a real need for using hashing algorithms that are SIMD aware such as BLAKE3, SHA256TREE or PSHA2. Ordinary hashing algorithms such as SHA-256 should provide near-identical performance, assuming you feed chunks to a "multi-buffer" hashing implementation (see 1, 2).

[sluongng]

Another reference point https://bench.cr.yp.to/results-hash.html

[EdSchouten]

BLAKE3 advertises itself as being faster than SHA-256. But it is important to realize that these gains only appear on larger inputs. For small inputs the performance of BLAKE3 is significantly worse than SHA-256.

Especially if you perform builds with remote execution and '--remote_download_minimal' enabled, you spend an awful amount of time hashing small objects. Not large ones. Think of the steps that Bazel needs to perform to construct a Merkle tree of an action. Many, many, many messages that are less than a kilobyte in size.

I therefore think that it's fine to have Bazel support BLAKE3, but enabling it by default? That doesn't sound like a good idea.

[sluongng]

From the conversation above, I concluded that there is no significant motivation for us to switch away from the default hashing algorithm anytime soon. 👍
This is a great source of information and references! Thanks, everyone!

Closing this.