Slow compiling every time?

[bazaglia]

Description

I've already gone through the docs and got these rules working with TypeScript. Almost everything worked, except setting output_mode to NO_PREFIX in js_grpc_node_library throws an error, so I didn't use this option even though I wanted to and stuck with the default.

But what surprises me is that every time I build – even if I try to build just the proto_library, so that's unrelated to my programming language – it takes a lot of time to compile. It seems to be building protobuf from the source every time? It is resulting in 90+ seconds to build only a proto_library. Anything I can do to have it compiling faster? I'm running macOS.

[aaliddell]

This is related to #161. As far as I can tell, this is a bug with the protobuf repo, whereby it is very sensitive to cache misses. The fact you are seeing a full rebuild every time locally is particularly odd though, is this completely reproducible?

I suggest you open an issue on the protobuf repo such that they have visibility of these issues, since nothing has changed on this front for a long while.

[bazaglia]

@aaliddell Thank you for clarifying. I've actually seen the issue you mentioned above but I thought it was unrelated. Yes, it is completely reproducible. I just opened a new shell and used the bare minimum env variables as you said in the other issue and it won't help.

Something worth mentioning is that I had protobuf@3.17 installed through homebrew before and these rules were completely failing. After I uninstalled it, the build worked (without protoc binary on my OS), but long times, so I had the impression protobuf was being compiled from the source all the time. Later I copied the compiled protoc for MacOS to /usr/local/bin and can verify it's in my PATH and shows version 3.19. Even though the rules from this repo doesn't fail with protoc on my OS installed like that, it still takes 90+ seconds, making me wonder if it isn't using protoc from my OS?

I'd love to bring visibility to protobuf repo but if I remove the rules from this repo from workspace, proto_library compiles super fast even at first time. So I'm wondering if those rules could be related to slowing down things that much?

INFO: Elapsed time: 89.793s, Critical Path: 21.57s
INFO: 386 processes: 7 internal, 379 darwin-sandbox.
INFO: Build completed successfully, 386 total actions
INFO: Build completed successfully, 386 total actions

[aaliddell]

So if you remove these rules, you'll end up using the pre-built protoc in rules_proto, which bypasses the build completely (this was done due to this exact rebuild bug). We are unable to use that pre-built protoc (see #88) in these rules.

The fact you see it on every build is particularly interesting though, as usually it's a little less sensitive than that. So even on two consecutive bazel build //... in the same directory you see it rebuild protobuf? If so and we can isolate what is special about your setup, we might have a hope of finding the cause of this bug! I don't have a MacOS system to play with to see if I can recreate this.

The pre-installed protoc also shouldn't interfere with these rules or rules_proto, since protoc is always referenced as a tool directly. So again something odd is going on if uninstalling it changed anything...

[bazaglia]

That was a nice clue. I tried to isolate anything that could be unique to my environment. I was using bazel installed through yarn (bazelisk). When I installed bazel 4.2.2 on my OS directly and tried to run the same command twice it didn't cause protobuf to recompile on the second time. Also, running from the first time took about 45s against 90s when using bazelisk wrapper.

[aaliddell]

OK, I'll see if I can recreate the cache miss by just using bazelisk via NPM/Yarn.

Could you check you are getting the same Bazel version through both methods? e.g. bazel --version and bazelisk --version, or however you were running in the two different methods.

[bazaglia]

Same versions (4.2.2).

Bazel:

$ bazel --version
bazel 4.2.2-homebrew

Bazelisk:

$ yarn bazel version
Bazelisk version: v1.11.0
Build label: 4.2.2
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 2 18:28:52 2021 (1638469732)
Build timestamp: 1638469732
Build timestamp as int: 1638469732

Since 90 extra seconds in a pipeline makes a huge difference I've investigated this further as I intend to use a binary instead of compiling at the first time (when a pipeline runs). I'm still new to Bazel but I've found a workaround to --proto_compiler not being exposed to starlack and maybe that leads to the same problem you described. There is a patch that I'll try and adapt to this project. I'll comment here if I succeed: bazelbuild/bazel#8485 (comment)

[bazaglia]

I can confirm the method from the comment on the issue I shared above worked and I'm able to always use binary instead of compiling protobuf.

For whoever is interested, I just used the files like specified on the issue mentioned and adapted to this project. The only change was to the git patch. For the current version of rules-proto-grpc it looks like the following:

diff --git a/protobuf/toolchains.bzl b/protobuf/toolchains.bzl
index ca90158f..7c5f4e77 100644
--- a/protobuf/toolchains.bzl
+++ b/protobuf/toolchains.bzl
@@ -12,7 +12,7 @@ protoc_toolchain = rule(
     attrs = {
         "protoc": attr.label(
             doc = "The protocol compiler tool",
-            default = "@com_google_protobuf//:protoc",
+            default = "@//backend/protobuf/rules:protoc",
             executable = True,
             cfg = "exec",
         ),

In my workspace file:

http_archive(
    name = "rules_proto_grpc",
    patch_args = ["-p1"],
    patches = ["//backend/protobuf/patches:rules-proto-grpc-protoc.patch"],
    sha256 = "8383116d4c505e93fd58369841814acc3f25bdb906887a2023980d8f49a0b95b",
    strip_prefix = "rules_proto_grpc-4.1.0",
    urls = ["https://github.com/rules-proto-grpc/rules_proto_grpc/archive/4.1.0.tar.gz"],
)