Expose crates_vendor deps to bzlmod #2372

dtolnay · 2023-12-28T16:57:32Z

Without this, I have been unable to update https://github.com/dtolnay/cxx to bzlmod. When I run bazel run //third-party:vendor (this rule) it failed with "unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~0.35.0":

ERROR: no such package '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~0.35.0]//file': The repository '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~0.35.0]' could not be resolved: No repository visible as '@cargo_bazel.buildifier-linux-amd64' from repository '@@rules_rust~0.35.0'
ERROR: ~/.cache/bazel/_bazel_david/ebce1d0721fb68dda9c70c0dd1405803/external/rules_rust~0.35.0/crate_universe/private/vendor/BUILD.bazel:3:27: no such package '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~0.35.0]//file': The repository '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~0.35.0]' could not be resolved: No repository visible as '@cargo_bazel.buildifier-linux-amd64' from repository '@@rules_rust~0.35.0' and referenced by '@@rules_rust~0.35.0//crate_universe/private/vendor:buildifier'
ERROR: Analysis of target '//third-party:vendor' failed; build aborted: Analysis failed

After this change, it works. My MODULE.bazel can be seen in dtolnay/cxx#1294.

crate_universe/private/vendor_utils.bzl

illicitonion

Thanks for the fix! I think this looks reasonable (though cc @matts1 for a better look) - would it be possible to expand https://github.com/bazelbuild/rules_rust/tree/main/examples/bzlmod/hello_world and https://github.com/bazelbuild/rules_rust/blob/main/.bcr/presubmit.yml and https://github.com/bazelbuild/rules_rust/blob/main/.bazelci/presubmit.yml to exercise the workflow that was broken for you?

illicitonion · 2023-12-28T17:44:36Z

#2373 should help with CI

dtolnay · 2023-12-28T17:48:14Z

Thanks!

I've added the crates_vendor usage in the first commit and implemented the fix in the second commit, so you can run bazel run //third-party:vendor in the first commit to see that it fails.

ERROR: no such package '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]//file': The repository '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]' could not be resolved: No repository visible as '@cargo_bazel.buildifier-linux-amd64' from repository '@@rules_rust~override'
ERROR: f2e42d6afe2730a9d87691f9ba152a3d/external/rules_rust~override/crate_universe/private/vendor/BUILD.bazel:3:27: no such package '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]//file': The repository '@@[unknown repo 'cargo_bazel.buildifier-linux-amd64' requested from @@rules_rust~override]' could not be resolved: No repository visible as '@cargo_bazel.buildifier-linux-amd64' from repository '@@rules_rust~override' and referenced by '@@rules_rust~override//crate_universe/private/vendor:buildifier'
ERROR: Analysis of target '//third-party:vendor' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.350s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

dtolnay · 2023-12-28T17:50:31Z

examples/bzlmod/hello_world/third-party/extension.bzl

+"""Module extensions for using vendored crates with bzlmod"""
+
+load("//third-party/crates:defs.bzl", _crate_repositories = "crate_repositories")
+
+def _crate_repositories_impl(module_ctx):
+    direct_deps = _crate_repositories()
+    return module_ctx.extension_metadata(
+        root_module_direct_deps = [repo.repo for repo in direct_deps],
+        root_module_direct_dev_deps = [],
+    )
+
+crate_repositories = module_extension(
+    implementation = _crate_repositories_impl,
+)


Separately, we should make crate_universe automatically generate a suitable module extension. Nothing in this file is project-specific.

I think #2181 is probably the umbrella issue for this... cc @matts1

dtolnay · 2023-12-28T17:57:06Z

.bcr/presubmit.yml

+      run_targets:
+        - "//third-party:vendor"


In CI this is succeeding on Linux ("bzlmod BCR presubmit on :ubuntu: 20.04 LTS (OpenJDK 11, gcc 9.4.0)") but failing on macOS ("bzlmod BCR presubmit on :darwin: (OpenJDK 11, Xcode)").

/private/var/tmp/_bazel_buildkite/4894c9dc50e84673867cbd80d0914a01/external/rules_rust~override~rust~rust_host_tools/bin/cargo: /private/var/tmp/_bazel_buildkite/4894c9dc50e84673867cbd80d0914a01/external/rules_rust~override~rust~rust_host_tools/bin/cargo: cannot execute binary file ERROR: /Users/buildkite/builds/bk-imacpro-7/bazel/rules-rust-rustlang/examples/bzlmod/hello_world/third-party/BUILD.bazel:3:14: //third-party:vendor depends on @@rules_rust~override~cargo_bazel_bootstrap~cargo_bazel_bootstrap//:binary in repository @@rules_rust~override~cargo_bazel_bootstrap~cargo_bazel_bootstrap which failed to fetch. no such package '@@rules_rust~override~cargo_bazel_bootstrap~cargo_bazel_bootstrap//': Process exited with code '126'

It looks like it's trying to use a Linux cargo binary to build cargo-bazel - I repro locally... See #2021 (comment)

cc @matts1 @scentini

It looks like the MODULE.bazel.lock contains platform-specific information, which gets re-used if nothing else has changed in the repo:

"rust_host_tools": { "bzlFile": "@@rules_rust~override//rust:repositories.bzl", "ruleClassName": "rust_toolchain_tools_repository", "attributes": { "name": "rules_rust~override~rust~rust_host_tools", "exec_triple": "x86_64-unknown-linux-gnu", "target_triple": "x86_64-unknown-linux-gnu", "allocator_library": "@rules_rust//ffi/cc/allocator_library", "dev_components": false, "edition": "2021", "rustfmt_version": "nightly/2023-12-07", "sha256s": {}, "urls": [ "https://static.rust-lang.org/dist/{}.tar.gz" ], "version": "1.74.1" } },

Looks like we need to do something like this: https://github.com/bazelbuild/rules_python/pull/1433/files

Good find. See also bazelbuild/rules_python#1643. Otherwise, according to bazelbuild/rules_python#1433 (comment), you and all downstream users need to regenerate their lockfiles separately from each platform they support. For example I, on Linux, would not be able to generate a lockfile that passes checks on macOS.

Started a thread on Slack: https://bazelbuild.slack.com/archives/C014RARENH0/p1703787329091939 - hopefully there's a recommendation we can follow :)

I tested this in #2374 - it works, but looks like it's going to churn the lockfile every time someone switches platform. Hopefully we can get some useful pointers soon :)

Yeah, for now let's just delete + .gitignore the MODULE.bazel.lock file in examples/bzlmod/hello_world and be happy

I believe there's a way to ensure that you don't use the lockfile for specific module extensions / repository rules. Don't know how though. @Wyverald should know, he implemented the lockfile.

dtolnay · 2023-12-28T18:42:50Z

Rebased on #2373 + cherry-picked the lockfile deletion from #2374.

illicitonion

Thanks!

@matts1 Can you take a look when convenient and follow up with anything you'd like added/changed? :)

illicitonion · 2023-12-28T18:48:02Z

Aah sorry we accidentally merged something else while CI was churning - if you can do one more rebase we can get this merged!

dtolnay mentioned this pull request Dec 28, 2023

Switch to bzlmod dtolnay/cxx#1294

Merged

dtolnay commented Dec 28, 2023

View reviewed changes

crate_universe/private/vendor_utils.bzl Show resolved Hide resolved

illicitonion reviewed Dec 28, 2023

View reviewed changes

dtolnay force-pushed the cratesvendordeps branch 4 times, most recently from be82581 to e4e83f3 Compare December 28, 2023 17:41

dtolnay commented Dec 28, 2023

View reviewed changes

illicitonion mentioned this pull request Dec 28, 2023

Support cargo_bazel_bootstrap for bzlmod #2021

Merged

dtolnay commented Dec 28, 2023

View reviewed changes

dtolnay force-pushed the cratesvendordeps branch from e4e83f3 to afc7847 Compare December 28, 2023 18:41

illicitonion approved these changes Dec 28, 2023

View reviewed changes

illicitonion enabled auto-merge (squash) December 28, 2023 18:45

illicitonion mentioned this pull request Dec 28, 2023

Release 0.36.1 #2376

Merged

dtolnay and others added 5 commits December 28, 2023 10:48

Exercise crates_vendor in bzlmod hello_world example

50291b3

Expose crates_vendor deps to bzlmod

3552932

Run 'bazel run //third-party:vendor' in examples/bzlmod/hello_world

c316e14

Run a bzlmod crates_vendor in presubmit

06d8e15

Try just deleting the lockfile

6745da8

auto-merge was automatically disabled December 28, 2023 18:48
Head branch was pushed to by a user without write access

dtolnay force-pushed the cratesvendordeps branch from afc7847 to 6745da8 Compare December 28, 2023 18:48

illicitonion enabled auto-merge (squash) December 28, 2023 18:48

illicitonion merged commit 5001ff6 into bazelbuild:main Dec 28, 2023
3 checks passed

dtolnay deleted the cratesvendordeps branch December 28, 2023 18:54

UebelAndre mentioned this pull request Dec 28, 2023

Releases not being tagged correctly #2377

Open

illicitonion mentioned this pull request Jan 2, 2024

Buildifier unknown_repo on linux arm64 with bzlmod #2384

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose crates_vendor deps to bzlmod #2372

Expose crates_vendor deps to bzlmod #2372

dtolnay commented Dec 28, 2023

illicitonion left a comment

illicitonion commented Dec 28, 2023

dtolnay commented Dec 28, 2023

dtolnay Dec 28, 2023

illicitonion Dec 28, 2023

dtolnay Dec 28, 2023

illicitonion Dec 28, 2023

illicitonion Dec 28, 2023

illicitonion Dec 28, 2023

dtolnay Dec 28, 2023

illicitonion Dec 28, 2023

illicitonion Dec 28, 2023

illicitonion Dec 28, 2023

matts1 Dec 28, 2023

dtolnay commented Dec 28, 2023

illicitonion left a comment

illicitonion commented Dec 28, 2023

Expose crates_vendor deps to bzlmod #2372

Expose crates_vendor deps to bzlmod #2372

Conversation

dtolnay commented Dec 28, 2023

illicitonion left a comment

Choose a reason for hiding this comment

illicitonion commented Dec 28, 2023

dtolnay commented Dec 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtolnay commented Dec 28, 2023

illicitonion left a comment

Choose a reason for hiding this comment

illicitonion commented Dec 28, 2023