Bzlmod-aware runfiles library #2566

dzbarsky · 2024-03-20T21:01:09Z

This implements the repo_mapping capabilities and provides a new runfiles API. The new API can be accessed explicitly as runfiles::Runfiles::rlocation_from or with the runfiles::rlocation! macro, which adds compile-time support for correctly embedding the external repo. This is a purely new API, existing usage continues to work, although we mark it deprecated because it's not fully correct. We can remove it at some point in the future.

This PR also transitions in-repo examples/tests to using it, in case anyone copies them.

matts1 · 2024-03-20T22:17:27Z

tools/runfiles/runfiles.rs

@@ -19,13 +19,13 @@
 //!     use runfiles::Runfiles;
 //!     ```
 //!
-//! 3.  Create a Runfiles object and use rlocation to look up runfile paths:
+//! 3.  Create a Runfiles object and use Rlocation! to look up runfile paths:


Should it be title-case? I was under the impression that naming conventions dictated that since it's a function, it should be rlocation

sure, I wasn't sure what the convention was for macros but I think you're right

matts1 · 2024-03-20T22:30:03Z

tools/runfiles/runfiles.rs

 }

 impl Runfiles {
    /// Creates a manifest based Runfiles object when
    /// RUNFILES_MANIFEST_ONLY environment variable is present,
    /// or a directory based Runfiles object otherwise.
    pub fn create() -> io::Result<Self> {
-        if is_manifest_only() {
+        let mode = if is_manifest_only() {
            Self::create_manifest_based()


IIRC @fmeum said that the canonical implementation is at rules_python and that other implementations should seek to follow that. In all other implementations, we create the manifest based one if RUNFILES_MANIFEST_FILE exists, otherwise we create it based on RUNFILES_DIR.

I believe we should instead have:

let mode = if let Ok(manifest_file) = std::env::var(MANIFEST_FILE_ENV_VAR) { create_manifest_based(Path::from(manifest_file))? } else { Mode::DirectoryBased(find_runfiles_dir()?) }

I'm happy to swap to that. It's a bit riskier since it's a behavior change but it's more correct you point out

RUNFILES_MANIFEST_ONLY is a trap that breaks cross-platform builds and should not be used. Instead, I would say the following flow is the best you can do at this point in time:

If at least one of RUNFILES_MANIFEST_FILE or RUNFILES_DIR is set:

If RUNFILES_MANIFEST_FILE is set and exists, use it.

If RUNFILES_DIR is set and exists, use it.

Error.

If the runfiles manifest exists in one of the well-known locations, use it.

If the runfiles directory exists in one of the well-known locations, use it.

Error.

matts1 · 2024-03-20T22:44:59Z

tools/runfiles/runfiles.rs

@@ -44,36 +44,49 @@ const MANIFEST_FILE_ENV_VAR: &str = "RUNFILES_MANIFEST_FILE";
 const MANIFEST_ONLY_ENV_VAR: &str = "RUNFILES_MANIFEST_ONLY";
 const TEST_SRCDIR_ENV_VAR: &str = "TEST_SRCDIR";

+#[macro_export]
+macro_rules! Rlocation {


I'm not going to say we should do it one way or the other, but we should definitely at least consider the folllowing API as an alternative:

let r = runfiles::create!() // Expands to runfiles::create(env!("REPOSITORY_NAME")) r.rlocation(path)

Pros:

More clean API (less macros)

More performant (only need to calculate the repo mapping once)

Simpler to migrate to (only migrate the runfiles constructor, not the rlocation calls)

Cons:

If you're passing the runfiles constructor across repo boundaries, it might do the wrong repo mapping. I can't see this ever happening though - I've never seen a public API that takes in runfiles, because they can just create the runfiles object themselves.

Yep I asked @fmeum the same and he mentioned passing across repo boundaries is a nice to have. I'm fine either way but let's figure it out in this thread before I refactor more :)

Re: more performant - I think it's the same? The mapping is only calculated at startup? Or if you're referring to the map lookups + string ops - you're about to do File IO, I don't think it matters :)

The migration path is also a good point. I think as-written the code doesn't require any client changes, but they can move to the new API to become more correct. I think with your proposal everyone using runfiles needs a minor code change?

Ah, good point about the migration path, I hadn't notice that we left the old method, so I thought this whole PR was backwards incompatible. If that's the case, I think we definitely just do it the way you've already done it.

rust/private/rustc.bzl

dzbarsky · 2024-03-21T23:58:05Z

rust/private/rustfmt.bzl

@@ -13,7 +13,7 @@ def _get_rustfmt_ready_crate_info(target):
    """

    # Ignore external targets
-    if target.label.workspace_root.startswith("external"):
+    if target.label.workspace_name:


driveby cleanup to not depend on external

dzbarsky · 2024-03-29T19:26:06Z

@UebelAndre this is passing tests now. I'm planning to add an external-repo integration test as well to double check, but assuming that is OK, can we merge this? None of us have a better idea how to make this work correctly and if I understand correctly the state of the world, enabling bzlmod in a repo with rust code will make rust-analyzer not work correctly, even if the rust rules are not using bzlmod. Would be good to get a fix in

illicitonion

LGTM, thanks! It'd be great to have that external repo test you mentioned.

tools/runfiles/runfiles.rs

illicitonion · 2024-04-02T16:29:28Z

tools/runfiles/runfiles.rs

+        .lines()
+        .map(|line| {
+            let parts: Vec<String> = line.splitn(3, ',').map(String::from).collect();
+            ((parts[0].clone(), parts[1].clone()), parts[2].clone())


It would be good to error rather than panic if this file is malformed and doesn't contain 3 elements

done, I also fixed the duplicative string copying while I was here

dzbarsky · 2024-04-02T18:38:24Z

LGTM, thanks! It'd be great to have that external repo test you mentioned.

I added it as a separate commit. It's a bit complicated setup but it's what we need to really verify we are using the correct repo_mapping of transitive repos instead of just the main one

UebelAndre · 2024-04-03T01:59:53Z

Does this break the existing API? Can you update the PR description to reflect the changes some more?

illicitonion

Works for me! Will leave for @UebelAndre to review/merge when happy

dzbarsky · 2024-04-03T15:04:02Z

Does this break the existing API? Can you update the PR description to reflect the changes some more?

No, this is a new API to make this non-breaking, and the existing API is marked deprecated. The PR summary already mentioned that, but I expanded it a bit, hopefully it's clearer now!

dzbarsky · 2024-04-09T02:15:54Z

@UebelAndre does the PR description make sense now? Anything else you'd like to see before this lands?

dzbarsky · 2024-04-21T14:24:43Z

@UebelAndre @illicitonion What can I do to get this unstuck and landed?

This PR is blocking other issues, such as #2615.

There have been other attempts to fix this for over a year now, such as: first one, second one.

I think I have spent more time trying to get this PR landed than it took to write the code, which is a little frustrating as you can imagine.

scentini · 2024-04-22T08:19:23Z

This PR LGTM, and as it was already approved by @illicitonion a while back, I'm going to to ahead and merge it. @UebelAndre please feel free to do post-merge review, and we'll follow up.

Dismissing review to enable merge when ready. The PR author has addressed the reviewer's comments.

UebelAndre · 2024-04-29T03:40:36Z

Sorry, this one fell through the cracks. I didn't mean to delay the change!

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [rules_rust](https://togithub.com/bazelbuild/rules_rust) | http_archive | minor | `0.42.1` -> `0.43.0` | --- ### Release Notes <details> <summary>bazelbuild/rules_rust (rules_rust)</summary> ### [`v0.43.0`](https://togithub.com/bazelbuild/rules_rust/releases/tag/0.43.0) [Compare Source](https://togithub.com/bazelbuild/rules_rust/compare/0.42.1...0.43.0) ### 0.43.0 ```python load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") http_archive( name = "rules_rust", integrity = "sha256-SnzocNvQp1YK1TW/aUVhR6RSROo1l2RilE1V20WFnK0=", urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.43.0/rules_rust-v0.43.0.tar.gz"], ) ``` Additional documentation can be found at: https://bazelbuild.github.io/rules_rust/#setup #### What's Changed - Bzlmod-aware runfiles library by [@dzbarsky](https://togithub.com/dzbarsky) in [bazelbuild/rules_rust#2566 - Add clippy_flag flag to allow flags to be added in succession. by [@granaghan](https://togithub.com/granaghan) in [bazelbuild/rules_rust#2625 - Add aspect_rules_js to MODULE.bazel by [@dzbarsky](https://togithub.com/dzbarsky) in [bazelbuild/rules_rust#2618 - Register a default rust toolchain. by [@matts1](https://togithub.com/matts1) in [bazelbuild/rules_rust#2624 - Nit: Fix link to example in rust_bindgen.md by [@hauserx](https://togithub.com/hauserx) in [bazelbuild/rules_rust#2628 - Add context to error messages by [@illicitonion](https://togithub.com/illicitonion) in [bazelbuild/rules_rust#2408 - Update `cargo_bootstrap_repository` interface to match dependency macros by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2630 - Added debug logging for spliced manifests to crate_universe by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2629 - also rewrite -isystem in addition to -sysroot by [@adrianimboden](https://togithub.com/adrianimboden) in [bazelbuild/rules_rust#2631 - Support loading http credentials from netrc by [@golovasteek](https://togithub.com/golovasteek) in [bazelbuild/rules_rust#2623 - Updated Buildifier version for crate_universe by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2634 - Follow-up documentation/fixes to lockfile API by [@illicitonion](https://togithub.com/illicitonion) in [bazelbuild/rules_rust#2637 - Add docs and examples of complicated build scripts by [@illicitonion](https://togithub.com/illicitonion) in [bazelbuild/rules_rust#2635 - Added Rust 1.78.0 by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2639 - Remove `incompatible_test_attr_crate_and_srcs_mutually_exclusive` by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2641 - Minor cleanup for crate_universe by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2644 - Use `cargo tree` to determine feature dependent optional deps by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2636 - Release 0.43.0 by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2642 - Update cross to fix crate_universe builds in releases by [@UebelAndre](https://togithub.com/UebelAndre) in [bazelbuild/rules_rust#2645 #### New Contributors - [@hauserx](https://togithub.com/hauserx) made their first contribution in [bazelbuild/rules_rust#2628 - [@adrianimboden](https://togithub.com/adrianimboden) made their first contribution in [bazelbuild/rules_rust#2631 - [@golovasteek](https://togithub.com/golovasteek) made their first contribution in [bazelbuild/rules_rust#2623 **Full Changelog**: bazelbuild/rules_rust@0.42.1...0.43.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/bazel-contrib/toolchains_llvm).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

dzbarsky mentioned this pull request Mar 20, 2024

Make runfiles library bzlmod compatible #2440

Closed

dzbarsky force-pushed the main branch 3 times, most recently from 82eab55 to 6236ef1 Compare March 20, 2024 21:24

matts1 reviewed Mar 20, 2024

View reviewed changes

dzbarsky force-pushed the main branch 3 times, most recently from cb7a2bf to c719e80 Compare March 21, 2024 15:27

UebelAndre previously requested changes Mar 21, 2024

View reviewed changes

rust/private/rustc.bzl Show resolved Hide resolved

dzbarsky force-pushed the main branch from c719e80 to 30159c1 Compare March 21, 2024 17:19

dzbarsky commented Mar 21, 2024

View reviewed changes

dzbarsky force-pushed the main branch from 4ccfbde to ebe4f80 Compare March 22, 2024 00:24

dzbarsky mentioned this pull request Mar 22, 2024

Support repo mapping for bazel runfiles. #1847

Open

fmeum approved these changes Mar 29, 2024

View reviewed changes

illicitonion reviewed Apr 2, 2024

View reviewed changes

dzbarsky added 4 commits April 2, 2024 12:44

Bzlmod-aware runfiles library

4464f20

Remove runfiles.current_repository()

1d7ede7

Cleanup more runfiles references

a275ba6

Review feedback

c437c0f

dzbarsky force-pushed the main branch from b9f617d to 486ad4e Compare April 2, 2024 18:37

dzbarsky force-pushed the main branch 2 times, most recently from f962dcb to 5ef713a Compare April 2, 2024 18:41

Add test

e5a57b7

dzbarsky force-pushed the main branch from 5ef713a to e5a57b7 Compare April 2, 2024 18:48

Fix usage in examples and tests

a6fb2ac

dzbarsky force-pushed the main branch from 79ebf3d to a6fb2ac Compare April 2, 2024 19:31

buildifier

5e0a723

dzbarsky requested a review from UebelAndre April 2, 2024 19:59

matts1 approved these changes Apr 3, 2024

View reviewed changes

illicitonion approved these changes Apr 3, 2024

View reviewed changes

dzbarsky mentioned this pull request Apr 13, 2024

gen_rust_project doesn't work with bzlmod #2615

Closed

Merge branch 'main' into main

fb25a90

scentini enabled auto-merge April 22, 2024 08:19

scentini disabled auto-merge April 22, 2024 08:21

scentini enabled auto-merge April 22, 2024 08:21

scentini added this pull request to the merge queue Apr 22, 2024

Merged via the queue into bazelbuild:main with commit 8d074a0 Apr 22, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bzlmod-aware runfiles library #2566

Bzlmod-aware runfiles library #2566

dzbarsky commented Mar 20, 2024 •

edited

matts1 Mar 20, 2024

dzbarsky Mar 21, 2024

matts1 Mar 20, 2024

dzbarsky Mar 21, 2024

fmeum Mar 21, 2024

matts1 Mar 20, 2024 •

edited

dzbarsky Mar 21, 2024 •

edited

matts1 Mar 22, 2024

dzbarsky Mar 21, 2024

dzbarsky commented Mar 29, 2024

illicitonion left a comment

illicitonion Apr 2, 2024

dzbarsky Apr 2, 2024

dzbarsky commented Apr 2, 2024

UebelAndre commented Apr 3, 2024

illicitonion left a comment

dzbarsky commented Apr 3, 2024

dzbarsky commented Apr 9, 2024

dzbarsky commented Apr 21, 2024

scentini commented Apr 22, 2024

UebelAndre commented Apr 29, 2024 •

edited

Bzlmod-aware runfiles library #2566

Bzlmod-aware runfiles library #2566

Conversation

dzbarsky commented Mar 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matts1 Mar 20, 2024 • edited

Choose a reason for hiding this comment

dzbarsky Mar 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dzbarsky commented Mar 29, 2024

illicitonion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dzbarsky commented Apr 2, 2024

UebelAndre commented Apr 3, 2024

illicitonion left a comment

Choose a reason for hiding this comment

dzbarsky commented Apr 3, 2024

dzbarsky commented Apr 9, 2024

dzbarsky commented Apr 21, 2024

scentini commented Apr 22, 2024

UebelAndre commented Apr 29, 2024 • edited

dzbarsky commented Mar 20, 2024 •

edited

matts1 Mar 20, 2024 •

edited

dzbarsky Mar 21, 2024 •

edited

UebelAndre commented Apr 29, 2024 •

edited