Precise capturing

[traviscross]

To fully stabilize, in Rust 2024, the Lifetime Capture Rules 2024 that we accepted in RFC 3498, we need to stabilize some means of precise capturing. This RFC provides that means.

This RFC adds use<..> syntax for specifying which generic parameters should be captured in an opaque RPIT-like impl Trait type, e.g. impl use<'t, T> Trait. This solves the problem of overcapturing and will allow the Lifetime Capture Rules 2024 to be fully stabilized for RPIT in Rust 2024.

One way to think about use<..> is that, in Rust use brings things into scope, and here we are bringing certain generic parameters into scope for the hidden type.

For some history about the progress toward this feature predating this RFC, see this comment.

Rendered

[juntyr]

It might be nice to allow use<..> from the start as an explicit way to capture all in-scope generic parameters. Using this short-hand together with explicit explicit captures or excluded captures should still remain future work.

Especially in code where precise captures would be frequently used, it might be preferable to always be explicit, even in cases where the implicit capture-all is what we want. Similar to how the syntax is used with struct initialisation, using use<..> could mean that we know that we will always want to capture all generics, even if more are added, whereas manually enumerating each one might mean that we want to explicitly have to choose with every newly added generic.

Furthermore, since the stabilisation strategy suggests that at first only capturing all parameters may be supported, supporting use<..> from the start would make this shorter to type.

[Jules-Bertholet]

An alternative meaning for use<..> is "capture all APITs".

[petrochenkov]

cannot be captured with use<..>

Are they captured implicitly now?
If they are, then what is the opt out of that?

Are unnamed lifetimes implicitly captured by RPIT?
What is the opt out in that case?
(If unnamed early bound lifetimes are possible at all.)

[compiler-errors]

Nope, we don't capture APITs or elided lifetimes with use<'a, T>. The opt-in here is to promote your APITs to real type generics and give your unnamed lifetimes names.

The only caveat here is that you can name the elided lifetime in the output in the same cases you are allowed to normally in RPITs -- i.e. fn hello(x: &u8) -> impl use<'_> Sized { x } works fine.

[petrochenkov]

Nope, we don't capture APITs or elided lifetimes with use<'a, T>.

My first question was whether they are captured if use is not written.
The text says that all generic parameters are captured (on 2024 edition at least), but it's not clear whether APITs and elided lifetimes are included into this "all".

[compiler-errors]

All type and const generic parameters (including APITs) are captured for all RPITs regardless if they're in edition 2021 or 2024.

The only change between 2021 and 2024 involves capturing all lifetimes parameters in scope (incl. elided lifetimes, since those become early- or late-bound lifetime params) -- previously we only captured lifetimes mentioned in the bounds of RPITs. The opt-out here is to use use<'a, T> with the set of lifetimes that previously showed up in your opaque's bounds.

The only corner case is when you have an APIT and a lifetime you want to not capture -- you'll need to turn your APITs into real type parameters to make use of the use<'a, T> syntax.

Regarding opt-out for capturing type or const generic parameters currently, it's not possible to do currently; we will likely support that eventually, but if you see https://github.com/rust-lang/rfcs/blob/TC/precise-capturing/text/3617-precise-capturing.md#stabilization-strategy, we're probably not going to stabilize that initially because it requires exercising new paths of the type system (namely, bivariant unconstrained type parameters), and it's not necessary to mitigate the fallout of the new edition 2024 lifetime capture rules.

[nikomatsakis]

@rfcbot fcp merge

We've tried hard to avoid an explicit syntax like this but I think it's clear by now that it will be useful and it unblocks important Edition work. We had a reasonably thorough deep dive into the syntactic options and I believe the RFC lays out the options pretty well and the tradeoffs around them. Personally while I have some minor qualms about overloading use, I think it's the best option overall.

Note that in this fcp I am explicitly wearing my @rust-lang/lang hat -- I think the @rust-lang/types team should (before stabilization) vet the overall semantics and our implementation thereof but that's not really a question to be answered in the RFC.

[rfcbot]

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[petrochenkov]

I suspect that it's going to get annoying pretty quick (compared to impl<..> Trait), if precise captures become something that is specified often (e.g. by convention to minimize captures).

[Jules-Bertholet]

I don't feel strongly about this syntax either way, but:

This would hardly be the only place where syntax for introducing generic parameters is similar to syntax for using them. fn foo<T>() vs foo::<T>() for example.

Also, there is a symmetry of a sort between the two kinds of impl generics. In a trait implementation, they introduce generic parameters available for use by the type and trait; in RPIT, they would introduce the generics available for use by the hidden type.

[Jules-Bertholet]

Why can't this "just work"? Shouldn't the compiler be able to figure out that the closure is not using T? There should be no semver hazard, because to return the closure you need RPIT or TAIT , at which point you as API designer have the opportunity to specify the captures you commit to in public API.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[kennytm]

Since this use<..> thing is part of the type signature, could you clarify if the capture list is changed, when it will be considered breaking change or not?

From the caller's point of view, if a function's RPIT (in covariant position only) has a capture removed, the hidden type's potential shortest lifetime is lengthened, which is compatible with existing caller code. So I think semver should allow removing captures from covariant-position RPIT in a minor update.

- fn callee1<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> &'a u8 { &0 }
+ fn callee1<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> &'static u8 { &0 }

- fn callee2<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> impl use<'a> Sized { &0 }
+ fn callee2<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> impl use<> Sized { &0 }

Meanwhile for TAIT, ATPIT and RPITIT changing the capture list in either direction should be considered a major breaking change.

(For APIT the capture list is irrelevant.)

[jackh726]

So, just procedurally...

While I do think solving the underlying ergonomic and semantic issues here are quite important, I do have to note that this syntax and RFC seem to have moved quite fast. Even for me - who, while doesn't attend lang meetings, does try to keep up to date of active lang things - I could have almost missed this. To put some dates to this on typical milestones in our current processes:

• The tracking issue was opened only 3 weeks ago
• The initial implementation PR a day after that
• The design meeting happened today
• This RFC was opened today
• And both PFCP and FCP have started today on this RFC

I know we're on a time crunch because of the edition, but I worry about this all moving just a bit too fast. Of course, accepting this RFC doesn't necessarily mean that actually stabilizing this feature will also happen quickly, but I worry that the pace so far is a harbinger for that. I hate to sit here and be the one that say "wait, we're moving too fast", especially because not moving fast here puts edition work in jeopardy, but I'd rather this concern be voiced than ignored.

I do want to be clear that my above comments have little reflection on my thoughts on the contents of this RFC or semantics of this feature. My concerns here apply to even the best of features (to put it into perspective: let-else, which imo is a very clear win all around took about a month from RFC open to RFC merge, with a fair amount of prior discussion; and this is what I consider a "fast" RFC process).

[tmandry]

As I explained in the lang team meeting, I prefer use<..> impl Trait over impl use<..> Trait. My points did not produce consensus in favor of changing the syntax: of the lang team members present, two preferred use-first, two preferred use-after, and one was agnostic. The RFC author preferred use-after, and I agreed not to block the RFC on the question.

The reasons for preferring this syntax are:

1. Subjective (accessibility): The mnemonic users have for this feature is impl Trait. By putting more things between the impl and the Trait we make it less recognizable.
2. Subjective (linguistic): This comes from english writing. It's simpler to say "using x, y, and z, I implement Foo" than "I implement – using x, y, and z – Foo". You would always prefer the first over the second.
3. Technical: The comparison to impl for<'a> Foo<'a> + Bar and impl Foo + for<'a> Bar<'a> is informative. for<'a> applies to a particular bound in the set of bounds, while use<> applies to the entire impl Trait opaque type. Putting it out in front distinguishes it from for<> in this way.¹
4. Future consistency: The code sample for closures puts use in front of the || (for good reason, in my opinion). We should strive for consistency in the way such syntax is used. It would be somewhat unfortunate to have both of these in the language:
   • use<> || expr
   • impl use<> Trait

impl Trait for () {
    type Ty = impl Fn();
    fn define<T>(_: T) -> Self::Ty {
        use<> || ()
    //  ^^^^^^^^^^^
    //  ^ Captures no generic parameters.
    }
}

The arguments in favor of use-after were later added to the RFC.

Most of the discussion against revolved around the fact that it would require another migration to macro fragment specifiers (specifically ty) in 2024. But I think it's important to note that this would not prevent usage of this feature in un-migrated macros. Users could always put parentheses around these types when using macros which are not migrated to the new fragment specifiers.

// Before 2024:
my_unmigrated_macro! {
    fn foo<'t, T>(_: &'t (), x: T) -> impl Sized { x }
}

// After 2024 (note the parentheses):
my_unmigrated_macro! {
    fn foo<'t, T>(_: &'t (), x: T) -> (use<T> impl Sized) { x }
}

// After 2024, macro has been migrated to new matchers:
my_migrated_macro! {
    fn foo<'t, T>(_: &'t (), x: T) -> use<T> impl Sized { x }
}

After all the discussion I still find myself preferring use-before and wish we had more time to experiment with syntax. But this is the RFC, not stabilization, and I do not wish to block the RFC or the feature itself on this particular question.

Footnotes

1. In my view, this goes against the idea, now expressed in the RFC, that use<> impl Trait "looks like a binder" ("binder" is jargon for for<'a>). ↩

[Jules-Bertholet]

use<..> impl Trait is also more greppable than impl use<..> Trait. (That being said, I don't know that grepping for impl Trait is a task one would need to perform often.)

[traviscross]

Thanks @tmandry for writing that up. I've now greatly extended that section of the document to incorporate these¹ and other points in favor of use<..> impl Trait².

See in particular the discussion of the fundamental tension here. In short, the RFC lays out two intuitions for use<..>:

• Intuition 1: use<..> applies generic arguments to the opaque type.
• Intuition 2: use<..> brings generic parameters into scope for the hidden type.

These intuitions are both true, but they might suggest two different syntaxes, and this may be related to why the choice here is challenging.

Footnotes

1. I did not include the bits you mentioned about invoking macros across editions as the document already did not put much weight on and did not go into much detail on the macro fragment specifier migration. I did add further language to suggest that this migration cost is not major. ↩

2. I've also removed the bit you mentioned in your footnote about it perhaps looking like a binder, as this indeed didn't seem a strong point to me either. ↩

[aliemjay]

This RFC does not specify what an RPITIT/ATPIT in a trait impl is allowed to capture in order for it to be compatible with the trait definition.

Here is an example of multiple trait implementations, the validity of which are not clear from the RFC.

// RPITIT only allowed to capture Y.
trait Trait<X, Y> {
    fn test() -> impl use<Y> Sized;
}


// Y = (A, B).
// Are we allowed to capture A?
// Is it considered a refinement to not capture B?
impl<A, B> Trait<(), (A, B)> for i8 {
    fn test() -> impl use<A> Sized {}
}

// Y = <A as Iterator>::Item.
// Are we allowed to capture A now that it appears in a projection type?
// I don't think so since projections do not constrain their parameters.
impl<A: Iterator> Trait<A, A::Item> for u8 {
    fn test() -> impl use<A> Sized {}
}

// Y = &'a str.
// The case for lifetimes and consts should be made clear.
impl<'a> Trait<(), &'a str> for i16 {
    fn test() -> impl use<'a> Sized {}
}

// Y = &'a str.
// RPITIT can't capture `'b` even though `'a == 'b`.
// Lifetime comparison is syntactic.
impl<'a: 'b, 'b: 'a> Trait<&'b str, &'a str> for u16 {
    fn test() -> impl use<'b> Sized {}
}

[kennytm]

// Y = <A as Iterator>::Item.
// Are we allowed to capture A now that it appears in a projection type?
// I don't think so since projections do not constrain their parameters.
impl<A: Iterator> Trait<A, A::Item> for u8 {
    fn test() -> impl use<A> Sized {}
}

Unlike the Captures<Y> trick you can't write impl use<A::Item> Sized 🤔 And what is meant by capturing A::Item for the impl anyway

It certainly cannot return impl use<A> Sized, otherwise <u8 as Trait<Chars<'b>, char>>::test() would be unsound by allowing 'b to escape.

OTOH returning impl use<> Sized would disallow use of 'a in <u8 as Trait<&'a [u16], &'a u16>>::test().

Basically you have to do refinement unless the capture list is relaxed to accept arbitrary GenericArgs.

[scottmcm]

use<..> impl Trait is also more greppable than impl use<..> Trait.

Hmm, do we have impl for<'a> Trait<'a> already?

[fmease]

We do

[Yokinman]

Would it be fair to say that use<...> Type might be a future possibility?

For instance, say you have a type called Flux that changes over time with a method that returns it at a specific Moment. The Moment is a transformation of the original value and requires no references, but to prevent confusion you may want to make a Flux immutable if one of its Moments is live. You can use PhantomData, but this could be a kind of syntax sugar for that:

use std::time::Duration;

struct Flux {
    value: f64,
    rate: f64,
}

struct Moment {
    value: f64,
}

impl Flux {
    fn at<'a>(&'a self, secs: Duration) -> use<'a> Moment {
        Moment {
            value: self.value + self.rate * secs.as_secs_f64(),
        }
    }
}

fn main() {
    let mut flux = Flux {
        value: 0.0,
        rate: 1.0,
    };
    let moment = flux.at(Duration::from_secs(1));
    
    flux.value = 2.0; // <- error: cannot assign to `flux.value` because it is borrowed
    
    println!("{}", moment.value);
}

I imagine use<...> T would be considered a different type from T - maybe one that automatically dereferences to T. Either way, the point is that this could be a future possibility; one that might also favor use before impl.

Edit: Maybe it could just capture parameters like how associated types do, but idk if that works

[nikomatsakis]

@Yokinman that's an interesting idea. It's definitely true that use<T> impl syntax gives us more room to generalize it to further contexts.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[traviscross]

Thanks @aliemjay for those great questions. We've now added examples and discussion to clarify each of those.

Note that the hardest subset of those questions relate to how things would work if we were able to capture less than all of the generic input parameters to the trait (including Self) in the trait definition. This is not likely to be possible in initial rounds of stabilization, and if it ever does become possible, these kind of questions are likely to receive extensive coverage in the stabilization report.

[Evian-Zhang]

Great work! I think this feature is a rather advanced feature and maybe this syntax appearing on simple APIs may discourage beginners when using fundamental functionalities in a crate. However, I have designed the following API pattern multiple times:

fn parse(path: impl Path) -> impl Iterator<Item = Foo> {}

I wonder if I should add a use<> in the return type position to indicate it does not capture anything?

I think this pattern is very common and basic in most crates. Should there be some special treatment in rustdoc for use<> (i.e. capture nothing) to make it more beginner-friendly?

[nikomatsakis]

@Evian-Zhang That's a good question (though not I think one that needs to block progress on the RFC). That said, I think even better would be if we could avoid using impl Trait for those kind of coercions. The right place for them to take place is in the caller side, the callee would prefer to just get a value of known type and hence be (more) monomorphic. We had at some point discussed a syntax option for this (e.g., writing ~String instead of impl Into<String>). That said, I hadn't until now considered the implications for borrow checking when you have an -> impl Trait return value.

(We've also discussed (and even done some exploration of) having the compiler recognize the pattern of a "mostly monomorphic" function that just does transforms in the beginning and avoid code duplication: that's worth doing as a first step, I believe, but it wouldn't have the borrow checker benefits.)

[Evian-Zhang]

@nikomatsakis Thank you for your response!

I agree that the coercions should be in the caller side, since I have found many places where the following code pattern appears in the Rust std source code:

fn foo(path: impl AsRef<Path>) {
    fn inner_foo(path: &Path) { ... }
    inner_foo(path.as_ref())
}

It is more graceful if this pattern can be automatically done by the compiler.

I am not familiar with the RFC discussion guidelines, and I think maybe I should put the rustdoc's use<> problem in the tracking issues after this RFC stabilized? Apologies for putting it here.

[traviscross]

@Evian-Zhang: It'd be better to open a thread on Zulip (or perhaps on IRLO) to discuss that further.

[nikomatsakis]

@Evian-Zhang (...and no apologies required)