Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

impl From for &mut UninitSlice to &mut [MaybeUninit<u8>] #548

Merged
merged 1 commit into from May 10, 2022

Conversation

erickt
Copy link
Contributor

@erickt erickt commented May 5, 2022

This adds an unsafe method to convert a &mut UninitSlice into a &mut [MaybeUninit<u8>]. This method is unsafe because some of the bytes in the slice may be initialized, and the caller should not overwrite them with uninitialized bytes.

This came about when auditing tokio-util's udp frame, where they want to pass the unitialized portion of a BytesMut to ReadBuf::uninit. They need to do this unsafe pointer casting in a few places, which complicates audits. This method lets us document the safety invariants the caller needs to maintain when doing this conversion.

@Darksonn
Copy link
Contributor

Darksonn commented May 5, 2022

Sorry, but this is not sound. You can reborrow a &mut UninitSlice to get a "subslice" to the same range. The subslice can then be converted to a &mut [MaybeUninit<u8>], but the original slice will still be usable once you stop using the subslice.

@Darksonn
Copy link
Contributor

Darksonn commented May 5, 2022

More generally, you need to be able to pass a &mut UninitSlice to untrusted code and trust that it wont deinitialize it. Even without reborrowing, this would break that.

@erickt
Copy link
Contributor Author

erickt commented May 5, 2022

Sorry, but this is not sound. You can reborrow a &mut UninitSlice to get a "subslice" to the same range. The subslice can then be converted to a &mut [MaybeUninit], but the original slice will still be usable once you stop using the subslice.

More generally, you need to be able to pass a &mut UninitSlice to untrusted code and trust that it wont deinitialize it. Even without reborrowing, this would break that.

Ah good point, I didn't consider reborrowing, yeah that would let us deinitialize UninitSlice. However, could you go into more detail about how this is unsound? I don't believe reborrowing would let us get overlapping &mut sections.

Would it be worthwhile to instead add a separate BytesMut::chunk_mut_slice(&mut self) -> &mut [MaybeUninit<u8>] (or something), since a lot of APIs seem to be using this type for working with uninitialized slices? Or would you rather hold off on that now, wait for something like https://doc.rust-lang.org/nightly/std/io/struct.ReadBuf.html to stabilize before considering new APIs?

@Darksonn
Copy link
Contributor

Darksonn commented May 5, 2022

I'm fine with adding an unsafe method for converting to &mut [MaybeUninit<u8>] if it's something we do often.

As for reborrowing, it really does give overlapping &mut sections. The reason it isn't UB is that the compiler will prevent you from using the original slice until after the last use of the reborrow. Since only one of them is usable at the time, the exclusivity is still upheld.

More fundamentally, the reason its unsound to do this is that the code that created the &mut UninitSlice knows that part of the slice is already initialized, and for safety reasons, it needs that part of the slice to remain initialized after giving out an &mut UninitSlice to the memory. Any way of deinitializing it given the &mut UninitSlice (or any reference or pointer derived from it) would cause issues.

Note that Tokio already has a stable variant of ReadBuf.

@erickt
Copy link
Contributor Author

erickt commented May 5, 2022

I'm fine with adding an unsafe method for converting to &mut [MaybeUninit] if it's something we do often.
...
More fundamentally, the reason its unsound to do this is that the code that created the &mut UninitSlice knows that part of the slice is already initialized, and for safety reasons, it needs that part of the slice to remain initialized after giving out an &mut UninitSlice to the memory. Any way of deinitializing it given the &mut UninitSlice (or any reference or pointer derived from it) would cause issues.

Ah, that's great to know. It'd be great to document this, since it'd be important to check for this invariant during reviews. Would you be okay with the name UninitSlice::as_uninit_slice_mut? It looks like the standard library may stabilize on that naming scheme: https://doc.rust-lang.org/std/primitive.pointer.html#method.as_uninit_slice_mut

Note that Tokio already has a stable variant of ReadBuf.

Oh sure, I just figured that when we stabilize std::io::ReadBuf, we might want to directly integrate support into the bytes crate. But we probably don't want to make the bytes crate depend on tokio for this API. We could lift tokio's ReadBuf into it's own crate, but I imagine that'd take some time to detangle it from everything else in tokio, so it might not be worth it.

@erickt erickt force-pushed the uninit branch 2 times, most recently from d66a2b0 to 39aa8fe Compare May 5, 2022 21:35
Copy link
Contributor

@Darksonn Darksonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that's great to know. It'd be great to document this, since it'd be important to check for this invariant during reviews.

Would you be able to add that to this PR?

@@ -124,6 +124,29 @@ impl UninitSlice {
self.0.as_mut_ptr() as *mut _
}

/// Return a `&mut [MaybeUninit<u8>] to this slice's buffer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Return a `&mut [MaybeUninit<u8>] to this slice's buffer.
/// Return a `&mut [MaybeUninit<u8>]` to this slice's buffer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks.

src/buf/uninit_slice.rs Show resolved Hide resolved
This adds an unsafe method to convert a `&mut UninitSlice` into a
`&mut [MaybeUninit<u8>]`. This method is unsafe because some of the
bytes in the slice may be initialized, and the caller should not
overwrite them with uninitialized bytes.

This came about when auditing [tokio-util's udp frame], where they want
to pass the unitialized portion of a `BytesMut` to [ReadBuf::uninit].
They need to do this unsafe pointer casting in a few places, which
complicates audits. This method lets us document the safety invariants
the caller needs to maintain when doing this conversion.

[tokio-util's udp frame]: https://github.com/tokio-rs/tokio/blob/master/tokio-util/src/udp/frame.rs#L87
[ReadBuf::uninit]: https://docs.rs/tokio/latest/tokio/io/struct.ReadBuf.html#method.uninit
@erickt
Copy link
Contributor Author

erickt commented May 9, 2022

Thanks for the review! I pushed up a new version that hopefully fixes your concerns.

Copy link
Contributor

@Darksonn Darksonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@Darksonn Darksonn merged commit b8d27c0 into tokio-rs:master May 10, 2022
/// ```
#[inline]
pub unsafe fn as_uninit_slice_mut<'a>(&'a mut self) -> &'a mut [MaybeUninit<u8>] {
&mut *(self as *mut _ as *mut [MaybeUninit<u8>])

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a subtle reason this is not just &mut self.0?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I didn't realize that this was possible when approving the PR.

lelongg pushed a commit to lelongg/bytes that referenced this pull request Jan 9, 2023
This adds an unsafe method to convert a `&mut UninitSlice` into a
`&mut [MaybeUninit<u8>]`. This method is unsafe because some of the
bytes in the slice may be initialized, and the caller should not
overwrite them with uninitialized bytes.

This came about when auditing [tokio-util's udp frame], where they want
to pass the unitialized portion of a `BytesMut` to [ReadBuf::uninit].
They need to do this unsafe pointer casting in a few places, which
complicates audits. This method lets us document the safety invariants
the caller needs to maintain when doing this conversion.

[tokio-util's udp frame]: https://github.com/tokio-rs/tokio/blob/master/tokio-util/src/udp/frame.rs#L87
[ReadBuf::uninit]: https://docs.rs/tokio/latest/tokio/io/struct.ReadBuf.html#method.uninit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants