How to make ReadableStreamByobReader.read_with_u8_array() work?

[rom1v]

Summary

I have a ReadableStream. I create a ReadableStreamByobReader by calling readable_stream.get_reader_with_options(…).

On this reader, I want to read to a user-provided array, so I use reader.read_with_u8_array().

But I always get the following error:

TypeError: Failed to execute 'read' on 'ReadableStreamBYOBReader': Could not transfer ArrayBuffer

Here is a sample for illustration:

fn sample(readable: web_sys::ReadableStream, buf: &mut [u8]) {
    let reader = readable.
        get_reader_with_options(
            web_sys::ReadableStreamGetReaderOptions::new()
                .mode(web_sys::ReadableStreamReaderMode::Byob),
            )
            .dyn_into::<web_sys::ReadableStreamByobReader>()
            .expect("Invalid readable stream");
    let promise = reader.read_with_u8_array(buf);
    let obj = JsFuture::from(promise).await;
    console::log_1(&JsValue::from_str(&format!("{obj:?}")));
}

In the console, I get:

Err(JsValue(TypeError: Failed to execute 'read' on 'ReadableStreamBYOBReader': Could not transfer ArrayBuffer

(btw, the console log seems truncated, I don't know why)

[daxpedda]

This is apparently a limitation by the spec.

1. read(view)
2. ReadableStreamBYOBReaderRead(reader, view, readIntoRequest)
3. ReadableByteStreamControllerPullInto(controller, view, readIntoRequest)
4. TransferArrayBuffer

Which already has a note there saying:

This will throw an exception if O has an [[ArrayBufferDetachKey]] that is not undefined, such as a WebAssembly.Memory's buffer.

---

This is actually not correct anymore, see the new spec:

Set buffer.[[ArrayBufferDetachKey]] to "WebAssembly.Memory".

Was fixed in 2018: WebAssembly/spec#889.

I guess browsers didn't catch up yet, tried it in Firefox and Chrome.
This probably needs a bug report there.

[daxpedda]

Ah! Missed the not!

[rom1v]

So (just to be sure to understand), there is no way read_with_u8_array() could work, is it correct? (if so, maybe it should be removed)

[daxpedda]

Good point.

[Liamolucko]

So (just to be sure to understand), there is no way read_with_u8_array() could work, is it correct? (if so, maybe it should be removed)

Yes.

There's actually a pretty good reason for this. ReadableStreamBYOBReader doesn't quite work like you'd expect: you can't just pass it a Uint8Array, wait for the promise to resolve, and then read the results out of the original buffer like you can in something like Rust's std::io::Read::read.

This is because it actually takes control over the Uint8Array while it's being read into. You can't access it anymore until the promise gives it back to you, and the original reference to it stays invalid forever. You can only access it through the new reference that the promise returns.

Critically, it doesn't just take control over the Uint8Array you pass it: it takes control of the entire backing ArrayBuffer. If it didn't, you could easily bypass it taking control of the Uint8Array by just making a new Uint8Array view of the same bit of the ArrayBuffer. This usually works fine, since the rest of the ArrayBuffer is just storing what you've read so far or something along those lines, and you don't really need to access it while reading.

However, when the Uint8Array is a view of Rust's memory, this means that its backing ArrayBuffer is... the entirety of Rust's memory! So attempting to use ReadableStreamBYOBReader to read into Rust's memory would cause it to take ownership of the whole thing, and your Rust code just wouldn't have any memory anymore. That would be really bad! So that's why ReadableStreamBYOBReader doesn't let you do this.

[MOZGIII]

I think read_with_u8_array should be unsafe since it takes ownership over the memory but takes a mere mut ref.
Would this be correct?

[daxpedda]

The API was already completely removed: #3582.
But yeah, unsafe would have been good here, luckily we know now the spec doesn't allow taking ownership of the Wasm linear memory like that.

[rom1v]

If I call reader.read_array_with_buffer_view() (instead of reader.read_with_u8_array()), then it works.

Here is a modified sample:

fn sample(readable: web_sys::ReadableStream, buf: &mut [u8]) -> Result<Option<usize>, JsValue> {
    let reader = readable.
        get_reader_with_options(
            web_sys::ReadableStreamGetReaderOptions::new()
                .mode(web_sys::ReadableStreamReaderMode::Byob),
            )
            .dyn_into::<web_sys::ReadableStreamByobReader>()
            .expect("Invalid readable stream");
    let typed_array = js_sys::Uint8Array::new(&JsValue::from(buf.len()));
    let promise = reader.read_with_array_buffer_view(&typed_array);
    let obj = JsFuture::from(promise).await?;

    // now it works, and I can retrieve the value

    let done = js_sys::Reflect::get(&obj, &JsValue::from("done"))?
        .as_bool()
        .unwrap_or(false);
    if done {
        Ok(None)
    } else {
        let array = js_sys::Reflect::get(&obj, &JsValue::from("value"))?
            .dyn_into::<js_sys::Uint8Array>()
            .expect("Unexpected read value");

        let len = array.byte_length() as usize;
        array.copy_to(&mut buf[..len]);
        Ok(Some(len))
    }
}

[daxpedda]

Well, yeah, because you aren't using the Wasm memory anymore, with the downside of requiring a copy.

By the way, you can use ReadableStreamReadResult to read the result instead of doing it manually.

[rom1v]

By the way, you can use ReadableStreamReadResult to read the result instead of doing it manually.

I'm not sure how to do that: this struct contains only setters (not getters). And how could I get this typed structure from the promise result?

[daxpedda]

I'm not sure how to do that: this struct contains only setters (not getters).

Ah, true ...

And how could I get this typed structure from the promise result?

With unchecked_into(), but I guess without getters that pointless.