Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: ReadableStream is malformed when protocol.handle'd #41872

Closed
3 tasks done
mnvr opened this issue Apr 16, 2024 · 2 comments · Fixed by #41894
Closed
3 tasks done

[Bug]: ReadableStream is malformed when protocol.handle'd #41872

mnvr opened this issue Apr 16, 2024 · 2 comments · Fixed by #41894
Labels
29-x-y 30-x-y arch/arm64 bug 🪲 component/protocol has-repro-gist Issue can be reproduced with code at https://gist.github.com/ status/confirmed A maintainer reproduced the bug or agreed with the feature

Comments

@mnvr
Copy link

mnvr commented Apr 16, 2024

Preflight Checklist

Electron Version

30.x, 29.x

What operating system are you using?

macOS

Operating System Version

macOS 14.4.1

What arch are you using?

arm64 (including Apple Silicon)

Last Known Working Electron version

No response

Expected Behavior

  1. [main] Define a custom protocol
protocol.handle("myscheme", async (request) => {
    try {  await fs.rm("/tmp/myscheme.txt")   } catch {}
    for await (const p of request.body) {
        await fs.appendFile("/tmp/myscheme.txt", p)
    }
    return new Response("", { status: 200 })
})
  1. [renderer] Make a request with the request body as a stream using standard web fetch + duplex "half"
const dataStream = () =>
    new ReadableStream({
        async start(controller) {
            for (let i = 0; i < 10; i++) {
                // await wait(1000)
                controller.enqueue(Array(1024 * 128).fill(+i).join("\n"))
            }
            controller.close()
        },
    }).pipeThrough(new TextEncoderStream())

fetch(new Request("myscheme://host", {
    method: "POST",
    body: dataStream(),
    duplex: "half",
}))
  1. [main] Verify the data (e.g. using shasum)

Expectation: The stream should come through intact.

Actual Behavior

The stream gets malformed.

In the attached fiddle gist, you can see two buttons.

  • Send button sends the entire payload in the request body
  • Stream button streams the payload in the request body (to pass a stream as the request body, we also need to set "duplex" to "half")

Send always works.

$ shasum /tmp/myscheme.txt
d269175ac71ed60bb96f08881c39453c8e8d7020  /tmp/myscheme.txt

Stream doesn't work. The resulting data (as evidenced by seeing its shasum) is different each time.

Testcase Gist URL

https://gist.github.com/mnvr/e08d9f4876fb8400b7615347b4d268eb

Additional Information

I hope I'm just missing something silly, but I tried to boil down the issue to a isolated gist, and I'm out of ideas about what to try next. I've tried with both the latest beta and the older stable 29.x.

FWIW, if the data is less than one chunk, the streaming seems to work fine (but then I guess it's not really streaming).

The gist also has a commented out await in the code that enqueues data to the stream. If we uncomment this, Stream starts to work - by adding a 1 second delay between these enqueues. But I'm not sure about this, smaller delays didn't work, and it is also possible that I've just been getting lucky with these 1 second delays.

@mnvr mnvr added the bug 🪲 label Apr 16, 2024
@electron-issue-triage electron-issue-triage bot added the has-repro-gist Issue can be reproduced with code at https://gist.github.com/ label Apr 16, 2024
mnvr added a commit to ente-io/ente that referenced this issue Apr 16, 2024
There is one piece of the puzzle still missing - the files are being
sent wholesale instead of being streamed - and this might cause memory
issues. I haven't benchmarked yet, leaving this until we get some
response from the upstream issue
electron/electron#41872 (hopefully it's just
some thing I missed).

But otherwise, ran export on a trivial library and it worked fine, so
everything is at least hooked up properly now.
@nornagon nornagon added component/protocol status/confirmed A maintainer reproduced the bug or agreed with the feature labels Apr 17, 2024
@nornagon
Copy link
Member

Thanks for the report, #41894 should address this!

@mnvr
Copy link
Author

mnvr commented Apr 19, 2024

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
29-x-y 30-x-y arch/arm64 bug 🪲 component/protocol has-repro-gist Issue can be reproduced with code at https://gist.github.com/ status/confirmed A maintainer reproduced the bug or agreed with the feature
Projects
Status: 🛠️ Fixed for Next Release
Status: 👀 Unsorted Items
Development

Successfully merging a pull request may close this issue.

3 participants