Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Use stream around MediaProxy and FileServer #9453

Closed
wants to merge 40 commits into from

Conversation

tamaina
Copy link
Member

@tamaina tamaina commented Jan 1, 2023

Resolve #7711
Based on #9459

What

Why

Additional info (optional)

@tamaina tamaina changed the title Use stream around MediaProxy and FileServer perf: Use stream around MediaProxy and FileServer Jan 1, 2023
@github-actions github-actions bot added the packages/backend Server side specific issue/PR label Jan 1, 2023
@tamaina tamaina closed this Jan 1, 2023
@tamaina
Copy link
Member Author

tamaina commented Jan 1, 2023

ReadableStream.tee() 相当のAPIがないとダメだわ

@tamaina tamaina reopened this Jan 7, 2023
@tamaina
Copy link
Member Author

tamaina commented Jan 7, 2023

ERR  1  [core]  Error [ERR_STREAM_PUSH_AFTER_EOF]: stream.push() after EOF
Trace: Error [ERR_STREAM_PUSH_AFTER_EOF]: stream.push() after EOF
    at new NodeError (node:internal/errors:393:5)
    at readableAddChunk (node:internal/streams/readable:285:30)
    at Readable.push (node:internal/streams/readable:234:10)
    at node:internal/webstreams/adapters:496:22
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'ERR_STREAM_PUSH_AFTER_EOF'
}
    at process.<anonymous> (file:///home/aqz/misskey/packages/backend/built/boot/index.js:43:17)
    at process.emit (node:events:525:35)
    at process.emit (node:domain:489:12)
    at process._fatalException (node:internal/process/execution:149:25)

が出てくる

@codecov
Copy link

codecov bot commented Jan 7, 2023

Codecov Report

Merging #9453 (cdef862) into develop (2c52655) will decrease coverage by 0.09%.
The diff coverage is 22.28%.

❗ Current head cdef862 differs from pull request most recent head cb2823d. Consider uploading reports for the commit cb2823d to get more accurate results

@@             Coverage Diff             @@
##           develop    #9453      +/-   ##
===========================================
- Coverage    22.72%   22.63%   -0.09%     
===========================================
  Files          734      731       -3     
  Lines        68747    68609     -138     
  Branches      2028     2033       +5     
===========================================
- Hits         15620    15527      -93     
+ Misses       53127    53082      -45     
Impacted Files Coverage Δ
packages/backend/src/server/FileServerService.ts 0.00% <0.00%> (ø)
...ages/backend/src/server/MediaProxyServerService.ts 0.00% <0.00%> (ø)
packages/backend/src/core/DownloadService.ts 51.49% <26.47%> (-5.79%) ⬇️
packages/backend/src/core/FileInfoService.ts 55.77% <26.53%> (-4.09%) ⬇️
...ackages/backend/src/core/ImageProcessingService.ts 50.34% <48.64%> (+2.98%) ⬆️
packages/backend/src/logger.ts 62.29% <100.00%> (-1.35%) ⬇️
packages/backend/src/misc/secure-rndstr.ts 90.47% <0.00%> (-9.53%) ⬇️
packages/backend/src/core/RoleService.ts 36.27% <0.00%> (-0.31%) ⬇️
...ackages/backend/src/models/entities/UserProfile.ts 97.84% <0.00%> (-0.12%) ⬇️
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@tamaina
Copy link
Member Author

tamaina commented Jan 7, 2023

nodejs/node#45026

これなんですかね

@tamaina tamaina marked this pull request as ready for review January 15, 2023 13:39
@tamaina
Copy link
Member Author

tamaina commented Jan 15, 2023

p1.a9z.devで動作中

(.node-versionを>= 18.13.0 < 19.0.0 >= 19.4.0みたいにしたいところだけど流石に許容するソフトがないか?)

@tamaina tamaina requested a review from syuilo January 15, 2023 13:41
@tamaina tamaina mentioned this pull request Jan 24, 2023
@tamaina
Copy link
Member Author

tamaina commented Jan 25, 2023

コンフリクト解消が面倒になったのでclose

@tamaina tamaina closed this Jan 25, 2023
@tamaina
Copy link
Member Author

tamaina commented Jan 25, 2023

というかこれ、NodeのWeb Streamが遅いらしくて、多分99%の場合は遅い
(重たい動画ファイルは別にして)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packages/backend Server side specific issue/PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

I want to process data around the drive as a Stream instead of storing the data in storage.
2 participants