-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read 20MBs but send response in chunks of 2MB #7990
base: refactor-usage-sn
Are you sure you want to change the base?
Conversation
$bytesToRead = min(APP_STORAGE_READ_BUFFER, $size - $bytesRead); | ||
|
||
// Read data from the device and append to the buffer | ||
$buffer .= $deviceFiles->read($path, $bytesRead, $bytesToRead); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should append to buffer, because this means by the end whole file in the memory, and this logic is to prevent exactly that. Just keep the recent chunk in the buffer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are not appending the entire file to the buffer. If you look at the loop below, we remove each chunk from the buffer after processing.
// Remove the sent chunk from the buffer
$buffer = substr($buffer, MAX_OUTPUT_CHUNK_SIZE);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do this, why not instead of $buffer .= $deviceFiles->read($path, $bytesRead, $bytesToRead);
simply use $buffer = $deviceFiles->read($path, $bytesRead, $bytesToRead);
and when loop ends simply $buffer = null
. The substr
adds more processing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please check the comment
What does this PR do?
The code efficiently reads data in chunks of APP_STORAGE_READ_BUFFER size and sends it in chunks of MAX_OUTPUT_CHUNK_SIZE, optimizing the transfer process for large files.
Test Plan
Tested using node SDK. Download time went down
4x
from 68528ms to 25188ms.Related PRs and Issues
Checklist