feat(lib-storage): improve performance by reducing buffer copies

[kuhe]

Issue

this is a refresh of #2612 taking into account migrated packages

Testing

• unit
• integration
• e2e

[kuhe]

Prior to changes, uploading data of 20 million bytes:

// Observer code
const ogBuffer = global.Buffer;
const ogBufferFrom = global.Buffer.from;

let bytesCopied = 0;

class BufferWrapper extends ogBuffer {
  static from(...args) {
    const buf = ogBufferFrom(...args);
    if (typeof args[0] === "string") {
      console.log(`Creating buffer from string with length=${buf.byteLength}`);
      bytesCopied += buf.byteLength;
    } else {
      console.log(`Static ${args.length === 3 ? "view-" : "copy-"}constructing buffer of length=${buf.byteLength}`);
      console.log("\t", args.length, args[0].constructor.name, ...args.slice(1));
      if (args.length === 3) {
      } else {
        bytesCopied += buf.byteLength;
      }
    }
    return buf;
  }
}

global.Buffer = BufferWrapper;
ogBuffer.from = BufferWrapper.from;

Static copy-constructing buffer of length=20240000
         1 Buffer
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 6848 20
Creating buffer from string with length=1286
Static view-constructing buffer of length=1286
         3 ArrayBuffer 6872 1286
Creating buffer from string with length=44
Creating buffer from string with length=8
Static view-constructing buffer of length=8
         3 ArrayBuffer 48 8
Creating buffer from string with length=9
Static view-constructing buffer of length=9
         3 ArrayBuffer 56 9
Creating buffer from string with length=2
Static view-constructing buffer of length=2
         3 ArrayBuffer 72 2
Creating buffer from string with length=12
Static view-constructing buffer of length=12
         3 ArrayBuffer 80 12
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 96 133
Static view-constructing buffer of length=383
         3 ArrayBuffer 0 383
Static view-constructing buffer of length=5242880
         3 ArrayBuffer 0 5242880
Static view-constructing buffer of length=5242880
         3 ArrayBuffer 5242880 5242880
Static view-constructing buffer of length=5242880
         3 ArrayBuffer 10485760 5242880
Static view-constructing buffer of length=4511360
         3 ArrayBuffer 0 4511360
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 656 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 680 1529
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 2256 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 2280 1529
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 3856 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 3880 1529
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 5456 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 5480 1529
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7016 133
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7152 133
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7288 133
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7424 133
Static copy-constructing buffer of length=5242880
         1 Buffer
Static copy-constructing buffer of length=4511360
         1 Buffer
Static copy-constructing buffer of length=5242880
         1 Buffer
Static copy-constructing buffer of length=5242880
         1 Buffer
Creating buffer from string with length=521
Creating buffer from string with length=521
Static view-constructing buffer of length=521
         3 ArrayBuffer 0 521
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 568 20
Creating buffer from string with length=1517
Static view-constructing buffer of length=1517
         3 ArrayBuffer 592 1517
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 2112 133
Creating buffer from string with length=521
Static view-constructing buffer of length=349
         3 ArrayBuffer 0 349
Static view-constructing buffer of length=349
         3 ArrayBuffer 0 349
DONE 200 Readable
DONE bytes copied: 40,491,715

After changes to this and node-http-handler

Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 6848 20
Creating buffer from string with length=1286
Static view-constructing buffer of length=1286
         3 ArrayBuffer 6872 1286
Creating buffer from string with length=44
Creating buffer from string with length=8
Static view-constructing buffer of length=8
         3 ArrayBuffer 48 8
Creating buffer from string with length=9
Static view-constructing buffer of length=9
         3 ArrayBuffer 56 9
Creating buffer from string with length=2
Static view-constructing buffer of length=2
         3 ArrayBuffer 72 2
Creating buffer from string with length=12
Static view-constructing buffer of length=12
         3 ArrayBuffer 80 12
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 96 133
Static view-constructing buffer of length=383
         3 ArrayBuffer 0 383
Static view-constructing buffer of length=5242880
         3 ArrayBuffer 0 5242880
Static view-constructing buffer of length=5242880
         3 ArrayBuffer 5242880 5242880
Static view-constructing buffer of length=5242880
         3 ArrayBuffer 10485760 5242880
Static view-constructing buffer of length=4511360
         3 ArrayBuffer 15728640 4511360
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 656 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 680 1529
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 2256 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 2280 1529
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 3856 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 3880 1529
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 5456 20
Creating buffer from string with length=1529
Static view-constructing buffer of length=1529
         3 ArrayBuffer 5480 1529
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7016 133
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7152 133
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7288 133
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 7424 133
Creating buffer from string with length=521
Static view-constructing buffer of length=521
         3 ArrayBuffer 7560 521
Creating buffer from string with length=40
Creating buffer from string with length=20
Static view-constructing buffer of length=20
         3 ArrayBuffer 8128 20
Creating buffer from string with length=1517
Static view-constructing buffer of length=1517
         3 ArrayBuffer 0 1517
Creating buffer from string with length=133
Static view-constructing buffer of length=133
         3 ArrayBuffer 1520 133
Static view-constructing buffer of length=349
         3 ArrayBuffer 0 349
Static view-constructing buffer of length=349
         3 ArrayBuffer 0 349
DONE 200 Readable
DONE bytes copied: 10,673

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.