Improve memory allocation behavior by replacing sync.Pool with custom pool implementation

[skmcgrail]

• Fixes memory usage of s3 upload manager for Go 1.13.5 and 1.13.6 #3075

Pool Benchmarks

name                  time/op
Pools/sync.Pool/0-72  23.3ms ± 6%
Pools/sync.Pool/1-72  44.1ms ± 3%
Pools/sync.Pool/2-72  51.6ms ±20%
Pools/sync.Pool/3-72   128ms ± 1%
Pools/custom/0-72     23.0ms ± 5%
Pools/custom/1-72     44.5ms ± 1%
Pools/custom/2-72     47.1ms ± 2%
Pools/custom/3-72      131ms ± 7%

name                  alloc/op
Pools/sync.Pool/0-72  10.6MB ± 0%
Pools/sync.Pool/1-72  15.9MB ± 0%
Pools/sync.Pool/2-72  26.6MB ± 0%
Pools/sync.Pool/3-72   164MB ± 0%
Pools/custom/0-72     10.6MB ± 0%
Pools/custom/1-72     10.7MB ± 0%
Pools/custom/2-72     16.0MB ± 0%
Pools/custom/3-72     59.0MB ± 0%

name                  allocs/op
Pools/sync.Pool/0-72   1.03k ± 0%
Pools/sync.Pool/1-72   1.29k ± 0%
Pools/sync.Pool/2-72   1.82k ± 0%
Pools/sync.Pool/3-72   13.4k ± 0%
Pools/custom/0-72      1.03k ± 0%
Pools/custom/1-72      1.28k ± 0%
Pools/custom/2-72      1.80k ± 0%
Pools/custom/3-72      13.4k ± 0%

Multi-Part Upload

• 150 GB File
• 16 MB PartSize
• 72 Concurrency

Expected Buffer Allocations: (72+1)*16 = 1168 MB

Before (sync.Pool)

As shown sync.Pool drastically performs over allocations, 2624 MB. This is over 124% more in allocations!

After

We see that we now allocate the exact amount of expected buffers, 1168 MB.

Performance Comparison

$ ~/go/bin/benchstat /mnt/ram/results.old /mnt/ram/results.new
name                                                         old time/op    new time/op    delta
Upload/540MB_File/72_Concurrency/5MB_PartSize/Unbuffered-72     1.50s ±27%     1.57s ±31%   +4.52%  (p=0.006 n=84+89)

name                                                         old alloc/op   new alloc/op   delta
Upload/540MB_File/72_Concurrency/5MB_PartSize/Unbuffered-72     595MB ± 0%     407MB ± 0%  -31.57%  (p=0.000 n=96+99)

name                                                         old allocs/op  new allocs/op  delta
Upload/540MB_File/72_Concurrency/5MB_PartSize/Unbuffered-72      151k ± 0%      151k ± 0%   +0.06%  (p=0.009 n=86+88)