Low performance when download a single large file by S3 Transfer Manger

[HanzhouTang]

Hi, we are trying to download a single large file (more than 70G) by Transfer Manger in I3.16xlarge, AL2 system.

Although the max network bandwidth is 25 Gbps according to doc, we found out that when downloading a multi-part single large file, the download speed is usually 4~6 Gbps.

We did several rounds of profiling and it seems like when multi-threads (about 90 threads) try to write to the same large file, most of the threads are competing to hold the file lock due to write operation is atomic.

After we split the large file into 10G chunks, we can improve the download speed to about 20Gbps.

I am wondering could you please help us confirm if the observation makes sense? And if it's a known behavior of the Transfer manager? Do you have any any suggestions to improve the download speed of a single large file besides splitting the file? (We are concerned about chunking because it could introduce file corruption issue).

Looking forward to your kind reply.