Log compression on archival

[MichielDeMey]

For some artifact repositories (such as S3 and GCS) we can leverage Gzip compression to reduce the log storage size.
Since text is highly compressible, I believe we can greatly reduce storage size and storage cost.

Both support Gzip compression, all we need to do is compress the logs before sending it to the storage provider.

We do need to tell the storage provider that the incoming file is Gzip compressed, but most storage SDKs provide a way for you to do this.

It usually comes down to setting the Content-Encoding header along with the correct Content-Type.

Content-Type: text/plain
Content-Encoding: gzip

As for downloading a Gzipped log file, we should not need to modify any existing code.
GCS can automatically decompress the Gzipped file.

Documentation: https://cloud.google.com/storage/docs/transcoding#decompressive_transcoding

As for S3, it should be similar although I have no practical experience.

Happy to discuss!

[alexec]

@MichielDeMey this sounds like a feature request?

[MichielDeMey]

Correct, I added this in the “Idea” category here. 😉