Use better gzip/deflate

[klauspost]

The standard library is more interested in maintaining status quo than improving compression.

I maintain a compression package that seems more suited to Hertz. It plugs in without any modifications, by simply replacing imports.

Below is a comparison between my package and the standard library, using 4GB of mixed web content:

out	level	insize	outsize	millis	mb/s	Files/s	MB/s	Reduction	Rel speed	Smaller	Gain
gzstd	1	4014735833	862232427	20605	185.81	2617	185.82	78.52%	1.00x	0.000%	0.000%
gzkp	1	4014735833	866657631	13860	276.24	3891	276.24	78.41%	1.49x	-0.513%	-0.110%
gzstd	2	4014735833	813642417	26380	145.14	2044	145.14	79.73%	1.00x	0.000%	0.000%
gzkp	2	4014735833	842584206	15322	249.87	3520	249.89	79.01%	1.72x	-3.557%	-0.721%
gzstd	3	4014735833	800582015	27842	137.51	1937	137.52	80.06%	1.00x	0.000%	0.000%
gzkp	3	4014735833	823209525	17003	225.17	3172	225.18	79.50%	1.64x	-2.826%	-0.564%
gzstd	4	4014735833	765294480	34710	110.31	1554	110.31	80.94%	1.00x	0.000%	0.000%
gzkp	4	4014735833	791607893	19778	193.58	2727	193.59	80.28%	1.75x	-3.438%	-0.655%
gzstd	5	4014735833	744808940	42672	89.72	1264	89.73	81.45%	1.00x	0.000%	0.000%
gzkp	5	4014735833	764683084	22471	170.38	2400	170.39	80.95%	1.90x	-2.668%	-0.495%
gzstd	6	4014735833	735318526	51993	73.64	1037	73.64	81.68%	1.00x	0.000%	0.000%
gzkp	6	4014735833	753831382	24787	154.46	2176	154.47	81.22%	2.10x	-2.518%	-0.461%
gzstd	7	4014735833	733144257	60588	63.19	890	63.19	81.74%	1.00x	0.000%	0.000%
gzkp	7	4014735833	746309058	34562	110.78	1560	110.78	81.41%	1.75x	-1.796%	-0.328%
gzstd	8	4014735833	730369312	77883	49.16	692	49.16	81.81%	1.00x	0.000%	0.000%
gzkp	8	4014735833	735523067	43270	88.48	1246	88.49	81.68%	1.80x	-0.706%	-0.128%
gzstd	9	4014735833	729905331	94517	40.51	571	40.51	81.82%	1.00x	0.000%	0.000%
gzkp	9	4014735833	727900632	94305	40.6	572	40.60	81.87%	1.00x	0.275%	0.050%
gzstd	-2	4014735833	2617482950	12742	300.48	4232	300.48	34.80%	1.00x	0.000%	0.000%
gzkp	-2	4014735833	2604521759	9281	412.49	5810	412.54	35.13%	1.37x	0.495%	0.323%

The standard library is poorly balanced, so levels 6-8 have very little difference. This is more spread out so the actual level reflects an increase in compression. Here is the compression/speed tradeoff visualized:

The package uses no cgo or "unsafe" code.

TLDR; With level 6 (default in your setup), you get 2.1x the throughput at 2.7% more output on the wire, with better options to tweak speed/compression.

Describe the solution you'd like

Replace gzip/flate imports with corresponding packages.

Additional context

The package also provides stateless compression modes which can be useful for long-running requests with little data throughput.

Each Write will be compressed immediately without any state being retained between calls. I don't know if this has any use here.

[li-jin-gou]

thanks @klauspost I think it's a great idea. What do you think? @welkeyever @Duslia @FGYFFFF

[welkeyever]

Sounds great! Really appreciate it, we will look into it~

[errocks]

I'm working on an object storage project , this sounds interesting, can I try this?

[li-jin-gou]

Hello @klauspost I'm going to start by using klauspost/compress in the gzip extension of hertz and see the benefits and our code generation tool will integrate the gzip extension by default.