GIF: 250 frames gifs exceed 30s processing limit enforced by CDN

[swistaczek]

Hey,
We are users of the pro version, and we noticed that resizing gifs that include 250 frames results in an extended processing time (based on longs, I assume it's above 60 seconds). Everything that ends up being processed above 30s will be terminated by GCP CDN.

Do you consider it a bug, or do you have any recommendations on how we could speed up the process to make it in the 30s CDN window?

testing image: https://cdn.fourthwall.com/offer/sh_8314a8a6-9ebd-4c7c-aca4-d20ab8eef781/8b8b7fc1-cb4e-4f7b-bc06-20d64326de5c.gif

[DarthSim]

Hi @swistaczek!

Oh, GIFs are a painful subject... There are multiple problems:

1. GIF encoding is slow. It involves the quantization and remapping of each frame. And this is a pretty slow process. We do our best to optimize it, but we can't overperform the CPU.
2. Since GIF compression is pretty poor, GIFs require additional optimizations like frame cropping and alpha optimizations. This takes some time too.
3. Encoding to GIF is poorly optimizable. Most steps can't be parallelized so using such things as SIMD instructions is simply impossible in most cases.
4. GIFs may contain many frames. The image you provided contains 250 frames, which means that imgproxy in fact needs to process and encode 250 images.
5. Resizing of GIFs usually produces larger files. GIF images may have 256 colors at the max, but resizing produces new colors. This leads to additional dithering and this leads to worse compression.

Considering all these points, the best way to optimize GIF processing is to not touch GIFs at all unless you resize them to a much smaller size. I know that some of our users just set IMGPROXY_SKIP_PROCESSING_FORMATS=gif.

As an alternative, I could recommend to convert animations to MP4. This produces much smaller files, and encoding to MP4 is much faster. However, this requires a special treatment of animations as you can't put video in the <img> tag.

Also, though we host our demo instances of imgproxy on GCP ourselves, Google Cloud Run's performance is far from being the best on the market. It's enough for processing static images, but large GIFs may be a problem as you noticed. I tested processing of pretty large GIFs on Amazon's c7g instances, and it was able to chew GIFs that GCR chokes with.

Sorry for not bringing any good news here, but GIF is a real headache 😓

[swistaczek]

Thank you for your comprehensive explanation and for presenting the available solutions!

[jjhuff]

We also ran into gif performance issues and took the "just skip it" path. In our case, it was a fairly large (2k x 2k) 100 frame animation. We wanted to resize it to something more reasonable. At the time (and it's been ~1 year), it seemed like it was the memory pressure and that libvips was decompressing the whole thing into a very large filmstrip before resizing and re-encoding.

In our use case, we typically want to resize them fairly small, so I'd expect the encoding time to be less of an issue.

Also, we'd happily consider some monetary sponsorship to help move this along!

[DarthSim]

Memory shouldn't be an issue since libvips doesn't decode the whole "filmstrip" into memory. Libvips stores only a single frame in the memory. I guess, the issue in your case is the resizing itself since imgproxy has to resize 100 2K images. However, things changed a lot in the last year: resizing was optimized with SIMD intrinsics and became much faster.

[jjhuff]

BTW: can confirm that 3.21.0 is ~2x faster than 3.10.0 based on some fairly crude benchmarks. I also can confirm that, yes, the gif encoding speed is a major factor.

Thanks @DarthSim !