Improve ActiveStorage asset linking performance

[ahukkanen]

🎩 What? Why?

Use direct links to the assets living at storage services in order to push most of the application load to the storage service instead of the website itself.

Currently all uploaded assets in Decidim are served either through the ActiveStorage redirect URLs or representation URLs. This would make sense in some situations (e.g. if the files are protected) but currently all assets served by Decidim should be considered public.

Therefore, we can serve these images directly through the storage service because otherwise there will be an extra request to the service for each and every image to be served. If the page has a lot of different images, such as typical front page or listing of proposals, this adds excessive load on the servers because in worst case situations they are doing hundreds of requests to the Rails service just to serve one page. Under heavy load this becomes a real problem, e.g. if you have 100 users loading the front page at the same time containing 100 images, you would generate 10 000 requests for the Rails app in a very short timespan (note that 100 people is not that many).

To solve this issue, we can serve the images directly through the storage service bypassing all these requests coming to the Rails app. Note that some requests still need to be loaded through the app, such as the variant representation URLs during their first load (when the variant is created). But after the processing of the image variant, they can be served directly through the storage service.

📌 Related Issues

• Related to Active storage migration #7598

Testing

• Create a live instance of Decidim with a storage service configured, such as S3, Azure, Google, etc.
  • Alternatively, ask someone for permission to use their instance for load testing
• Upload a bunch of images through ActiveStorage that are displayed on the front page
• Install a benchmarking tool (e.g. using siege as pointed out by Andrés or similar tools)
• Open top at the machine running Decidim
• Run the benchmark, e.g. create 10k requests with a concurrency of 100 (depending on your server size)
• After the test finishes, store the results
• Apply this fix and repeat the test
• See highly decreased amount of requests and CPU usage

[andreslucena]

Install a benchmarking tool (first I thought of ab but it won't work in this case because it doesn't request the images)

siege does that exactly. It even has an option to disable it. From its man page:

   --no-parser
       Turn off the HTML parser. **When siege downloads a page, it parses it for additional page elements such as style-sheets, javascript and images**. It will make additional
       requests for any elements it finds. With this option enabled, siege will stop after it pulls down the main page.

(Asterisks are mine)

[alecslupu]

2 small improvements to be done.