Introducing SSR benchmarks

[qin-guan]

In reference to this comment, it might be possible to look into introducing SSR benchmarks for frameworks.

Some considerations I have off the top of my head:

• What is a framework? Should benchmarks use library primitives (React/Vue renderToString) or meta-framework implementations?
• How to handle caching? Should it be completely disabled?
• What metrics to look out for, would purely throughput be a good representation of user/real world experience?
• What scenarios to benchmark? Initial thoughts maybe can make an SSR version of TodoMVC

[pi0]

Thanks for opening up the discussion ❤️

Regarding caching, one option would be using ssr meta-frameworks like Next.js and Nuxt to provide page-level caching options they all provide.

However, we might also use HTTP framework + simple SSR component rendering. I guess no major difference in using which SSR framework and disabled cache can be best baseline. My main reason for initiating this idea was to make "HTTP framework benchmarks" more realistic (i.e., closer numbers as the complexity of an SSR scenario adds up to the more real-world scenario)

To make it even more real, we might also consider having a local "API" server that during rendering components fetch them. This is often of the times the real performance bottleneck.

[qin-guan]

Hey @pi0!

My apologies 😅 , I am referring to the cache at the component rendering level. I think at page-level does make more sense and I’ve updated in the main post.

I was misinformed about React having renderer cache built in. For Vue, I was originally referring to the lack of cache highlighted in this issue: vuejs/core#1791

But I agree disabling cache is the best way to do this benchmarking though, so not a big concern here!

[pi0]

Indeed. Component-level caching for Vue 3, was deprecated as it wasn't a popular option amongst developers and brought lots of edge cases in ssr rendering... Which is why mentioned about page-level caching is which more "common comparable" feature and indeed benchmarking without cache is best 👍🏼

(However i would also be up to have "Cache" system benchmarks. Both React/Next and Nitro/Nuxt provide caching APIs which we can benchmark and compare. If the final goal of benchmarks is to give users realistic scenarios, cache is not an exception to this... however probably another story for another day...)