Running on serverless Docker (Azure Container Apps, Google Cloud Run) #6063
-
|
I realize that if I want to maximize performance I should just benchmark my different options and choose the best one for my use case, but as a rule of thumb: should I use the multi-threaded runtime when running on these sorts of server-less platforms? Where each container is getting allocated less than 1 cpu? I doubt the increased cold start time would be worth it (?). |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 9 replies
-
|
Great question! There isn't a hard and fast rule here. It's going to depend on what you're doing, what your traffic looks like, and what the environment you are running on is like. I'd say that, in general, multi-thread is probably going to be optimal, but that you should definitely experiment on production traffic with different settings to see what gives you better results. In particular, you're going to want to tune to see how many threads is ideal if you are using multi-thread. I will caution strongly against the use of synthetic benchmarks here. They are almost never applicable when trying to tune performance like this. |
Beta Was this translation helpful? Give feedback.
-
|
How much traffic are you expecting here? Generally I'd expect a few allocated cores per container with multi-thread to be the sweet spot. |
Beta Was this translation helpful? Give feedback.
-
|
no traffic at times, and 200 to 1000 rq/s at other times. It won't ever really get past that. ACA has a maximum of 4 CPUs per replica. AWS Fargate has a maximum of 16 CPUs now if I'm not mistaken. Google Cloud Run has a maximum of 1 CPU. each request is just making a request to another service and awaiting. Also I'm deploying a few other services that are gonna handle like 10 rq/s max. So I guess for the first one I should allocate more cores and use multi threaded rt. Except if I end up deploying on Cloud Run (?) |
Beta Was this translation helpful? Give feedback.
-
|
I'd run some experiments here with different platforms and core counts (with 1 core using current_thread) in an A/B manner to look at what gets you the best tail latency at the lowest cost. Don't run these experiments at separate times or dates, as traffic noise will drown out your signal. I'd recommend instead sharding your connections equally between experiment groups using something like round-robin DNS, and looking at which groups get the best tail latency (P95 at peak traffic or another critical point is often useful here) for the lowest price. I'd recommend putting your data into grafana or similar graphs so that you can look at how things look at various points in time. I've run a lot of experiments such as this in the past, so feel free to ask more questions. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! You've been incredibly helpful! I think I have a good plan for now. I'll bump this gain in case anything comes up! |
Beta Was this translation helpful? Give feedback.
-
|
No problem! Good luck with the project! |
Beta Was this translation helpful? Give feedback.
Great question!
There isn't a hard and fast rule here. It's going to depend on what you're doing, what your traffic looks like, and what the environment you are running on is like.
I'd say that, in general, multi-thread is probably going to be optimal, but that you should definitely experiment on production traffic with different settings to see what gives you better results. In particular, you're going to want to tune to see how many threads is ideal if you are using multi-thread.
I will caution strongly against the use of synthetic benchmarks here. They are almost never applicable when trying to tune performance like this.