Dynamic Deployment and Routing

[thegagne]

It would be fantastic if there was a way to deploy scripts to some sort of self-run orchestrator that was capable of deploying and running multiple scripts, updating existing scripts, and configuring a lightweight routing layer, and targeting these deployments via wrangler.

Dev XP:
In the wrangler.toml, allow definition of deployment_target = http://workers-api.domain.com.
When this is defined, wrangler publish contacts this endpoint instead of Cloudflare.

API:
This could API would have identical surface as the standard worker apis (or maybe a subset). Under the hood, operation would obviously be a little different.

Routing:
Configurable from the api, this would setup a lightweight routing engine that is capable of creating routes with deployed scripts as backends.

Script deployment:
This gets tricky and I'm not going to propose a specific method for actually running the various scripts. Not my area of expertise, but something that requires minimal footprint, but could potentially scale horizontally if necessary.

The end result should be a fully self-run worker environment, capable of running multiple scripts, exposing routes, etc. Bonus if it eventually supports some of the other worker features such as shared KV storage, Crons, Cache, etc..

This raises a real question, how much of Cloudflares full toolset could be run locally, and why would they give away this capability?

I think there are real benefits to having the ability to run this stuff in your own private network. Building things like internal tools without having to switch to use a different toolset would be amazing. Running this in your own private network does not remove the need for Cloudflares other offerings, especially the fantastic network and DDOS protection. I think it rather complements it and provides some peace of mind that if CF had a major outage for a couple days, you could potentially have a DR plan and limp along with something you run yourself.

[kentonv]

Hi @thegagne,

Yes, we need some sort of deployment story. I agree that you should be able to "wrangler publish" to your own self-hosted cluster.

That said, I'm not sure that recreating the whole Cloudflare service is the right approach. I think we want something that integrates well into the self-managed stacks people are already using today, so that workerd can easily live along-side other server technologies. For example, I think we should support deploying to a kubernetes cluster.

Basically what I'm imagining is wrangler could be given a set of Workers-based applications, build them all, and then bundle them together into a single workerd config bundle (maybe even a binary with workerd compile). Then, it would create a container image with that, and deploy that to k8s.

That said, I personally don't have much experience with k8s, and I could be way off-base on how this should work.

What do you think?

[kentonv]

The way we do things internally is very different and very custom. Workers is one component of the overall Cloudflare stack. We do co-host multiple workers in the same process, but with a whole bunch of safety measures around that -- some of which are not just code, but more like business processes, so we couldn't even open source them if we wanted to.

But let's say we could somehow package up our whole stack including business processes. It actually wouldn't be very useful to most people -- unless your goal was specifically to build a Cloudflare competitor which, well, we'd rather you not. :) Our goal with workerd is really to address the needs of developers, not competitors. That means: (1) enable realistic local testing, (2) enable people to self-host their own applications on top of common non-Cloudflare cloud infrastructure (to avoid being locked into Cloudflare).

So for these specific goals, I think we actually want something pretty different from how Cloudflare works. Deploying to things like kubernetes makes a lot of sense since that's the environment a lot of people have available to them. This is new to me, though, since this actually isn't how we deploy the workers runtime internally.

[thegagne]

Makes sense. to sum up a little what we have talked about here:

A typical pattern would be each script is it's own deployment packaged with workerd.
The alternative is workerd deploys by itself along with an api to allow scripts to be dynamically added and routed.

Option A:

• very typical and comfortable for people who self-host, but very different than normal worker deployment
• other than the run-time, not as proprietary
• probably more extensible to add custom logging, or apm, etc with normal tooling
• less responsibility for Cloudflare to provide a full solution, as it is expected that self-hosted solutions already exist

Option B:

• maybe more comfortable for those that start with workers and work back to self-hosting, rather than the other way around
• One set of tooling for both cloud and self-hosted
• more proprietary, more responsibility on Cloudflare to get this right and continue to support it long term

I'm tempted to agree with your very strong points, but it might be nice to get some other opinions.

[berkant]

The alternative is workerd deploys by itself along with an api to allow scripts to be dynamically added and routed.

I'm working on a prototype for this (where workerd is a system service, meaning it is present in every node in the data plane), implementing a basic RPC interface to dynamically manage workers and their config. I'm also building a custom resource-controller pair to integrate this with Kubernetes. Will share the link here when I deem it's ready for a test drive.

[sanderpick]

Hey @berkant, I'd love to hear how this is going / collaborate / test!

[tom-sherman]

Hey, same! I've been thinking about this a lot and one aspect I'm not sure is easily solved is evicting/swapping of isolates. See #351

At some point you will hit the v8 memory limit, so there needs to be some way to evict non-active isolates and swap them out with others. An approach outside of workerd could look like tracking which workers are not in use, and periodically restarting workerd with a different set of in-use workers. I'm not sure how well this would work though, and certainly increases the complexity (what if the new workerd process fails to start).

I spoke to Kenton on discord about this, and one main thing that you need to be able to do is start another workerd process, and then gracefully start routing traffic to the new process. This can be done at the k8s pod level I presume, but that seems a bit clunky for a worker deploy so it makes sense to do this in a wrapping program that spawns workerd.