DigitalOcean: automate app creation and redundancy

[Ambient-Impact]

Problem

On more than one occasion, our deployment process has face planted and rollbacks have not worked, leaving the site unusable, usually due to an issue on DigitalOcean's end. The most recent example occurred yesterday, and is now resolved for us, but underscores the need for more redundancy to reduce the chances of this happening.

Blockers

At the moment, it's tedious to set up a second App Platform app due to all of the environment variables that need to be set up manually, with App Platform seemingly not picking up our .do/app.yaml automatically to set all those up along with the various components. Automating the whole process of creating a new app instance with all of the components and environment variables seems like an important step and requirement so that we can easily deploy a new instance when needed without having to run it continuously and incur hosting costs for something we're not using most of the time.

Solutions

The ideal solution is probably a form of the blue/green deployment strategy so that we can verify that a deployment has succeeded before it goes live. App Platform essentially does this with its Kubernetes-based architecture, but when it fails like in the incident linked above, it leaves us without the ability to do anything to recover on our end, so an additional layer of redundancy is preferable.

The first step will be to automate the creation of a second App Platform app, complete with all the environment variables and components. It looks like DigitalOcean's doctl supports using local .env files to populate/override environment variables in some cases, so if this can work for creating apps, that would be the first choice; if that isn't supported, doctl apps create does support reading the app spec from stdin which should allow us to load our app spec, replace placeholders with environment variable values, and pass the result to doctl. Plain text environment variables marked as type: SECRET get automatically encrypted on DigitalOcean's end, so that part should just work.

The above will solve the biggest issue of verifying that App Platform does fully deploy successfully, but it doesn't offer an easy way to switch between apps outside of waiting for DNS changes to propagate after manually switching domains between apps, which is hardly ideal. One possibility may be to set up our own reverse proxy (e.g. Varnish) on a Droplet which we can switch over to the other app instantly. The downside to hosting our own reverse proxy is that we'd have to maintain it and it may incur significant bandwidth costs. Another option is to use an external service such as Deflect as the reverse proxy if they support quick switch overs in some form.