[RFC] Consolidated dokku app and plugin configuration

[josegonzalez]

One thing we've been sorely lacking is a single way to refer and store configuration in the dokku world.

• Application-related plugins usually store their data in the app's git directory in /home/dokku
• The official datastores store their data (and mount information) in /var/lib/dokku/services/SERVICE/
• Community plugins use stuff like /home/dokku/.o_mysql
• Dokku-wide config is usually stored in /home/dokku

Because of all the random ways in which data can be stored, there isn't exactly a simple way to save the state of an existing dokku installation. You can very easily push the container images to a registry and have backups occur from datastores, but actual config? Fffft.

Our backup plugin has been more or less ignored in the 0.3.x line - totally my fault - and i think we need to start getting serious about this.

At one point, there was a note in a roadmap to store everything in a git directory. I think it's time we head down that road for 0.5.x.

---

generic spec

• Data should be stored in a directory such as /var/lib/dokku/config.
  • /var/lib/dokku/config/apps - stores application data. Each application should get it's own directory, corresponding to the application name.
  • /var/lib/dokku/config/plugins - stores plugin data. Each plugin should get it's own directory, corresponding to the plugin name.
• We should provide a new dokku-config binary that is aware of the above paths and can interact with them.
  • Apps will interface with it using the --app APP flag, and can perform general crud actions against their own config.
  • Plugins will interface with it using the --plugin PLUGIN flag. Ditto on crud actions
  • The binary is a generic key-value store, and will be capable of triggering hooks on actions. For instance, we might want to log access, or ensure all actions get saved to an external datastore (git!).
• The backup system will be retrofitted to only work with the above system. If you want a backup, you basically get a tar of that directory. Users that want a backup of datastores should interface with those datastores either directly or through their plugin interfaces.
  • The official datastores all have ways of backing up/restoring data, so this is not a problem. We also cover all major datastores.
• All dokku related files should be named with a DOKKU_ prefix. For instance, DOCKER_OPTIONS_* becomes DOKKU_DOCKER_OPTIONS*. Inconsistent config files conventions #2199

open questions

• Do we handle encryption? I don't think it's worth it.
• How do we handle stuff like nginx? Does the nginx include path just change here?
• Should we serialize the data somehow, to json maybe?
• How do we ensure that data doesn't get accidentally overwritten by a shitty plugin/app/user?
• What language should it be written in? Golang is the obvious one to me, though handling plugins in it is annoying.

@Flink @michaelshobbs thoughts?

[josegonzalez]

#1571

[michaelshobbs]

Overall I'm in agreement. Some questions and comments

• Why do we need to provide a new binary for this function as opposed to making the config plugin as it is today more robust?
• I'm not clear on the difference in interaction regarding how a plugin would interface with config vs. an app.
• 👍 on config VCS
• Involving encryption include key management. I think this is best left up to the reader for now
• I think it's fine to move nginx config into this space as well
• Why serialize to anything? I think this makes the interaction more complex unless we build that into the aforementioned binary. Doing it in bash seems hairy and overtly complex. We need a solid use case for taking this on.
• I'm not sure it's up to us to save a user from a "shitty" plugin/app/user.
• Again, why create a new binary dependency?

[josegonzalez]

I guess moving it to a separate binary would allow us to iterate on that quicker than we would on the config plugin. As well, handling input/output in bash is kind of annoying, and it would be nice to be able to use a higher-level language to handle this (not to mention it could also decrease the amount of logging output if we write it correctly).

[Flink]

Interesting ideas as I think also that we need some sort of central repository for managing config. Indeed it needs to be versioned somehow with git for example. We need to have a way to bind a revision of an application config with a tagged image to be able to do rollbacks and things like that (like Heroku) :)
Another thing I saw the other day while playing with their addons is that when an addon declares a config var on your app, you can’t manage it directly. Only the addon can manage it so this is an interesting lead to explore to avoid unwanted rewrites and thing like that.
Separate “config stores” for nginx and plugins seems a good idea. As for the serialization, if we really want that, maybe we can use toml?
For a new binary, I don’t know if it’s necessary nor if Go is a good choice ;) (I find it way too verbose among other things)

Otherwise 👍 :)

[u2mejc]

@josegonzalez so looking this over, I have a few questions:

• Do you have a copy of the source for the "binary"? I assume this is just because strings are hard to handle in bash?
• You are proposing to move all app data out of ~/dokku/, to /var/lib/dokku/config/apps/ correct?
• For data not committed in git; any changes to quasi-ephemeral data like environmental vars (ENV_FILE)?

Moving the location of the apps is simply handful of greps, is there an advantage to a new location or is it just to force adoption for thirdparty plugin authors? In the same line of light, excluding all backup abilities (manual or automated), what other advantages are gained by the dokku community with these changes? One less part written in bash is a good answer too. 😉

[josegonzalez]

• I haven't written any code for said binary.
• Not the git directories, just any config files. For instance, we'd no longer be copying nginx.conf.template or whatever into /home/dokku/DERP/, and instead to the /var/lib/dokku/config/apps/DERP/ directory.
• Any files that dokku creates that are config would somehow go in this new config dir.

The advantage is separating all our config from the bare git directory, which allows us to both simplify backup/copies of applications, but also removes any doubt about what files go where. In the past we just never gave anyone any direction on where to put stuff, so shit has gone everywhere.

[josegonzalez]

I started giving this a good, hard look. I believe once the config changes from #2751 land, we can look into this. I'd be down for something like an attributes or properties plugin. It could be a minimalist wrapper around something like bash-filedb, while allowing users to swap in/out the underlying binary that is called (in case they want to use something else, like redis or vault).

I'm not sure how data migration would work - maybe some sort of read-through layer that allowed us to fallback to a config key? - but thats something to think about. I definitely want to make sure we move all config out of the application repositories though.

[josegonzalez]

The way forward here is the properties functions - implemented in golang and shell - which we should use to migrate plugin-specific config over to new paths. An example of how this could work is in the git install hook, which migrates DOKKU_DEPLOY_BRANCH over.

[mcohen9]

Whats the status on this? Looking for a way to backup and restore my dokku apps,plugins and configurations.

[josegonzalez]

@mcohen9 Hi! Any "status" would likely be posted on this issue - either in the form of a comment, linked pr, or issue closure.

As this is an open source project, any change in "status" to an issue will have to be contributed by our users - this includes you! - or project maintainers at their leisure. If you'd like to sponsor the work on this, it's probably ~20 hours of my time to move everything over to the new system, so feel free to email me if thats something you want to consider.

As always, pull requests welcome!

[josegonzalez]

Open question: Where should application repositories be stored? /var/lib/dokku/data/git might be reasonable. One thing moving the repositories breaks is how applications are listed/verified as existing, so we'd want to track that separately as well.

[t0xicCode]

@josegonzalez anything I could do to help?

[josegonzalez]

Biggest thing would be to see what files are being read from the DOKKU_ROOT dir that aren't git-related and move them to the appropriate place in /var/lib/dokku. I haven't gone through and documented these just yet, so that would be a good first step.

[josegonzalez]

Update:

• IP/PORT files can probably be computed on the fly and removed.
• ENV file should move to /var/lib/dokku/config/config/$APP/ENV
• DOCKER_OPTIONS_* can move to /var/lib/dokku/config/docker-options/$APP/phase-*, where * is the name of the phase. This will also allow us to support process-specific options at some point.
• nginx.conf files (and certificates) should move to /etc/nginx/dokku.d (and tls somewhere appropriate). There is a ticket open for that (Move nginx config to /etc/nginx/dokku.d #3443).

I'm going to try and move the ENV/docker options files and refactor the ip/port stuff in the next PR, after which we can close this issue as everything else is tracked separately.