A plugin architecture for self-hosted shields instances

[chris48s]

📋 Description

This doesn't come up all the time, but occasionally someone pops up and asks us to add a badge which would only make sense for a self-hosted install.

We usually say no to these requests. If there is no public instance of the service, it is difficult for us to test that this code works (either manually or via integration tests), making it hard to support, maintain or refactor.

That said, this has come up a non-zero number of times and the requests are not completely invalid. For some subset of self-hosting users, these badges could be useful.

So here's an idea: We provide some kind of plugin hook or extension point that allows self-hosting users to write and maintain a custom service. This would allow this category of users to self-serve their own feature requests without them having to maintain a fork or us having to take on the maintenance of code that would be zero use to the users of shields.io.

The thing that has made me think of this recently is Artifactory. I know this issue has come up before, but I can't think of any other examples to hand. So my first question is: Can anyone remember any other cases? It would be useful to collect up a few examples to help think about this. Also it would be useful to get some idea of whether there are enough use-cases to warrant solving the challenges this presents us with.

Here's some initial thoughts about what a plugin system might look like:

• We make a directory called local-services (for the sake of argument).
• In the repo, it is just an empty dir with a .gitkeep in it.
• When we bootstrap the server, as well as searching the services dir, the loader also searches the local-services dir.
• If any services are found in there, they are loaded. We mount them under a URL like /local to namespace them. That means there won't be clashes with other URLs we add to core in future.

That sounds kinda reasonable-ish, but it does bring up a number of problems, tradeoffs and questions:

• There are probably some people who self-host by running from source. However, my assumption is that most people who run their own instance use a docker image. What is a good pattern to recommend for allowing users to inject additional files into a docker image? I guess one option would be for users to write their own dockerfile extending shieldsio/shields and copying some extra files into it. Another might be to mount local-services as a volume and give it some files that way? Does anyone know of any examples of other projects that have this kind of setup? It would be interesting to look at some examples.
• As soon as you've got other people writing services, our internal API becomes a public API. For example, if we make a breaking change to BaseJsonService at the moment, its a bit of a pain but fundamentally we only have to account for that in our own codebase. Once you have self hosting users extending that class, a non-backwards compatible change has wider implications.
• In my proposed architecture, I've covered how we could discover and register custom service classes. However that is not a 100% solution. For example, I would expect that the subset of services that make sense for a user to develop as a custom plugin are going to be disproportionately the type of services that need auth or credentials. Doing that through the "official channels" would require the user to be able to do more than just register a custom service because they'd need to be able to add things to privateConfigSchema. Of course, that wouldn't stop someone directly calling process.env.MYVAR in the service code, but I feel like we could do better. What other obvious important things does my strawman implementation overlook?

[calebcartwright]

Asking more for the sake of spurring the discussion as opposed to leaning one way or the other, but what are the reasons why the Custom Endpoint model couldn't suffice?

[chris48s]

Yeah its an entirely valid question. If you want to use that approach, it requires you to run some kind of single endpoint microservice to sit between your shields install and the service you want it to talk to.

A plugin would allow your shields instance to talk directly to that service via its API.

Maybe that's not enough of a reason though.

[calebcartwright]

I think the other point of consideration would be the simplicity factor relative to config and authorization (much of which would have to be reimplemented within the Custom Endpoint model)

[PyvesB]

Can anyone remember any other cases?

None come to mind right now.

[chris48s]

Another though on this: As well as things like BaseJsonService effectively becoming a public API, if we did this, it would also make all the internal helper functions ( metric(), latest(), etc) a public API too.

That is quite a big downside.