External metrics tracking

[gabek]

The topic of externally tracking metrics have come up a few times. I wanted to open this issue as a place to discuss it.

1. What is the end goal?
2. What metrics are useful?
3. Who is the audience of this feature?

I've thought about the following as options:

1. Build dynamically loaded plugins using Go's Plugin feature. https://golang.org/pkg/plugin/. We create an interface that any metrics plugin has to implement and then we load it in. These plugins would work as an adapter between Owncast and a metrics provider and the plugin can do whatever it needs to do (such as integrating the Prometheus client libraries). We could offer some plugins ourselves for common uses as separate projects.
2. Support pushing metrics to an external API web service and/or create an endpoint where metrics can be polled.

[jmhobbs]

I know it's backlog, but if you have any desire to have this worked, I would love poking around at that plugin interface.

Or for the second option you could support a couple standard protocols, maybe statsd and something on mqtt? You could also expose the existing internal metrics collector over a Prometheus endpoint, which has a lot of momentum. Those could all be built as plugins for option 1 though too.

[gabek]

Thanks John! I have some thoughts around this, but mostly questions :)

I'd love to hear about what problems people are looking to solve before a true solution is built. I do like the idea of using statsd and mqtt, since there's large ecosystems around it. I think Prometheus is an obvious choice, but I'm not sure what of those would be most useful right now because it's not clear to me who's problem, and what their typical stack is, that we're solving for at the moment.

As for particulars of specific metrics, the hardware metrics that we're collecting is to help alert the user and help them troubleshoot things, but for external metrics they'd already have much more detailed hardware metrics so that's not terribly useful.

Maybe start with just one obvious item and go from there. And in this case the one thing I could think that's small and obvious is viewer counts over time. While that's not performance related, it is an important metric, and I'm assuming this is something anybody doing external monitoring would want.

I'd love to hear thoughts from anybody else who has particular use cases. I'm not opposed to a spike and a proof of concept just to see how things would work, but I'm hesitant to add a new feature for something that's not clearly addressing a use case.

[jmhobbs]

Fair enough!

You have a good point on the hardware. If they are setting up monitoring those things are often covered out of the box.

I may tinker around with the plugin system for kicks, then if this or anything else could leverage that, I’m happy to give a primer on it or do the work. But I won’t pursue this until there is a more defined problem.

Thanks!

[gabek]

I'd be super interested to hear about what you might find with plugins! The idea of plugins, in a more generic sense, is super interesting to me and I think could be leveraged in really cool ways. Auth, paywalls, and specific storage providers are all things that have come to mind, aside from instrumenting metrics, that I think could make a ton of sense as a plugin. Instantiating a plugin that opens up some new endpoints or works as some kind of middleware for specific functionality seems like a sweet solution for things.

[gabek]

When chatting with @muesli he shared some of the nitty gritty about Go Plugins, and that there may be more catches than we were aware of: https://owncast.rocket.chat/channel/general?msg=igmByfp9y5PJP6od4

1. You need an exactly matching Go compiler version for both the binary and the plugin
2. macOS support is still tagged as "experimental". No Windows support as mentioned
3. If you use the same deps in the binary & plugin, you must use the exact same version of that package
4. There's a bit of a GOPATH dependency still. This sucks when using Go modules
5. don't get me started on cgo issues with plugins 😄

He suggested for plugin type things we check out https://github.com/hashicorp/go-plugin.

[gabek]

I did the hello world of go-plugin last night. It works, but there's a lot of boilerplate. My guess is if you wanted people to build plugins for whatever interface you came up with you'd create a scaffolding that included the interface, all the boilerplate hidden away in its own file, and then a bare bones implementation that people can fill in. Otherwise it's not clear or easy. It actually makes a localhost server in the plugin that your host app connects to, and then talks back and forth over grpc. It's a lot :)

[gabek]

I was toying the idea of a plugin that simply is told when a live stream starts/stops. And then you can fill in what you want it to do. Post to Twitter, send an email, post to the Fediverse, send a Slack message, etc. Seems like the simplest plugin you can get. At some point I'll take a stab at it seriously.

[graywolf336]

Don't kill me for this, but wonder if we should do a simple JS/TS runtime and embed that to run plugins. Although how that could work, I am not sure. First thing that comes to mind is something like deno (https://github.com/denoland/deno) or after looking into js runtimes, there could be https://github.com/dop251/goja. Just a thought as I know things such as Zapier have their backend running via one language and then their scripting engine/service is JavaScript based.

[gabek]

It's not a crazy idea. To take it one step further, when looking at go-plugin, and saw how it's really just running a binary as an external process and then talking back and forth, I was thinking how really you could do a super simple version of that and run a script in any language, and just pass JSON back and forth through stdin/stdout.

[graywolf336]

The only "issue" I have with running a script in any language is that we wouldn't have a unified plugin system. Which would be confusing as an end user whenever they want to run a plugin and download it only to find out it is a ruby script and now they have to have the ruby runtime installed on their system. Then if we are running in docker, we would have to add a whole bunch of language runtimes to support the vast amount of languages out there.

My vote would be to pick one and stick with it. Or if we do want more than one, do at max two or three...although that sounds like a hassle to me.