Feature Roadmap

[dgduncan]

I do not believe there has been a decided upon place for such discussions; however, I wanted to create a space to discuss and then we can move or change however we see fit. From monitoring discussions, issues, and talking with some people I know who currently use the project, there are a few features that stick out; server side subscribing, basic file based configuration, simple clustering.

There have been multiple issues and discussions opened about server side subscribing in a pattern close to the current server.Publish where the want is to be able to have an embedded subscriber that gets routed messages in a similar pattern as if they had subscribed via the standard mqtt way with a privileged account. Some are happy to utilize hooks to accomplish this, while others are looking for that simplicity; however, the desire remains.

Some folks I have talked to are using the server in a "non embedded" way and simply utilizing it straight from a Dockerfile with minimal configurations. The expressed interest is to have some sort of way in which say a .yml or .json file is used to configure the basic capabilities of the server without having to go in and adjust that in code.

This last one is a bit tricky as clustering is no easy task; however, some have wanted to utilize the server in place of something like EMQX or the like where the decision for choice is based on the real, or perceived, benefits of running multi-node for reliability and performance.

Just wanted to surface these things to see where everyone else stands on them and to give their opinions!

[mochi-co]

Really good idea for a discussion @dgduncan. Some thoughts:

• For server-side subscriptions, I do feel the hooks solution is a bit of a hack and that we should be providing an direct method for subscribing. It's something I've wanted so many times in the past, but for some reason I never got it implemented. The ideal solution would probably involve inline clients and a convenience method which abstracts this all away from the developer. Channels are likely the answer, but they can be a big source of performance problems if we get it wrong.
• We definitely need improved docker container and file based config support. I'd love to see this with yaml/json support for hook configs, even (auth comes to mind). But I agree it would be more professional to provide this.
• Cluster is a priority and it's been on my mind for a while. When I get some free time I will try to do some more research into the options. I suspect it can be divided into two main topics - service discovery, and data transmission. For service discovery, there are lightweight options like https://github.com/hashicorp/mdns which could work. MQTTv5 provides features for data retransmission - if done with some care, I think this can be implemented in a fairly minimal way. I still believe it's possible to do using hooks/adding more hooks, but I am open to implementing it into the core repo if it doesn't pollute the existing codebase too much (not everyone wants clustering and it makes tests more complex).

I've added a couple of new teams for @mochi-mqtt/telemetry and @mochi-mqtt/cluster - people should request to join them if they have an interest, and that will help us figure out where to go from there. Interested to hear everyone else's thoughts on this also.

[bkupidura]

@werbenhu building distributed (geographically) cluster is tricky and hard. Not sure if we should even consider clusters over WAN.
Maybe we should add some kind of 'mirror'/'bridge' feature which will allow to forward local messages to external instance. But i would try not to mix local HA cluster with remote instance.

[dgduncan]

I know we can't get this type of information, but I would love to know the most common use cases and environments in which the broker gets deployed. I personally deploy the broker in a cloud environment, essentially making a mdns solution useless to me. I would also love to know if people have more of an interest in a strongly/weakly consistent solution.

I personally like the idea of starting small with a mirror/bridge feature as mentioned above by @bkupidura if we choose to move forward at all.

[bkupidura]

Im also against mdns as its limited to L2 domain. I can image 3 local broker instances, running in different L2 network communicating over L3 (ip). In this scenario, we would need to deploy mdns-proxy/repeater/forwarder.
But mdns is only used to discover cluster members, further communication should take place over IP.
So we can do regular DNS discovery/static discovery (list of members in a config) and mdns. :)

[zippetto]

@dgduncan about the common use cases and following the current enterprise tech trend, I think it's important to support Cloud environments especially with Kubernetes-based orchestrators. In this case mDNS is not generally applicable due to the limitations in virtual networks of some cloud provider.

[martin31821]

It would be awesome to provide pluggable service discovery, e.g. Kubernetes headless services (basically a DNS name that returns all pod names), possibly raft/zookeper for people that want to embed it, or maybe even mDNS for local usage.

[IshanDaga]

Thanks @dgduncan for kicking this discussion off !
Adding my two cents here :

On the server.Subscribe feature

• Coming from a pure "mqtt-broker" PoV, I think it would be good to add an example as a "best practice" on how to do this using the current Hooks system, which imho would not be that complicated.
• I feel allowing a broker to "subscribe" to mqtt topics is a bit of an anti-pattern
• alternatively, one solution could be to embed a mqtt client along side the broker (enabled via a config option) that allow to publish / subscribe to topics and messages the usual way with the paho client.

Standalone Server Config

• this is awesome! no notes here :)

Distributed Broker

• @wind-c and @bkupidura had made efforts on this in Cluster MQTT broker based on mochi-co/mqtt #75 should this continue to be tracked there ?

Telemetry / OTEL / Logging

• Also requesting to be added to the @mochi-mqtt/telemetry team :)
• Since the move to slog is complete, using an external pkg such as this would be trivial to implement logging with OTEL, but we would still need to do something about having a single trace across multiple hooks.
• I have added my current approach of modifying the incoming packet to have the trace-id, and removing when sending to a client in OpenTelemetry Traces & Logging #227 (which while a terrible hack, works well).
• The best alternative here would be to have a new context be part of every packet, unsure of how expensive this would be and what the performance impact looks like for the same.

[IshanDaga]

1. Makes sense @dgduncan and @mochi-co, I agree that there is a significant interest in having this feature, so we might as well support it
2. If adding a context object to every packet doesn't add too much of a performance issue, then that would be the ideal case. Everything OTEL for go passes around traces, spans, child spans, metrics and SpanContext by using go's native context package.

The way I am looking at it, is primarily to add support for OpenTelemetry, via which, metrics, logs, traces can be exported to Prometheus/SigNoz/Uptrace or other similar solutions. OpenTelemetry would require us to add instrument in code, and we could do this using OLTP. OLTP also allows us to wrap / replace the global logger, allowing logs, errors, spans, traces and metrics through a similar interface.
EG:

ctxWithTrace, span := trace.StartSpan(parentCtx) 
defer span.End()
otelslog.L().InfofContext(ctxWithTrace, "this is a log belonging to the current trace and within this span)
otelslog.L().Infof("this is a log not belonging to any span / trace, but is still exported by OLTP")

I'll try to open a PR with OTEL using context over the weekend to better showcase this.

Using context tied to a packet would also allow us to pass other data between hooks if required in a very go-fashion

[IshanDaga]

-> missed note here: we would be able to replace the global slog instance, and that would allow folks to use the in-hook slog object to log things as needed, with or without context

[mochi-co]

Everything OTEL for go passes around traces, spans, child spans, metrics and SpanContext by using go's native context package.

I wasn't aware of this - it makes context the correct solution in my mind as we don't want to reinvent the wheel.

[mochi-co]

Using context tied to a packet would also allow us to pass other data between hooks if required in a very go-fashion

Definitely a good benefit.

[IshanDaga]

Taking this up since we've merged #245

[dgduncan]

If no one else wants it, I think I may take a crack at a simple MVP for file based configuration. At least something simple that can configure the basic capabilities of the server with a .yml . And if that works, moving forward with things such as auth and the like.

[mochi-co]

Sounds good to me! I wonder if we need a Deployment team to cover things like this. Maybe I am being over enthusiastic with creating teams :)

[SimonMacIntyre]

Very interested in this one! I am sort of doing this manually now locally by cloning the repo into my dockerfile, overwriting the main.go and dropping in my hooks and certificates. I'd be happy to help test this feature!

[dgduncan]

@SimonMacIntyre @mochi-co @IshanDaga @torntrousers
I opened up a new discussion thread to discuss file based config here for anyone interested.

[dgduncan]

Made a new comment and push for anyone interested here.

[torntrousers]

I am already using the server.Publish so can see a server.Subscribe could be pretty handy, though it seems a lot harder to do than the server.Publish.

Would be really interested in the clustering from a High Avalaibilty persepective, but don't have the bandwidth to help with that at the moment sorry so will mainly be an interested observer and happy to try things out as it progresses.

Its nice to see there is so much activity in the project!