Support for Wildcards/Topic-lists for Subscribers

[alex-gaudreault]

I'm wondering if there are any plans of adding wildcard (or topic list) support for subscribers (think: MQTT)? This feature would greatly simplify the implementation of a multi-topic subscriber, and enable the use of a hierarchical topic structure.

Currently, the developer can achieve this through:

1. A monitor & relay process: A process which monitors for new publishers with a certain sub-string in their topic name, subscribes to these topics, and then re-transmits the messages to a different "global" topic. Ex. Monitoring for my-topic-*, and relaying messages to my-topic-global. This involves an additional process & acts as a single POF for the global channel.
2. Publishing to a process-specific topic + a global topic: Publishers must publish to both their process-specific topic (my-topic-001), as well as to the global topic (my-topic-global). This is quite inefficient since it requires two transmissions from each publisher.
3. Monitoring & wildcard matching for each and every subscriber: Each subscriber will act as a monitor, and subscribe to the topics matching a certain wildcard expression. This would result in an ungodly amount of threads on any given system, assuming we were to attach the same callback for all subscribers. (Although, you could wrap multiple subscribers into a single object & do a round-robin polling of each of these).

For most developers, options 2 & 3 are out of the question since they are a very inefficient use of an otherwise very efficient technology. That leaves us with option 1, but then our communication network becomes centralized to an extent...

[KerstinKeller]

Hi Alex,
this issue has come up before, but we previously decided against it.
In particular, I would be interested if you want to combine topics of different types, or only those of the same type?
On a binary level that might not be a problem, but when interpreting the data, the subscriber has to take care of interpreting the data with respect to topic type.

That said, I am wondering about your specific usecase.
If you do have the need to subscribe based on a wildcard, maybe you are drawing the topic line at the wrong level?
E.g. you send bigger messages, and subscribers are just reading the part of the message that is of interest to them?
(E.g. this is especially efficient with message formats that have fast access and no deserialization step, e.g. CapnProto or Flatbuffers. Google Protobuf is not so well suited because of the required deserialization)

[alex-gaudreault]

Hi Kerstin, sorry for the delayed response - I've been quite busy!

My current problem requires multi-cast, bus-style and exclusive-pair styles of communications.

Summary

The project involves the development of a complex, high volume sensor-data processing network. There are 4 input data streams, each of which has 10 individual channels through which the incoming data can be processed. Each of our processing channels must be independently configurable, while also having the ability to be grouped together into cells. A single cell is controlled as a single logical unit (i.e. multi-cast configurations to all channels in a cell).

There is also the requirement that each and every channel reports their status to the system status monitoring component (bus-style or broadcast-style communication).

My Predicament

I guess my problem isn't necessarily the lack of support for "wilcard" topics, but instead revolves around eCAL's threading model...

• Because each channel needs to be capable of processing >100 Million points per second, the communications must remain very lightweight as to not contend with the processing threads. This includes minimizing the number of threads running parallel to the main processing thread.
• Because the configuration of each channel is done on-demand, and due to project requirements, the communications must be very fast (less than ~2 milliseconds) as to reduce the delay between reception & the application of channel configurations.

I guess my thinking behind having a "wildcard" subscription mechanism, was actually intended to try and minimize the number of communications threads. I guess a better solution would involve the ability to use a single reading thread for multiple topics, while not losing the ability to use asynchronous callbacks. This mechanism would involve IO multiplexing using a single thread, thinking similar to ZeroMQ's polling mechanisms.

I can achieve my desired functionality by writing my own polling class, which will handle the polling of multiple subscribers using a round-robin strategy. Regardless, I think this type of feature would be well situated as part of the eCAL framework. (A bonus would be the ability to introduce a bias to the polling frequency of certain topics in a set - essentially favouring the latency of a given topic, over that of the other topics in the same topic-set 😃 )

TL;DR

My issue stems from needing to minimize the number of communication threads. A wildcard style of subscription is not necessarily the best solution for this, but it was the first thing I thought of - Instead, allowing for multiple topics for a single subscriber would be better suited (think sort of like ZeroMQ's poller mechanism, which allows I/O multiplexing with a single thread).

[KerstinKeller]

Hi @alex-gaudreault,
I don't think eCAL is cut out to be what you are looking for.

There are 4 input data streams,

eCAL doesn't work stream based, eCAL deliberately works message based. Our focus is also high volume sensor data, but under the assumption, that at timestamp t, we capture messages x, y and z, and transfer them separately to everyone who is interested in either x, y or z.
On an SHM implementation level that implies, that there will be 3 different memory files, each large enough to hold one message each. With synchronization mechanisms also briefly described here, recipients are notified when new data is available, and when it is their turn to process the available data.

But at the moment (and we know it's suboptimal, it might change in future versions), it means 1 thread per memory file, and 1 memory file per subscriber (per publisher). (E.g. when there are two publishers on the same topic, they will write into separate memory files, and a subscriber to that topic will read from two different memory files, and will have two separate threads to watch for incoming data).

On with UDP as transport layer, things work a bit differently, but essentially you will have the same problem.

However, from what you describe, it sound more like you want a low latency multi process queue (N to 1?), from which you can poll data as it comes in.
Speaking about 100 Million points per second, this sounds like you need to push them out (maybe grouped, maybe not 1 by 1), but this also sounds more like streaming than message based transport.
Maybe like this git concurrentqueue git repo, but not multithreaded but over shm.

I'm sorry, but I am afraid eCAL might not be the right choice for your usecase, unless you use binary pub/sub on a single publisher / subscriber and build a whole lot of logic on top of that.