Connecting multiple remote sites

[freol35241]

Hi,

I am currently evaluating if ecal is a suitable fit for our usecase and would really benefit from your input! The usecase is a research setup in the maritime domain where we plan for a cyberphysical system of systems simplified as:

• One (or several?) main site (i.e. LAN with very good internet connection)
• Several remote "sites/robots/entities" (separate LANs with 4G connections) where each remote site has its own set of sensors/actuators/control algorithms etc

Is it possible to configure ecal so that:

• A subset of the data topics from each remote site is tunneled to the main site? And also vice versa (i.e. from the main site to a remote site) (Manually configured based on available bandwidth)
• Monitoring, recording system management tools that are built-in to ecal are fully functional (specifically, the monitoring and the distributed recording are critical features).

What other tools/solutions (in addition to ecal) would you recommend for such a setup? VPN? SSH tunneling? Or maybe a separate middleware altogether for the data and vpn or ssh for only the built-in-features of ecal?

[FlorianReimold]

Hi @freol35241,

One (or several?) main site (i.e. LAN with very good internet connection)

This is the perfect scenario for eCAL. eCAL scales very when all you nodes (i.e. eCAL Programs) run

• either on the same physical machine
• or on another machine that is connected with a "good" ethernet connection.

the term "good" is flexible, as it depends on the actual throughput you want to achieve.

Several remote "sites/robots/entities" (separate LANs with 4G connections) where each remote site has its own set of sensors/actuators/control algorithms etc

I understand this as a setup, where you connect multiple clusters (those machines are connected via an ethernet switch) over internet. This over-the-internet-setup isn't exactly what eCAL was made for, even though we have tunnel-solutions for this purpose that work well.

The reason why a native eCAL approach will not scale well over the internet, is that eCAL makes heavy use of UDP Multicast (as that scales very well in local networks, even for many subscribers). UDP multicast means that each publisher sends UDP packages that are delivered to any machine that has joined this multicast group. When sending these packages out to the internet, they will just be dropped by some router, as delivering those packages to all machines on the entire internet would be devastating.
You can get around this issue by creating a VPN network, which can then also transport UDP multicast packages.

The other (and probably better) way is to use a proxy that transmits the data over a different protocol. We have the ecal-mqtt-bridge available for that purpose. MQTT may not be as performant as eCAL, but it has an entirely different architecture and can easily work over the internet. When using this, each of your clusters would have one machine running the ecal-mqtt-bridge. You would use the configuration file of the bridge to tell it about any topic that you want to be transferred over MQTT. The Bridge will then create an eCAL Subscriber on one end, send the data via MQTT and publish it to eCAL on the other end again.

Is it possible to configure ecal so that:

• A subset of the data topics from each remote site is tunneled to the main site? And also vice versa (i.e. from the main site to a remote site) (Manually configured based on available bandwidth)

As I mentioned, the mqtt-ecal-bridge can serve the purpose of transferring a specific set of eCAL topics over the internet.

Inside the main-site, eCAL will only transfer data over network, that is actually needed by some other remote node.
E.g. If you have 3 Topics A, B, C, but only Topic C has a remote subscriber, then only the data of C is sent via UDP / TCP. The other topics A, B may be subscribed locally (i.e. by a node running on the same host) though, but this will cause the data to be transferred via Shared memory.

This configuration is however implicit. eCAL will automatically decide which topics need to be transferred over network. There currently is no way to configure that with an explicit configuration technique.

Is it possible to configure ecal so that:

• Monitoring, recording system management tools that are built-in to ecal are fully functional (specifically, the monitoring and the distributed recording are critical features).

Even if you are using a tunneling solution like mqtt-ecal-bridge, the eCAL Monitor will continue to work. But of course it will only see the remote topics that you configured the mqtt-ecal-bridge to transfer. You will always be able to see all topics that are published by any node in the entire eCAL network though (but not all that are published in another remote eCAL network that is just connected via MQTT).

Something that is currently not being transferred via MQTT are the eCAL Services. The eCAL Services are for instance used to tell a recorder client that is shall start a distributed recording. Of course you can still create a distributed recording by just triggering the eCAL Recorder at roughly the same time on the main and all remote-sites, but out recorder GUI will not be able to help you with that.

Distributed recordings within one of your sites will obviously be fully functional.

You may need to know, that additional to the actual data transport, every eCAL node periodically sends metadata information about itself and the topic it publishes and subscribes. That information is always transferred via UDP and therefore always visible by all other eCAL nodes in the network. So the eCAL Monitor will be able to tell you information about all topics, even if they are not published on your current host. If you open the inspection window of eCAL Mon however, to inspect the data content, eCAL will obviously start transferring the data of that remote topic over network, as well.
If you are creating a distributed recording, the topics are subscribed locally, by local recorder clients, so the data doesn't have to be transferred to a remote machine while creating a recording.

[freol35241]

Thank you very much for your prompt and detailed reply @FlorianReimold !

Yes, mqtt is certainly a possible option (thanks for the link) and something that we have prior knowledge of. I hope that it is ok that i ask some follow-up questions:

Something that is currently not being transferred via MQTT are the eCAL Services.

Am I correct to assume that there are no actual (known) technical barriers to this, only the usual: time, need, interest? I guess the Request/Response pattern introduced with mqttv5 should be a good fit.

Of course you can still create a distributed recording by just triggering the eCAL Recorder at roughly the same time on the main and all remote-sites, but out recorder GUI will not be able to help you with that.

My understanding of the lines above is that even though recordings on different hosts are started and stopped manually they can be merged to a single eCAL measurement and be replayed as a single measurement in correct order according to their timestamps (assuming all hosts have correctly synced clocks). Is that correctly understood? How would that be done in practice? I couldnt find any documentation about this specifically.

Nvm, just found this.

every eCAL node periodically sends metadata information about itself and the topic it publishes and subscribes. That information is always transferred via UDP and therefore always visible by all other eCAL nodes in the network.

Are there any show-stoppers for also bridge these metadata broadcasts over mqtt so that another "cluster" may know what topics and data types are available through the eCAL Monitor (even though it will only be possible to subscribe to those data topics that are also being bridged). I would expect retained payloads to be useful here (in the mqtt world).

[FlorianReimold]

Thank you very much for your prompt and detailed reply @FlorianReimold !

You are welcome 😊

Am I correct to assume that there are no actual (known) technical barriers to this, only the usual: time, need, interest? I guess the Request/Response pattern introduced with mqttv5 should be a good fit.

No, there is no technical reason. MQTT 5 just didn't exist when we implemented ecal-to-mqtt. This looks like a perfect fit and I would be happy to see this implemented. We use libmosquitto for ecal-to-mqtt, which also supports MQTT 5.0 according to their website. So basically everything is ready for that feature except ecal-to-mqtt. Contributions are welcome, as we are probably not getting around implementing it ourselves any time soon.

Are there any show-stoppers for also bridge these metadata broadcasts over mqtt so that another "cluster" may know what topics and data types are available through the eCAL Monitor (even though it will only be possible to subscribe to those data topics that are also being bridged). I would expect retained payloads to be useful here (in the mqtt world).

The metadata is bridged, but only for topics that you configured to be bridged. You could configure all topics to bridged, but that would cause a massive data traffic, as ecal-mqtt-bridge will always bridge the entire payload traffic, even if there is no subscriber on the other side. So there is definitively some room for improvement, the line of code is here:
https://github.com/eclipse-ecal/ecal-mqtt-bridge/blob/a377003d2a83f8c693b1284378b79070e001254a/src/Bridge.cpp#LL795C7-L795C7