Latching topics ?

[ahoarau]

Hello,

Does ecal has the same "latching" topic mecanism implemented, as in ROS ?
I can't find the answer in the docs or code.
Thanks !

[FlorianReimold]

Hi Antoine,

I guess not, but maybe you can elaborate what the term latching topic refers to, so we can tell you whether there is a replacement! :)

Best Regards
Florian

[ahoarau]

Arf, too bad. Any chance this could be a feature request ? Is it aligned with current design decisions ?

That being said, having a callback called whenever a new subscriber appears could do the trick. I need to see it in action to understand if this is actually practical.
Sending periodically could work, but loads the network for nothing, and prevent the nice pattern "publish a message on event changes".

[FlorianReimold]

I don't want to say no to ever implementing something like that, but at the moment I honestly don't see enough need for this feature to implement it myself. Did I understand you correctly, that this is a publisher feature? So if you have 10 publishers, each having the latching=true, each new subscriber will be flooded by 10 messages, right?

[ahoarau]

Yes. It should of course be an option, in some cases you don't want to have the topic latching.

[KerstinKeller]

This feature that you're asking for would bring alot of additional logic with it on the eCAL side.
Right now eCAL is implemented as "fire and forget". There are (usually) no buffers, no queues, no nothing. Subscribers get signaled when new data is there.
To implement such a latching feature, publishers suddenly need to buffer data. They need to keep track who has received what data.
On some data layers it might be possible (on shared memory a publisher can signal the respective subscribers), but on others, such as UDP multicast, such a feature would be really hard to implement. Sending something again would mean that everyone receives that datum again, which is probably also not intended behavior.

I don't know if it would be possible to have "Latching" Publishers, that communicate only on a certain eCAL Transport Layer, e.g. TCP, where there is actually an underlying 1:1 connection.
Also for me it would be conceptually different from a publisher. We would have to evaluate if there is a benefit for a wider user base.

[ahoarau]

Definitely not expecting to see it on the UDP layer.

An option only available on the TCP layer is perfectly fine.

[rex-schilasky]

This sample is demonstrating the Events API for a publisher.

[ahoarau]

@FlorianReimold any updates on this ?

[FlorianReimold]

Sorry to disappoint you, but latching topics are not on our TODO list, at the moment. As I understand it, to have such a feature, eCAL would have to:

• Store the latest data on the publisher side in order to re-transmit it
• use 1:1 connections in order to only transmit it to new nodes
• Use a reliable transport protocol, as subscribers would probably rely on receiving that data, as the entire reason for this feature probably is to not periodically send out frames

While the TCP layer should in theory be able to satisfy those requirements, any other layer is not and it would require a lot of effort to implement such a feature in eCAL.

What you currently can do is use the Event API in your publisher and then simply re-sending the latest data. (See the post by rex-shilasky above for an example). Honestly I think such a user-space workaround should be fine and would be rather easy to implement. Have you tried it?

[GJYang]

When communicating via shared memory, may be able to use the shared memory as the buffer without additional buffering logic.

I modified the following two files(just for test) and possible to subscribe data like latched

1. For latched publisher(only shared memory tlayer), need to publish even without subscriber
   comment if condition
   https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/src/pubsub/ecal_publisher.cpp#L368
2. if published topic is latched then try read it when a subscriber created
   bool has_unprocessed_data = true;
   https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/src/io/shm/ecal_memfile_pool.cpp#L155

but, for the publisher option posted by @ahoarau in #774 (reply in thread)_ need to modify more ...

[KerstinKeller]

The next few days we'll discuss eCAL roadmap for the upcoming year, and to be honest, I am not a big fan of the feature (even though the implementation you suggested is quite elegant, because it requires little changes), because in case of other transport layers it will require more effort.

So we would have to understand more clearly the requirements of this, especially the "why". So why should we implement it.
And then we need to understand the "how". E.g. who decides about latching? Publisher for sending, subscriber for receiving, or both? or publisher always sends "latched", but subscriber decides if they want to possibly receive old data.

Also, should this be reflected in the user callback then? does the user need to know if the first data that is being received is old or not?

So - potentially a lot of questions 😄

[ahoarau]

The next few days we'll discuss eCAL roadmap for the upcoming year, and to be honest, I am not a big fan of the feature (even though the implementation you suggested is quite elegant, because it requires little changes), because in case of other transport layers it will require more effort.

So we would have to understand more clearly the requirements of this, especially the "why". So why should we implement it.
And then we need to understand the "how". E.g. who decides about latching? Publisher for sending, subscriber for receiving, or both? or publisher always sends "latched", but subscriber decides if they want to possibly receive old data.

Also, should this be reflected in the user callback then? does the user need to know if the first data that is being received is old or not?

So - potentially a lot of questions 😄

The question was initially posted as I was searching for a cross-platform alternative to ROS. At the time ecal had TCP support, necessary for not losing messages, but was indeed lacking latching topics. Use case is pretty simple: initialisation of nodes.

A new node might arrive "late" in the system and needs for example the robot_state message to be initialized, or some parameters. For robot_state, is it usually streamed, so you won't have to wait very long to get a message, but for event-based topics, it's a way of getting the latest message without having the server to issue an "empty update" for everybody.
This also has the benefit of not coding extra intialization logic (as suggested) to manually get the latest info.

I understand this can be worked-around without too much effort. Latching topics simply offers a bit for more control over the dataflow.

For your extra questions, my guess would be to copy the ROS behavior (= server decides, and it's completely transparent for client). That being said, having the info that the message is latched or not is very interesting 👍