How to write proto messages with HDF5Meas Class?

[oskr1709]

Hello,

I´m trying to write an HDF5 file with protobuf messages using class HDF5Meas

std::string channel_name = "osi3_SensorView";
std::string channel_type = "proto:osi3.SensorView";
eCAL::eh5::HDF5Meas HDF5Meas(path, eCAL::eh5::CREATE);
HDF5Meas.SetFileBaseName("test");
HDF5Meas.SetChannelType(channel_name, channel_type);
HDF5Meas.SetChannelDescription(channel_name, description);

std::cout << "Has Channel?: " << HDF5Meas.HasChannel(channel_name) << std::endl;//returning flase

std::set<std::string> channelnames;
channelnames = HDF5Meas.GetChannelNames();//empty

for (auto const &channel : channelnames) {//no entering this cycle
	std::cout << "Channel: " << channel << std::endl;
	description = HDF5Meas.GetChannelDescription(channel);
	std::cout << "Channel type: " << HDF5Meas.GetChannelType(channel) << std::endl;
	std::cout << "Description: " << description << std::endl;
	std::cout << "Has Channel?: " << HDF5Meas.HasChannel(channel_name) << std::endl;
}

osi3::SensorView currentOut;
...
HDF5Meas.AddEntryToFile(&currentOut, currentOut.ByteSize() , send_time, receive_time, "osi3_SensorView", id, clock);

For the description, currently I´m assigning the description for another channel description that I want to reproduce.

The instruction HDF5Meas.HasChannel(channel_name) is returning false in all process and the line HDF5Meas.GetChannelNames(); is empty. Additionally, with the HDF5Meas.AddEntryToFile is giving me an "memcpy.asm not found" exception.

Previously, I was using the API IMeasurement but I need to custom the maximum size of file and declaring the object as global to later define the file name so this approach is not the one I require.

What am I not considering or is there any example to write/read hdf5 files with HDF5Meas class ? I only found the class documentation

[KerstinKeller]

Hi @oskr1709 ,

it is highly recommended to use the IMeasurement API, because it does everything what is necessary to properly write the data.
The HDF5Meas is really easy to use incorrectly, and we don't recommend it.
What exactly are you missing from IMeasurement API?

Previously, I was using the API IMeasurement but I need to custom the maximum size of file and declaring the object as global to later define the file name so this approach is not the one I require.

About the Maximum file size: we can probably modify the API to also take options about the created files, e.g. about their size.
About the filename: the filename is totally necessary to have before you start writing, because, well the file needs to be created before you can write to a measurement.

[oskr1709]

Hello @KerstinKeller , thank you very much for your answer.

I keep trying to integrate the API to my code.

If API can take options like the size, it would be great and I understand that it is necessary a filename before write a measurement, but what I mean is that if it is possible to instantiate an eCAL::measurement::OMeasurementobject without the filename and with another method or other way define the filename in order that the OMeasurement object could be a global variable but assign a name in some scope different from the one when I write the measurement.

Additionally, I´m facing an issue when I use #include <ecal/measurement/omeasurement.h> , for some reason in runtime it crashes and the error is undefined symbol: _ZN4eCAL3eh58HDF5Meas15SetFileBaseNameERKSs , then if i set:

eCAL::measurement::OMeasurement meas(".");
eCAL::measurement::OChannel<osi3::SensorView> sensorview_channel = meas.Create<osi3::SensorView>("osi3_SensorView");

in some part of my code, the error changes to: undefined symbol: _ZNK6google8protobuf7Message11GetTypeNameEv

Basically in my CMakeLists.txt I´m including

find_package(Protobuf REQUIRED)
find_package(eCAL 5 REQUIRED)

and later:
target_link_libraries(${PROJECT_NAME} eCAL::core eCAL::hdf5 eCAL::message)

In Windows and some individual scripts in Linux are working properly just with this changes, but when I´m integrating in some bigger code, this errors appear. So, I can say that my installation is correct but somehow it is not finding the .so but I don´t know if I need to configure something else or is linking with another libraries

[FlorianReimold]

What you are getting there are linker errors. So the source files can be compiled successfully, but the linker fails to resolve the symbols. One reason could be that are compiling against a different version of a library than your linker tries to link against.

If the issue vanishes when you don't integrate in your big project, then something in the configuration of that big project or your system is crooked. I know, linker errors are a nightmare to debug, this is awfully complicated and the error messages are usually not very helpful. That's also why I cannot tell you at all where your problem is. But I would tell you, if I knew 😉

Btw.: Here is also some example from a unit test on how to use the ecalhdf5 API directly, without using the eCAL::measurement abstraction. It looks uglier, does the same thing, but maybe has some more options.

ecal/testing/contrib/ecalhdf5/hdf5_test/src/hdf5_test.cpp

Lines 105 to 148 in 4ead87d

	eCAL::eh5::HDF5Meas hdf5_writer;
	if (hdf5_writer.Open(meas_root_dir, eCAL::eh5::eAccessType::CREATE))
	{
	hdf5_writer.SetFileBaseName(base_name);
	hdf5_writer.SetMaxSizePerFile(max_size_per_file);
	}
	else
	{
	FAIL() << "Failed to open HDF5 Writer";
	}
	EXPECT_TRUE(hdf5_writer.AddEntryToFile(
	t1_data.data(), // data
	t1_data.size(), // data size
	t1_snd_timestamp, // snd_timestamp
	t1_rcv_timestamp, // rcv_timestamp
	t1_name, // channel name
	t1_id, // id
	t1_clock // clock
	));
	EXPECT_TRUE(hdf5_writer.AddEntryToFile(
	t2_data.data(), // data
	t2_data.size(), // data size
	t2_snd_timestamp, // snd_timestamp
	t2_rcv_timestamp, // rcv_timestamp
	t2_name, // channel name
	t2_id, // id
	t2_clock // clock
	));
	EXPECT_TRUE(hdf5_writer.AddEntryToFile(
	t3_data.data(), // data
	t3_data.size(), // data size
	t3_snd_timestamp, // snd_timestamp
	t3_rcv_timestamp, // rcv_timestamp
	t3_name, // channel name
	t3_id, // id
	t3_clock // clock
	));
	EXPECT_TRUE(hdf5_writer.Close());

[oskr1709]

Thank you very much Florian for your explanation. I was able to write a simple hdf5 with protobuf and the ecalhdf5 API directly, but when I try to save more information and then try to visualize it with ecal monitor or reading directly, I'm getting an error to parse the data. The code is pretty much this:

#include <ecalhdf5/eh5_meas.h>
#include <osi_sensorview.pb.h>

std::string channel_name = "osi3_SensorView";
std::string channel_type = "proto:osi3.SensorView";
eCAL::eh5::HDF5Meas HDF5Meas(path, eCAL::eh5::CREATE);
HDF5Meas.SetFileBaseName("test");
HDF5Meas.SetChannelType(channel_name, channel_type);
HDF5Meas.SetChannelDescription(channel_name, description);
osi3::SensorView currentOut;
...
long long time = 0
std::string* sensorview_out_current_buffer;
sensorview_out_current_buffer = new std::string();
currentOut.SerializeToString(sensorview_out_current_buffer);
hdf5_writer.AddEntryToFile(sensorview_out_current_buffer, sensorview_out_current_buffer->size(), time, time, channel_name, time, time);
delete sensorview_out_current_buffer;

If I save just one or two fields of the sensorview proto file I got no problem, but when I try to save more information, the error appears.

In addition, I have one question. To set the channel description, in this moment, I´m obtaining it from another description that I read, but I think it can obtained directly from the protobuf file but not sure how

[FlorianReimold]

eCAL Mon uses the "descriptor" string to visualize the data (at least when we are talking about protobuf and the generic protobuf plugin). The Descriptor is not a human readable description, but a machine readable schema information. This descriptor string can be obtained by the libprotobuf API.

I think you are maybe setting a wrong descriptor string, so the eCAL Monitor is unable to use it for dynamic deserialization. Here is how you could obtain that:

ecal/contrib/message/include/ecal/msg/proto/message.h

Line 47 in 4ead87d

return eCAL::protobuf::GetProtoMessageDescription(message);

Another cause for this message would be that your binary message data is actually corrupt.