File abnormally large with C++ API

[yun-li]

I have written some MCAP file with C++ API. The written file is 8.6MB with the following mcap info

library:   libmcap 1.3.0
profile:   protobuf
messages:  1
duration:  0s
start:     2024-01-12T11:32:32.748535-08:00 (1705087952.748535000)
end:       2024-01-12T11:32:32.748535-08:00 (1705087952.748535000)
compression:
	: [1/1 chunks] [303.00 B/303.00 B (0.00%)]
channels:
	(1) rule_uuids                      1 msgs   : data_logger.McapRuleIds [protobuf]
	(2) left_side_camera_frame_signal   0 msgs   : moz.msg.ChunkedMessage [protobuf]
	(3) right_side_camera_frame_signal  0 msgs   : moz.msg.ChunkedMessage [protobuf]
	(4) blinker_state_signal            0 msgs   : moz.msg.BlinkerState [protobuf]
attachments: 0
metadata: 0

After compression with mcap compress the info is

library:   mcap go v1.1.0; libmcap 1.3.0
profile:   protobuf
messages:  1
duration:  0s
start:     2024-01-12T11:32:32.748535-08:00 (1705087952.748535000)
end:       2024-01-12T11:32:32.748535-08:00 (1705087952.748535000)
compression:
	zstd: [1/1 chunks] [303.00 B/238.00 B (21.45%)]
channels:
	(1) rule_uuids  1 msgs   : data_logger.McapRuleIds [protobuf]
attachments: 0
metadata: 0

The size changed from 8.6MB to 924B, which is huge change. I did not write a lot of things in my code but the file size is too much. I hope to write the smaller size to begin with, did I made some configuration wrong?

My example code is as follow:

mcap::McapWriter writer_;
writer_.open(file_name, mcap::McapWriterOptions("protobuf"));

I did not change any other McapWriterOptions. What did I do wrong? Thanks!

The files are as follow:
Original File
Compressed File

[defunctzombie]

You didn't do anything wrong.

mcap compress saw that the left_side_camera_frame_signal, right_side_camera_frame_signal, and blinker_state_signal channels had no messages and removed them from the resulting file. I guess the removal of those schemas is what brought the size down. Compression of the remaining channel brought it down further.

[yun-li]

Thanks for your reply.

The rule_uuids is just some strings and should take only several bytes, and there is not a lot of room for compression.
Would the schema take several MBs? I do not think is very possible.

I have another file with left_side_camera_frame_signal, before compression, it is 6MB and after compression it becomes 70KB, the the schema should not be greater than several KB if that is the case.

Would there be other reasons where I may write "ghost" data? I saw there is a default chunk size set to 768K, would increasing that number help in compression? Do I need to explicitly set some configure values?

Thanks very much!

[defunctzombie]

Would the schema take several MBs?

It is possible. You have some large schemas. Try running mcap list schemas /path/to/file.mcap and you'll see there is a lot of schema data.

You are not writing ghost data - you have a lot of schema data overhead. As for the other file with camera frame data - maybe the camera data compressed well and that's why the file got so much smaller.

[yun-li]

Will the same schema getting compressed differently in different MCAP files?

I have a mcap file with right_side_camera_frame_signal message, the original size is ~6MB, and the compressed one is 43K (schema and message). If the schema takes large space(>1MB), the compressed one should still be huge, but it seems it is not the case (see attached file).

The left_side_camera_frame_signal and right_side_camera_frame_signal are of the same size so that it seems that it is not only the schema that takes MB of spaces.

right_signal.mcap
right_signal_compress.mcap