Null values on streaming ingestion of proto2 messages

[layoaster]

Affected Version

28.0.1

Description

After upgrading from Druid 27 and keeping the SQl compatible mode on Druid 28 we've noticed that fields with Null values started popping up on our datasources filled with our streaming ingestion tasks (Kafka Supervisor) reading protobuff (proto2) messages.

Sample:

As you can see the field RecivedFromNodeId (string) has a Null value.

This field is defined at proto schema like an optional string:

  optional string receive_from_node_id = 7;

According to proto2 documentation, an optional string field will have a default value or, in its absence, a type-default value (empty string for Strings).

When a message is parsed, if the encoded message does not contain an optional element, accessing the corresponding field in the parsed object returns the default value for that field.

Regardless of the proto version, Null values are not allowed/supported in the Protobuff spec. Not set fields should always return the default value (proto2 only) or the type-specific default (proto2 and proto3).

So my question/issue is why is the protobuf-extension not honoring this behavior?

For more context, the proto messages are created/sent from a Golang backend.