Modify DNSTAP UDR behavior

[johnhtodd]

• Program: Recursor
• Issue type: Feature request

Short description

Modify the UDR behavior for DNSTAP (and possibly for PowerDNS protobuf format) such that only RRs that are "unique" (aka: newly seen RRs mapped to that qname/qtype) are transmitted. This might be a new flag, or it may just be modifying the existing behavior to match expectations.

Usecase

I would like to have a DNSTAP stream that contains only UDR data, meaning events would appear in the stream containing RRs that are newly seen or at least which do not match the Bloom filter on the recursor.

Current behavior is that any message that is a response that contains ANY RRs that are UDRs will transmit the full set of RRs, even those that are not UDRs, and distinguishing between the UDR and non-UDR RRs is impossible which defeats the purpose of the UDR stream being unique qname/qtype/RR sets.

Description

Discussions with Neil & Otto indicate that UDR flagging happens in the proprietary PowerDNS protobuf format. However, there is no such flag in DNSTAP, and we have standardized on DNSTAP for transport and ingestion of telemetry data. Neil suggested possibly adding an index in the "Extra:" field (in protobuf format? JSON?) that indicated which of the RRs was a UDR. I'm not quite as fond of that, because it seems to be contradictory to the concept of the UDR stream in general. Only RRs that are UDRs should be transmitted, in my view, since that is the whole point of the UDR stream - there is a completely separate flag that indicates "send only things that are UDRs" so this should not be a surprising behavior. For our purposes, we have no interest in the other RRs that are associated with a response if they've already been seen.

This may be very simple - just the addition of one or two boolean tests. If there are no RRs that are UDRs, then do not include them in the DNSTAP message. If there are no RRs that are UDRs ,then don't send any message at all (which I think is already the case.)

If it is seen as confusing, then perhaps an additional flag in the dnstapNODFrameStreamServer command could be added, like logUDRminimized=true with a default value of "false", which would prevent this being a breaking change. Comments were made that perhaps we were one of perhaps two users of this feature, so maybe this as a breaking change also wouldn't be traumatic. :-)

For consistency, maybe even this could be added to the proprietary logging command of outgoingProtobufServer as well. While there is already a flag built into the proprietary protobuf definition which makes identification of UDR RRs easier, it still does transmit them which may be considered wasteful when they are not meeting the definition of UDR data.

[omoerbeek]

I can see that having protobuf or dnstap messages that only contain UDR marked elevant record(s) is useful for a consumer, but on the recursor side it would require some extra work, especially in the dnstap case, where at the moment the dnstap message just contains a copy of the packet sent to the client. Only including the UDR marked RRs would require building a separate packet for dnstap purposes only, adding complexity and processing work.