Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JSON output format modification & compatibility with other tools #99

Open
pspacek opened this issue Dec 16, 2022 · 8 comments · May be fixed by #100
Open

JSON output format modification & compatibility with other tools #99

pspacek opened this issue Dec 16, 2022 · 8 comments · May be fixed by #100

Comments

@pspacek
Copy link
Collaborator

pspacek commented Dec 16, 2022

TL;DR

This is a vague proposal to work on modified JSON output format, with an ultimate goal - sharing the same output format with dnsperf and also DNS Shotgun.

Motivation

All three tools do essentially the same thing, send queries and statistically evaluate answers, but each tool does that in slightly different manner and fits slightly different use-case. At the same time, much of my data post-processing is in the end the same - I want to plot response rates, RCODEs, and latencies over time. Because each tool has a different format I have to switch parts of post-procesing pipeline to, in the end, get the same plots.

I think it would be common benefit if all three tools used the same JSON format and could reuse the same post-processing and plotting scripts.

What's going on in other projects

  • I'm planning to modify DNS Shotgun to use the new JSON format - and it already uses format with separate output per-generator, so the fundamental split is the same as in Flamethrower
  • I have a WIP version of dnsperf with "a" JSON output format - and plan to submit it to DNS-OARC/Jerry for review and merge after Christmas

So, I think that with buy-in on your end we can make the common format happen!

Discussion about format modification

This is not a firm proposal, take it as kickoff for discussion.

  1. It would seem cleaner/easier for third-party parsers if each output line/JSON object had a separate field like "msgtype" which would clearly indicate that it is a "header" (currently "cmdline"), "period stats" or "total".

  2. Somewhat related is number format and resolution. I would add header field with "timer units per second" information. That would define resolution used for time fields produced by particular implementation:

  • Flamethrower varies between 1 sec vs. ms in some fields - I would standardize on one of them, preferably ms to avoid dealing with floating point
  • DNS Shotgun uses ms = field value 1 000
  • dnsperf uses usec - field value = 1 000 000
  1. Deduplication
    I'm not sure why some field names and field values are duplicated... E.g.
  • Could we keep runid just once in header instead of repeating it everywhere?
  • Similarly, combining period_* fields with total_* fields in the very last message is confusing. Can it be split?
  • In a similar vein, if we have "message type" in header maybe we can remove the period_ and total_ prefixes?
  1. Header nits
  • Maybe cmdline could be an array of strings
  • I would add a separate version number for software used to generate the JSON and another for the JSON format itself

Wow, that is a long post. Sorry! If you made it here, what do you think about it?
@pspacek
Copy link
Collaborator Author

pspacek commented Dec 16, 2022

Oh, I forgot to mention that plotting scripts already exist. Basic example outputs:

These scripts can do much fancier output, like per-rcode charts:
all-response_rate-rcodes

or show aggregate results for "groups" of runs with different parameters etc., but I'm not going to post all the variants here.

Of course not all of chart types are applicable to Flamethrower today as it does not seem to track fine-grained latencies, but I think it does not preclude us from sharing format.

@weyrick
Copy link
Contributor

weyrick commented Dec 16, 2022

Thanks @pspacek, with just a quick look I think it's a great idea and I'd be happy to collaborate here. I'll review in more detail.

@pspacek
Copy link
Collaborator Author

pspacek commented Jan 16, 2023 •
edited

It turns out I'm not the first with the idea! I've just discovered Flent which, besides other things, offers interactive GUI for browsing & editing charts. I plan to have a look if there is a way to converge on something compatible across even larger ecosystem.

Edited to add link: Flent data format

@weyrick
Copy link
Contributor

weyrick commented Jan 17, 2023

@pspacek in terms of a common format that allows better interaction with other tools - in my observability project we have standardized on openmetrics and opentelemetry ... since this is time series data, I think it could make some sense. There are some existing libraries available, but in the end the OTLP format is protobuf, which I believe could be serialized to JSON. This would enable users to easily get the data into their observability stacks, since it supports many different types of "exporters".

@pspacek
Copy link
Collaborator Author

pspacek commented Jan 17, 2023

I'm familiar with openmetrics but it might make sense as well. So many options 🤯 😀

@pspacek
Copy link
Collaborator Author

pspacek commented Feb 15, 2023 •
edited

Proposal, all values are integers or strings. Time precision us only.

prettyjson.json.zip

@weyrick weyrick linked a pull request Feb 17, 2023 that will close this issue
json metrics changes #100
Draft
@nicki-krizek
Copy link
Contributor

Feedback on the proposed format:

  • merged means that the data has been added up from all senders/threads. I don't think this should be a property of the header, but rather the "data" entry (i.e. stats_periodic). It might also be a special reserved value for the threadid instead.
  • there should be some value representing bad/discarded packets -- whether due to network conditions, invalid input data, premature exit, or other error conditions that prevented the packet from being sent
  • latency buckets: consider whether their range needs to be specified in every data entry or if it's sufficient to specify them once in the header entry
  • add data for connection tracking, e.g. shotgun uses:
    • conn_active: number of active connections
    • conn_handshakes: number of handshakes performed
    • conn_handshakes_failed: number of failed handshakes
    • conn_resumed: number of connection resumed with TLS resumption
    • conn_quic_0rtt_loaded: number of connection for which 0RTT was accepted on a QUIC protocol level
  • shotgun also has a couple of fields for tracking success rate of 0RTT queries send over QUIC:
    • quic_0rtt_sent: number of requests for which 0RTT data was used over QUIC
    • quic_0rtt_answered: number of answers which were received for requests sent as 0RTT data over QUIC

@pspacek
Copy link
Collaborator Author

pspacek commented May 22, 2024

We have moved the unified format description into a separate repo: https://github.com/DNS-OARC/dns-metrics

@jelu jelu mentioned this issue May 27, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

json metrics changes DNS-OARC/flamethrower
3 participants
@weyrick @nicki-krizek @pspacek