Add mcap du command
#1021
Add mcap du command
#1021
Conversation
defunctzombie
force-pushed
the
roman/add-du-command
branch
from
0850810
to
bf25d21
Compare
| @@ -0,0 +1,19 @@ | |||
| { | |||
| // total message size by topic name | ||
| topicMessageSize map[string]uint64 | ||
|
|
||
| totalSize uint64 |
There was a problem hiding this comment.
totally pedantic, but if we're going to come up with totalSize by summing up all the bytes that come out of the lexer, we'll need to add 8 bytes each for the header and trailer magic.
| } | ||
|
|
||
| printTable(os.Stdout, rows, []string{ | ||
| "record kind", "sum bytes", "% of total file bytes", |
There was a problem hiding this comment.
| "record kind", "sum bytes", "% of total file bytes", | |
| "record kind", "sum bytes", "% of total file bytes (after chunk decompression)", |
| return nil | ||
| } | ||
|
|
||
| func printTable(w io.Writer, rows [][]string, header []string) { |
There was a problem hiding this comment.
use utils.FormatTable here for consistency with other commands?
There was a problem hiding this comment.
The help text for this subcommand should remind the user that this reads the entire file.
Regarding showing per-record-type usage
We've had questions around this before, sometimes non-message records can take up a surprising amount of room. For example when @amacneil was coming up with plot test data, his example file with millions of tiny messages had almost half the space taken up with MessageIndex records. We've also previously had writers who re-wrote schema records for every new channel, which can also add up. I think this is valuable to know.
Adjusting for compression
There's some nuance missing here around compression.
If i'm looking for the total % contribution of some record type or topic to the file's size on disk, I need to see the effect that compression is having on that size. Right now you display percentages where the denominator is a sum of all record bytes after decompression. this will de-emphasise the impact of uncompressed records, particularly MessageIndex records.
Given:
- sum of uncompressed record bytes
ru - sum of uncompressed record bytes in chunk n
rc[n] - the compressed data size of chunk n
tc[n] - the uncompressed data size of chunk n
tu[n] - the total size of the file
f
I could imagine synthesizing a record impact estimate as:
impact % = (ru + sum(rc[n] * (tc[n] / tu[n]) for all n)) / f
which assumes all bytes in a compressed chunk contribute equally to the compressed output size, which isn't true but there's no easy way to know the right answer.
If this feels too complicated to present, you could consider presenting some of these values separately and allowing the user to do their own mental arithmetic.
Schemas
IMO it's worth parsing schema records and including the schema names and encodings when presenting per-topic statistics.
Human-readable bytes
We have a humanBytes function used in info.go, you could re-use that to make the byte counts more presentable (perhaps under a flag).
Problem: When looking at mcap files I want to understand how much space particular topics take up. I might want to know if I have some heavyweight topics I need to trim or if some debug topic is taking a large amount of space relative to its worth. Having a command that shows me how much "space" is used by each topic can help me answer such questions.
This change introduces a new
mcap ducommand. This command reads and mcap file and outputs "disk usage" statistics about the mcap file.Below I show invocations of this command on some sample mcap files. The output shows the space taken by each kind of record and then the space per topic (relative to all topics). For me, the primary use-case for this command to show the "topic" information but wanted to offer both sets of "usage" for discussion. I would probably hide the "record" breakdown under a flag or remove it entirely.
Implementation notes:
infocommand as a flag.