Summarize diffs

[Centril]

When writing a PR to rustc you often want either to track the source of a perf regression or to track an improvement. Absolute values don't matter as much.

To that end, it would be nice if you could do summarize pid-$baseline pid-$change where $baseline could correspond to the master branch of rustc and $change could correspond to your PR which made changes. You could possibly also write summarize diff ... instead to simplify the argument parsing logic and infer less.

Example: I wanted this when working on rust-lang/rust#59288; it was hard to work with two summarize outputs side by side.

[wesleywiser]

It would be great to have an idea of what the output should look like. Each query has a number of different facets that can be compare. For example, one way to render this might be something like:

+--------------------------+-----------+-----------------+------------+------------+--------------+-----------------------+
| Item                     | Self time | % of total time | Item count | Cache hits | Blocked time | Incremental load time |
+--------------------------+-----------+-----------------+------------+------------+--------------+-----------------------+
| LLVM_emit_obj            | -0.62s    | -1.2            | -1         | 0          | 0.00ns       | 0.00ns                |
+--------------------------+-----------+-----------------+------------+------------+--------------+-----------------------+
| LLVM_module_passes       | +0.57s    | +1.0            | +2         | 0          | 0.00ns       | 0.00ns                |
+--------------------------+-----------+-----------------+------------+------------+--------------+-----------------------+

[Centril]

@wesleywiser That output looks what I expected it to.

[wesleywiser]

I'd be willing to help somebody implement this if they're interested. Otherwise, I'll probably work on this sometime in the next week or two.

Here's some help to get started:

• We'll want to add a subcommand to the command line interface as mentioned above

  • Something like summarize diff base change where base and change are files.
  • We should support both the "raw" profiling files we get from rustc (.events, .string_index, and .string_data) as well as the json output files generated from summarize --json.
  • We should autodetect the file type via extension.

• Internally, each query or activity is represented by a struct:

  measureme/summarize/src/analysis.rs

  Lines 10 to 18 in b3bd354

  	pub struct QueryData {
  	pub label: String,
  	pub self_time: Duration,
  	pub number_of_cache_misses: usize,
  	pub number_of_cache_hits: usize,
  	pub invocation_count: usize,
  	pub blocked_time: Duration,
  	pub incremental_load_time: Duration,
  	}

  • We should impl Sub for QueryData so that we can do change - base
  • We'll need to change the usize fields to be signed. i64 is probably appropriate.

• When loading files, if the files are json files we can just deserialize the data from the file. Otherwise, we'll need to run the current analysis on the file contents to get the set of QueryData objects to work with.

• We should support the --json argument to output the diff results to a json file.

• As a bonus, it would be cool to have green/red highlighting when outputting to the terminal.

[Darksonn]

I'm going to take a look at implementing this.

[Darksonn]

I don't think StructOpt supports a diff subcommand pattern without making the single-command pattern a subcommand too.

[wesleywiser]

It looks like it might work if the subcommand is wrapped in Option: TeXitoi/structopt#68

[Darksonn]

My laptop has run into some hardware issues, so it might take a bit before I get around to this.