4.08 switch creation time on raspian - +164% !

[progman1]

it took so long I thought something was hanging:

4.07.0 25m
4.08.0 66m

is there a known reason for this increase?

[nojb]

Maybe related to #8776 ?

[gasche]

One important difference in 4.08 is the switch to a Menhir-generated parser generator, which generates a parser.ml that is noticeably slower to compile than the ocamlyacc-generated one in 4.07.

To measure this, I would recommend running the following from a clean checkout of the OCaml compiler (on both versions):

./configure
time make parsing/parser.cmo

(On my machine this takes 8s on 4.08 compared to 2s on 4.07.)

@progman1, do you observe a significant difference between 4.07 and 4.08 on the build of the parser?

[gasche]

I measured the times of various parts of the build on my usual machine, under 4.07 and 4.08.

make core
  4.07: 1m10s
  4.08: 1m26s

make world
  4.07: 0m35s
  4.08: 0m41s

make opt
  4.07: 0m33s
  4.08: 0m40s

make ocamlc.opt
  4.07: 0m48s
  4.08: 1m04s

make opt.opt
  4.07: 0m58s
  4.08: 1m09s

My take-aways:

• everything seems to get a bit slower, there is no single cost center at this level of separation
• the fact that "make core", which is the very first step, is already affected (although you have to remove the ~6s difference explained by the move from ocamlyacc to Menhir) means that it should be possible to bisect the difference.

[progman1]

some numbers on individual compiler/parser invocations that mention ml/mli/mly/mll/cma/cmxa
I fixed ocamlc/yacc/ocamlrun to those under boot/yacc so that probably messed with the opt stage results.

take-away? @nojb has a much faster machine than I!

4.08 world 20m.53

64.58 COMPILING: parsing/parser.ml
20.37 COMPILING: typing/typecore.ml
16.44 COMPILING: parsing/pprintast.ml
16.26 COMPILING: camlinternalFormat.ml
16.09 COMPILING: camlinternalFormat.ml
15.31 COMPILING: typing/ctype.ml
14.95 COMPILING: odoc_html.ml
11.24 COMPILING: bytecomp/matching.ml
10.11 COMPILING: typing/typemod.ml
9.13 COMPILING: typing/printtyp.ml
9.03 COMPILING: typing/env.ml
8.07 COMPILING: odoc_latex.ml
7.96 COMPILING: typing/parmatch.ml
7.80 COMPILING: typing/printtyped.ml
7.51 COMPILING: parsing/printast.ml
7.37 COMPILING: typing/typeclass.ml
6.78 COMPILING: typing/typedecl.ml
6.47 COMPILING: odoc_texi.ml
6.44 COMPILING: middle_end/flambda.ml

4.07 world 14m.58 !?

107.43 COMPILING: ocamldoc link ...
20.14 COMPILING: typing/typecore.ml
14.56 COMPILING: parsing/pprintast.ml
14.09 COMPILING: camlinternalFormat.ml
13.40 COMPILING: typing/ctype.ml
12.19 COMPILING: parsing/parser.ml
10.86 COMPILING: odoc_html.ml
9.62 COMPILING: bytecomp/matching.ml
7.37 COMPILING: typing/printtyp.ml
7.13 COMPILING: typing/typedecl.ml
7.08 COMPILING: typing/typemod.ml
6.93 COMPILING: typing/parmatch.ml
6.78 COMPILING: typing/env.ml
6.67 COMPILING: typing/typeclass.ml
6.54 COMPILING: parsing/printast.ml
6.52 COMPILING: typing/printtyped.ml
6.03 COMPILING: odoc_latex.ml
5.56 COMPILING: middle_end/flambda.ml
5.08 COMPILING: odoc_texi.ml

4.08 world.opt 32m21

87.77 COMPILING: parsing/parser.ml
81.27 COMPILING: odoc_html.ml
55.01 COMPILING: typing/typecore.ml
33.18 COMPILING: asmcomp/cmmgen.ml
32.51 COMPILING: parsing/pprintast.ml
32.18 COMPILING: typing/ctype.ml
26.16 COMPILING: odoc_texi.ml
25.98 COMPILING: camlinternalFormat.ml
25.61 COMPILING: camlinternalFormat.ml
24.48 COMPILING: odoc_args.ml
20.68 COMPILING: bytecomp/matching.ml
19.68 COMPILING: typing/typemod.ml
19.54 COMPILING: odoc_latex.ml
18.72 COMPILING: odoc_man.ml
18.16 COMPILING: asmcomp/cmmgen.ml
16.30 COMPILING: driver/optmain.ml
16.03 COMPILING: asmcomp/selectgen.ml
16.01 COMPILING: typing/printtyp.ml
15.81 COMPILING: typing/env.ml
15.36 COMPILING: typing/typeclass.ml

4.07 world.opt 27m.23

97.89 COMPILING: odoc_html.ml
48.65 COMPILING: typing/typecore.ml
31.77 COMPILING: parsing/pprintast.ml
29.97 COMPILING: asmcomp/cmmgen.ml
28.12 COMPILING: typing/ctype.ml
26.53 COMPILING: odoc_args.ml
25.39 COMPILING: odoc_latex.ml
24.77 COMPILING: ocamloptp.ml
24.21 COMPILING: camlinternalFormat.ml
23.73 COMPILING: camlinternalFormat.ml
23.21 COMPILING: parsing/parser.ml
21.61 COMPILING: driver/optmain.ml
21.00 COMPILING: odoc_man.ml
18.71 COMPILING: bytecomp/matching.ml
15.98 COMPILING: asmcomp/cmmgen.ml
14.84 COMPILING: odoc_texi.ml
14.59 COMPILING: asmcomp/selectgen.ml
14.35 COMPILING: typing/typeclass.ml
14.11 COMPILING: typing/typemod.ml
13.29 COMPILING: typing/typedecl.ml

[gasche]

I haven't looked in details but the overall result don't seem terribly surprising: parsing/parser.ml is noticeably slower than before, but the rest is about the same with some slowdowns here and there -- which isn't that surprising as we keep adding more code to the compiler codebase, and rarely remove existing code, so there is simply more to compile. I cannot see anything in those detailed numbers that explains the huge slowdown you observed when creating the issue; are you sure you can reproduce it?

For the record I tried to bisect a change in compilation time of core between 4.07 and 4.08, but those things are very hard to measure with reasonable noise levels and I got essentially nowehere -- I can tell that the 4.08 core is slower to build than 4.07 by about 10 seconds (on my machine), but it's hard to find where those extra seconds where introduced.

[pmetzger]

It's probably easier to profile the compiler (and the build process for compiler switches) and find ways to speed it up than to find specific regressions.

[xavierleroy]

The conclusion seems to be "the Menhir-generated parser takes longer to compile, live with it". If it's correct, I move to close. If there is more to say and investigate, please do.

[gasche]

The report gave two measurements; the first suggested a very large increase (164%) in compilation time, the second (with detailed numbers) suggests a noticeable but much smaller increase (+18% for world.opt). The second difference is explained by the Menhir-based parser; the first difference remains unexplained but may be a non-reproducible outlier, and we haven't heard back with more measures to help uncover it.

I think that this particular issue is dealt with, but it also calls attention to the fact that we don't have any robust metric for compilation speed for the compiler development, and it would be nicer if we had. My main take-away is that it should be someone's* priority to find and manage a volunteer for long-term continuous performance monitoring of the compiler trunk.

[github-actions]

This issue has been open one year with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. Comments that provide new information on the issue are especially welcome: is it still reproducible? did it appear in other contexts? how critical is it? etc.

[gasche]

No further progress on this issue, so closing.