Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO) for Lapce

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are available here. According to the tests, PGO can help with achieving better performance in many cases for many applications. Since on the Lapce website I see "lightning fast" statement - I think trying to optimize Lapce with PGO can be a good idea. There are multiple places that could benefit from PGO: LSP server, different parsers, RPC proxy, etc.

I can suggest the following action points:

• Perform PGO benchmarks on Lapce. If it shows improvements - add a note to the documentation about possible improvements in Lapce performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize Lapce according to their workloads.
• Optimize pre-built Lapce binaries.

Testing Post-Link Optimization (PLO) techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Here are some examples of how PGO optimization is integrated in other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

Here are some examples of how PGO-related documentation could look in the project:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html
• tsv-utils: https://github.com/eBay/tsv-utils/blob/master/docs/BuildingWithLTO.md

Below are listed some BOLT results:

• Rustc:
  • Rustc itself (GitHub PR)
  • LLVM in Rustc (Reddit)
• CPython: GitHub PR
• YDB: GitHub comment
• Clang:
  • Slides
  • Results on building Clang
  • Linaro results
  • on AMD 7950X3D
  • GitHub comment from LLVM
• LDC: GitHub comment
• HHVM, Proxygen and others: Facebook paper
• NodeJS: Blog
• Chromium: Blog
• MySQL, MongoDB, memcached, Verilator: Paper

For the Rust projects I recommend using cargo-pgo for performing PGO and PLO optimizations.

I tried to find an existing benchmark ecosystem for Lapce but found nothing. If anyone can suggest how Lapce can be benchmarked - would be great!

[panekj]

Hi, I evaluated, we are not going to do that, since the improvement is not significant

[zamazan4ik]

Could you please share:

• How did you perform PGO on Lapce?
• What was the training and evaluation scenario?
• OS, compiler version
• Performance numbers before and after PGO

It would be really interesting to read about your experience with PGO for Lapce. Thanks in advance!

[panekj]

There was no testing on Lapce, PGO in itself just doesn't bring much value and to base it on benchmarks is foolish idea

[zamazan4ik]

There was no testing on Lapce, PGO in itself just doesn't bring much value and to base it on benchmarks is foolish idea

I definitely can say that PGO brings from the performance perspective as it shown in the different testing scenarios linked by me above. I cannot understand how to show performance improvements without some benchmarks... How else could we show the improvements? :)

E.g. there is a great post from the Rust dev team about speeding up the compiler: https://blog.rust-lang.org/inside-rust/2020/11/11/exploring-pgo-for-the-rust-compiler.html . It's not a "synthetic" bench suite - that's a real-life workload that was improved (and already deployed for many users). And I have many other examples.

So right now I cannot understand your point about PGO - probably I missed some details and/or insights.

[panekj]

I cannot understand how to show performance improvements without some benchmarks... How else could we show the improvements? :)

If the improvement is not visible without benchmark, is it really an improvement? As far as this goes, it's only different numbers without real impact on users.

I definitely can say that PGO brings from the performance perspective as it shown in the different testing scenarios linked by me above.

That either say the performance was insignificantly better or worse.

It's not a "synthetic" bench suite - that's a real-life workload that was improved (and already deployed for many users). And I have many other examples.

This might be great for projects that have traction, deal with a plethora of different hardware/software/etc. and projects that closely deal with compiling software (like rustc).
For us this will most likely not bring any improvement and only complicate our work by adding additional complexity and tools to do something that is not necessary.

[zamazan4ik]

If the improvement is not visible without benchmark, is it really an improvement? As far as this goes, it's only different numbers without real impact on users.

What else measurable ways could you suggest to show the performance improvements? From my point of view, compilation speed (for compilers), RPS/latency (for servers like HAproxy and others), indexing/latency time (for stuff like Clangd/Rust-analyzer) is definitely a thing that is visible to the user. My latest example - Vtracer. Me, as an image converter user, definitely see a difference in 10 minutes for converting one image - it directly influences the software UX.

That either say the performance was insignificantly better or worse.

If you check my benchmarks, you will see that in many cases the difference could be up to 30-40% based on several metrics. Of course, "significance" word meaning is a subjective topic and is context-dependent. E.g. if improving the compilation speed by 20% or improving TPS by 30% for a database is not significant to you - okay, it's a fair point. I just have enough opinion.

deal with a plethora of different hardware/software

PGO does not depend on the hardware features, to be honest (at least directly). The most significant improvements from PGO in practice - the better inlining decisions (it affects a lot of other compiler optimizations), the better hot/cold code splitting. About "software" I do not understand your point. If you mean "different software input" - it's a fair point, sometimes it could be difficult in practice to prepare one PGO profile for all. But at least users are able to use PGO and "train" it for their workload without relying on the provided by the upstream profiles (like it's done for ClickHouse as an example).

and projects that closely deal with compiling software (like rustc).

As shown in my link above, PGO helps to achieve better performance results with many other software: databases, image converters, log solutions, networking stuff, LSPs and static analysis (okay, these two are similar to the compilers but I should mention them here IMO), browsers, search engines (according to the Google papers). So PGO benefits definitely are not limited only to the compilers.

For us this will most likely not bring any improvement and only complicate our work by adding additional complexity and tools to do something that is not necessary.

I understand this point. But before performing the actual Lapce benchmarks probably will be too early to say that PGO will not bring improvements to Lapce. If you as a maintainer are not interested in evaluating PGO - it's totally fine. I can suggest leaving the discussion open, and probably someone from the open-source community will decide to give a try for PGO in Lapce, perform some measurements in real-life scenarios, measure improvements (the editor latency, CPU/battery consumption in real-life scenarios), and report the results back. Only when we get the actual results and the results will not be good enough - close the discussion.