Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including many databases like PostgreSQL, ClickHouse, Redis, MongoDB, etc.) - the results are available here, database-related results could be checked here. That's why I think it's worth trying to apply PGO to Readyset to improve the application performance/reduce CPU usage.

I can suggest the following things to do:

• Evaluate PGO's results on Readyset.
• If PGO helps to achieve better performance - add a note to Readyset's documentation about that. In this case, users and maintainers will be aware of another optimization opportunity for Readyset.
• Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their workloads.
• Optimize prebuilt Readyset binaries with PGO.

Here are some examples of how PGO is already integrated into other projects' build scripts:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag
• ISPC: CMake scipts
• NodeJS: Configure script
• Android Open Source Project (AOSP):
  • Official documentation
  • Committed PGO profiles: repository
• DMD: Custom build rule
• LDC: GitHub action
• tsv-utils: Makefile
• Erlang OTP: Makefile
• Clingo (PGO enabled only in Spack): Package recipe
• SWI-Prolog:
  • Script
  • CMake module
• hck: Justfile

Here are some examples of how PGO-related documentation could look like in the project:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html
• tsv-utils: https://github.com/eBay/tsv-utils/blob/master/docs/BuildingWithLTO.md

After PGO, I can suggest evaluating PLO with LLVM BOLT as an additional optimization step after PGO.

Below are listed some LLVM BOLT results:

• Rustc:
  • Rustc itself (GitHub PR)
  • LLVM in Rustc (Reddit)
• CPython: GitHub PR
• YDB: GitHub comment
• Clang:
  • Slides
  • Results on building Clang
  • Linaro results
  • on AMD 7950X3D
  • GitHub comment from LLVM
• LDC: GitHub comment
• HHVM, Proxygen and others: Facebook paper
• NodeJS: Blog
• Chromium: Blog
• MySQL, MongoDB, memcached, Verilator: Paper

For trying PGO I recommend using cargo-pgo project. cargo-pgo supports both PGO and PLO optimizations (via LLVM BOLT). However, I recommend starting with PGO - it's much more stable compiler optimization compared to BOLT.

I found the issue about the benchmark roadmap - #432 . So probably will be a good idea to wait for the benchmark suite stabilization, and only after that try to apply PGO/PLO to Readyset.

I will be happy to answer all your questions about PGO/PLO.