Add pgo support for go 1.20

[prestonvanloon]

What type of PR is this?

Feature

What does this PR do? Why is it needed?

This PR add support for profile guided optimization for the go compiler.

The design is that we add a new label flag in go/config/BUILD.bazel which is used to set the a files target which would provide the first file to the compiler via -pgoprofile when compiling the standard library, all dependencies, and the go_binary target itself. This is achieved in a similar manner as other go.mode settings.

The flag -pgoprofile only supports a single file and therefore this implementation only uses the first file provided by the target in //go/config:pgoprofile. If more than one file is provided, then an error is raised at build evaluation time. However, this functionality could be extended to allow multiple files which are merged with go tool pprof -proto a.pprof b.pprof > merged.pprof. See the relevant pgo doc section for "Merging Profiles".

Which issues(s) does this PR fix?

Fixes #3637

Other notes for review

I am not sure how to add unit tests for this. For reviewers, you can build //tests/core/go_binary:pgo with the -s flag and see that the GoStdlib and GoCompilePkg steps are completed with a -pgoprofile flag with the appropriate pprof file. A diff test is added with guidance from @fmeum. You may also observe that other targets do not apply any pgoprofile flag unless provided with pgoprofile in the go_binary attributes or with --@io_bazel_rules_go//go/config:pgoprofile=//path/to/some/file:target.

Example subcommands output of bazel build //tests/core/go_binary:pgo -s

SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: b0dbd3c3bd8c0564c88fa23e59f31c539cbe3ccb49a924d6c82845295c3ba768, execution platform: @local_config_platform//:host]
(cd /home/preston/.cache/bazel/_bazel_preston/996ab6b61f6a3f8e34862586db434593/execroot/io_bazel_rules_go && \
exec env - \
/bin/bash -c 'GOMAXPROCS=1 GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/k8-opt-exec-2B5CBBC6-ST-3b75f3d342c9/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: b0dbd3c3bd8c0564c88fa23e59f31c539cbe3ccb49a924d6c82845295c3ba768
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 07d9ce9fb2f8271fc191c57adfe801eb192a97195eca61eabd82bbd5ad6ea279, execution platform: @local_config_platform//:host]
(cd /home/preston/.cache/bazel/_bazel_preston/996ab6b61f6a3f8e34862586db434593/execroot/io_bazel_rules_go && \
exec env - \
/bin/bash -c 'GOMAXPROCS=1 GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder -trimpath go/tools/builders/ar.go go/tools/builders/asm.go go/tools/builders/builder.go go/tools/builders/cgo2.go go/tools/builders/compilepkg.go go/tools/builders/cover.go go/tools/builders/edit.go go/tools/builders/embedcfg.go go/tools/builders/env.go go/tools/builders/filter.go go/tools/builders/filter_buildid.go go/tools/builders/flags.go go/tools/builders/generate_nogo_main.go go/tools/builders/generate_test_main.go go/tools/builders/importcfg.go go/tools/builders/link.go go/tools/builders/pack.go go/tools/builders/read.go go/tools/builders/replicate.go go/tools/builders/stdlib.go go/tools/builders/stdliblist.go go/tools/builders/path.go')
# Configuration: 07d9ce9fb2f8271fc191c57adfe801eb192a97195eca61eabd82bbd5ad6ea279
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //:stdlib [action 'GoStdlib stdlib_/pkg', configuration: 31bad24ddf09a77ea685a480c8bf55014ef816a7a7a67ca249f4fe95b563bd70, execution platform: @local_config_platform//:host]
(cd /home/preston/.cache/bazel/_bazel_preston/996ab6b61f6a3f8e34862586db434593/execroot/io_bazel_rules_go && \
exec env - \
  CC=/usr/bin/gcc \
  CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
  CGO_ENABLED=1 \
  CGO_LDFLAGS='-fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm' \
  GOARCH=amd64 \
  GOEXPERIMENT=nocoverageredesign \
  GOOS=linux \
  GOPATH='' \
  GOROOT=external/go_sdk \
  GOROOT_FINAL=GOROOT \
  PATH=/usr/bin:/bin \
bazel-out/k8-opt-exec-2B5CBBC6-ST-3b75f3d342c9/bin/external/go_sdk/builder_reset/builder stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild-ST-3b75f3d342c9/bin/stdlib_ -package std -package runtime/cgo -gcflags '' -pgoprofile tests/core/go_binary/pgo.pprof)
# Configuration: 31bad24ddf09a77ea685a480c8bf55014ef816a7a7a67ca249f4fe95b563bd70
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //tests/core/go_binary:pgo [action 'GoCompilePkg tests/core/go_binary/pgo.a', configuration: 1681c2d1d1eaeaaa5005f3a2f911b4dd5000ea89900fefdc87107732f5b8c22a, execution platform: @local_config_platform//:host]
(cd /home/preston/.cache/bazel/_bazel_preston/996ab6b61f6a3f8e34862586db434593/execroot/io_bazel_rules_go && \
exec env - \
  CGO_ENABLED=1 \
  GOARCH=amd64 \
  GOEXPERIMENT=nocoverageredesign \
  GOOS=linux \
  GOPATH='' \
  GOROOT=bazel-out/k8-fastbuild-ST-3b75f3d342c9/bin/stdlib_ \
  GOROOT_FINAL=GOROOT \
  PATH=/usr/bin:/bin \
bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src tests/core/go_binary/pgo.go -embedroot '' -embedroot bazel-out/k8-fastbuild/bin -embedlookupdir tests/core/go_binary -importpath pgo -p main -package_list bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/packages.txt -o bazel-out/k8-fastbuild/bin/tests/core/go_binary/pgo.a -x bazel-out/k8-fastbuild/bin/tests/core/go_binary/pgo.x -gcflags '' -asmflags '' -pgoprofile tests/core/go_binary/pgo.pprof)
# Configuration: 1681c2d1d1eaeaaa5005f3a2f911b4dd5000ea89900fefdc87107732f5b8c22a
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //tests/core/go_binary:pgo [action 'GoLink tests/core/go_binary/pgo_/pgo', configuration: 1681c2d1d1eaeaaa5005f3a2f911b4dd5000ea89900fefdc87107732f5b8c22a, execution platform: @local_config_platform//:host]
(cd /home/preston/.cache/bazel/_bazel_preston/996ab6b61f6a3f8e34862586db434593/execroot/io_bazel_rules_go && \
exec env - \
  CGO_ENABLED=1 \
  GOARCH=amd64 \
  GOEXPERIMENT=nocoverageredesign \
  GOOS=linux \
  GOPATH='' \
  GOROOT=bazel-out/k8-fastbuild-ST-3b75f3d342c9/bin/stdlib_ \
  GOROOT_FINAL=GOROOT \
  PATH=/usr/bin:/bin \
bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder link -sdk external/go_sdk -installsuffix linux_amd64 -package_list bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/packages.txt -o bazel-out/k8-fastbuild/bin/tests/core/go_binary/pgo_/pgo -main bazel-out/k8-fastbuild/bin/tests/core/go_binary/pgo.a -p tests/core/go_binary/pgo -- -extld /usr/bin/gcc '-buildid=redacted' -s -w -extldflags '-fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm')
# Configuration: 1681c2d1d1eaeaaa5005f3a2f911b4dd5000ea89900fefdc87107732f5b8c22a
# Execution platform: @local_config_platform//:host

Observe -pgoprofile tests/core/go_binary/pgo.pprof on StdLib and GoCompilePkg steps.

Thank you to @fmeum for your guidance on the implementation. This would not have been possible without your help.

[fmeum]

An idea for testing: We could build the sample application twice in a go_bazel_test, once with and once without pgo, and compare that the results differ.

[prestonvanloon]

An idea for testing: We could build the sample application twice in a go_bazel_test, once with and once without pgo, and compare that the results differ.

@fmeum I've given that a try in b1786e1 and responded to your feedback in 4f3ec47

[sluongng]

This is fantastic.

Do you know whether go build would extend PGO to cover the standard library as well? Because I would suspect the current approach is missing out on GoStdLib.

[prestonvanloon]

This is fantastic.

Do you know whether go build would extend PGO to cover the standard library as well? Because I would suspect the current approach is missing out on GoStdLib.

@sluongng Yes, the pgoprofile flag is applied when building the standard library.

See changes in go/tools/builders/stdlib.go and go/private/actions/stdlib.bzl.