Update go1.21.4 #79
Merged
Update go1.21.4 #79
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Done with: go get golang.org/x/net@internal-branch.go1.21-vendor go mod tidy go mod vendor go generate net/http # zero diff since CL 534235 already did this For golang#63417. For golang#63427. For CVE-2023-39325. Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90 Reviewed-on: https://go-review.googlesource.com/c/go/+/534415 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
This CL is a roll-forward (tweaked slightly) of CL 467715, which turned on text section splitting for GOARCH=arm. The intent is to avoid recurrent problems with external linking where there is a disagreement between the Go linker and the external linker over whether a given branch will reach. In the past our approach has been to tweak the reachability calculations slightly to try to work around potential linker problems, but this hasn't proven to be very robust; section splitting seems to offer a better long term fix. Updates golang#58425. Fixes golang#63317. Change-Id: I7372d41abce84097906a3d0805b6b9c486f345d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/531795 Reviewed-by: Cherry Mui <cherryyz@google.com> Run-TryBot: Than McIntosh <thanm@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit 1e69040) Reviewed-on: https://go-review.googlesource.com/c/go/+/532096 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
…argument function's type See the comment in the (very small) fix for a detailed description. Use the opportunity to introduce a generic clone function which may be useful elsewhere. Fixes golang#63339. Change-Id: Ic63c6b8bc443011b1a201908254f7d062e1aec71 Reviewed-on: https://go-review.googlesource.com/c/go/+/532157 Run-TryBot: Robert Griesemer <gri@google.com> Reviewed-by: Robert Findley <rfindley@google.com> Reviewed-by: Robert Griesemer <gri@google.com> Auto-Submit: Robert Griesemer <gri@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-on: https://go-review.googlesource.com/c/go/+/531998 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
This has caused performance issues in production environments. MADV_COLLAPSE can go into direct reclaim, but we call it with the heap lock held. This means that the process could end up stalled fairly quickly if just one allocating goroutine ends up in the madvise call, at least until the madvise(MADV_COLLAPSE) call returns. A similar issue occurred with madvise(MADV_HUGEPAGE), because that could go into direct reclaim on any page fault for MADV_HUGEPAGE-marked memory. My understanding was that the calls to madvise(MADV_COLLAPSE) were fairly rare, and it's "best-effort" nature prevented it from going into direct reclaim often, but this was wrong. It tends to be fairly heavyweight even when it doesn't end up in direct reclaim, and it's almost certainly not worth it. Disable it until further notice and let the kernel fully dictate hugepage policy. The updated scavenger policy is still more hugepage friendly by delaying scavening until hugepages are no longer densely packed, so we don't lose all that much. The Sweet benchmarks show a minimal difference. A couple less realistic benchmarks seem to slow down a bit; they might just be getting unlucky with what the kernel decides to back with a huge page. Some benchmarks on the other hand improve. Overall, it's a wash. name old time/op new time/op delta BiogoIgor 13.1s ± 1% 13.2s ± 2% ~ (p=0.182 n=9+10) BiogoKrishna 12.0s ± 1% 12.1s ± 1% +1.23% (p=0.002 n=9+10) BleveIndexBatch100 4.51s ± 4% 4.56s ± 3% ~ (p=0.393 n=10+10) EtcdPut 20.2ms ± 4% 19.8ms ± 2% ~ (p=0.079 n=10+9) EtcdSTM 109ms ± 3% 111ms ± 3% +1.63% (p=0.035 n=10+10) GoBuildKubelet 31.2s ± 1% 31.3s ± 1% ~ (p=0.780 n=9+10) GoBuildKubeletLink 7.77s ± 0% 7.81s ± 2% ~ (p=0.237 n=8+10) GoBuildIstioctl 31.8s ± 1% 31.7s ± 0% ~ (p=0.136 n=9+9) GoBuildIstioctlLink 7.88s ± 1% 7.89s ± 1% ~ (p=0.720 n=9+10) GoBuildFrontend 11.7s ± 1% 11.8s ± 1% ~ (p=0.278 n=10+9) GoBuildFrontendLink 1.15s ± 4% 1.15s ± 5% ~ (p=0.387 n=9+9) GopherLuaKNucleotide 19.7s ± 1% 20.6s ± 0% +4.48% (p=0.000 n=10+10) MarkdownRenderXHTML 194ms ± 3% 196ms ± 3% ~ (p=0.356 n=9+10) Tile38QueryLoad 633µs ± 2% 629µs ± 2% ~ (p=0.075 n=10+10) name old average-RSS-bytes new average-RSS-bytes delta BiogoIgor 69.2MB ± 3% 68.4MB ± 1% ~ (p=0.190 n=10+10) BiogoKrishna 4.40GB ± 0% 4.40GB ± 0% ~ (p=0.605 n=9+9) BleveIndexBatch100 195MB ± 3% 195MB ± 2% ~ (p=0.853 n=10+10) EtcdPut 107MB ± 4% 108MB ± 3% ~ (p=0.190 n=10+10) EtcdSTM 91.6MB ± 5% 92.6MB ± 4% ~ (p=0.481 n=10+10) GoBuildKubelet 2.26GB ± 1% 2.28GB ± 1% +1.22% (p=0.000 n=10+10) GoBuildIstioctl 1.53GB ± 0% 1.53GB ± 0% +0.21% (p=0.017 n=9+10) GoBuildFrontend 556MB ± 1% 554MB ± 2% ~ (p=0.497 n=9+10) GopherLuaKNucleotide 39.0MB ± 3% 39.0MB ± 1% ~ (p=1.000 n=10+8) MarkdownRenderXHTML 21.2MB ± 2% 21.4MB ± 3% ~ (p=0.190 n=10+10) Tile38QueryLoad 5.99GB ± 2% 6.02GB ± 0% ~ (p=0.243 n=10+9) name old peak-RSS-bytes new peak-RSS-bytes delta BiogoIgor 90.2MB ± 4% 89.2MB ± 2% ~ (p=0.143 n=10+10) BiogoKrishna 4.49GB ± 0% 4.49GB ± 0% ~ (p=0.190 n=10+10) BleveIndexBatch100 283MB ± 8% 274MB ± 6% ~ (p=0.075 n=10+10) EtcdPut 147MB ± 4% 149MB ± 2% +1.55% (p=0.034 n=10+8) EtcdSTM 117MB ± 5% 117MB ± 4% ~ (p=0.905 n=9+10) GopherLuaKNucleotide 44.9MB ± 1% 44.6MB ± 1% ~ (p=0.083 n=8+8) MarkdownRenderXHTML 22.0MB ± 8% 22.1MB ± 9% ~ (p=0.436 n=10+10) Tile38QueryLoad 6.24GB ± 2% 6.29GB ± 2% ~ (p=0.218 n=10+10) name old peak-VM-bytes new peak-VM-bytes delta BiogoIgor 1.33GB ± 0% 1.33GB ± 0% ~ (p=0.504 n=10+9) BiogoKrishna 5.77GB ± 0% 5.77GB ± 0% ~ (p=1.000 n=10+9) BleveIndexBatch100 3.53GB ± 0% 3.53GB ± 0% ~ (p=0.642 n=10+10) EtcdPut 12.1GB ± 0% 12.1GB ± 0% ~ (p=0.564 n=10+10) EtcdSTM 12.1GB ± 0% 12.1GB ± 0% ~ (p=0.633 n=10+10) GopherLuaKNucleotide 1.26GB ± 0% 1.26GB ± 0% ~ (p=0.297 n=9+10) MarkdownRenderXHTML 1.26GB ± 0% 1.26GB ± 0% ~ (p=0.069 n=10+10) Tile38QueryLoad 7.47GB ± 2% 7.53GB ± 2% ~ (p=0.280 n=10+10) name old p50-latency-ns new p50-latency-ns delta EtcdPut 19.8M ± 5% 19.3M ± 3% -2.74% (p=0.043 n=10+9) EtcdSTM 81.4M ± 4% 83.4M ± 4% +2.46% (p=0.029 n=10+10) Tile38QueryLoad 241k ± 1% 240k ± 1% ~ (p=0.393 n=10+10) name old p90-latency-ns new p90-latency-ns delta EtcdPut 30.4M ± 5% 30.6M ± 5% ~ (p=0.971 n=10+10) EtcdSTM 222M ± 3% 226M ± 4% ~ (p=0.063 n=10+10) Tile38QueryLoad 687k ± 2% 691k ± 1% ~ (p=0.173 n=10+8) name old p99-latency-ns new p99-latency-ns delta EtcdPut 42.3M ±10% 41.4M ± 7% ~ (p=0.353 n=10+10) EtcdSTM 486M ± 7% 487M ± 4% ~ (p=0.579 n=10+10) Tile38QueryLoad 6.43M ± 2% 6.37M ± 3% ~ (p=0.280 n=10+10) name old ops/s new ops/s delta EtcdPut 48.6k ± 3% 49.5k ± 2% ~ (p=0.065 n=10+9) EtcdSTM 9.09k ± 2% 8.95k ± 3% -1.56% (p=0.045 n=10+10) Tile38QueryLoad 28.4k ± 1% 28.6k ± 1% +0.87% (p=0.016 n=9+10) Fixes golang#63335. For golang#63334. Related to golang#61718 and golang#59960. Change-Id: If84c5a8685825d43c912a71418f2597e44e867e5 Reviewed-on: https://go-review.googlesource.com/c/go/+/531816 Reviewed-by: Michael Pratt <mpratt@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Michael Knyszek <mknyszek@google.com> (cherry picked from commit 595deec) Reviewed-on: https://go-review.googlesource.com/c/go/+/532255 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
This fixes cherry-pick CL 531998. For golang#63339. Change-Id: I6dac0909ca85d68684ce36025284d25db32e0b15 Reviewed-on: https://go-review.googlesource.com/c/go/+/535135 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Alan Donovan <adonovan@google.com>
…ield naming When creating the struct type to hold variables captured by a function literal, we currently reuse the captured variable names as fields. However, there's no particular reason to do this: these struct types aren't visible to users, and it adds extra complexity in making sure fields belong to the correct packages. Further, it turns out we were getting that subtly wrong. If two function literals from different packages capture variables with identical names starting with an uppercase letter (and in the same order and with corresponding identical types) end up in the same function (e.g., due to inlining), then we could end up creating closure struct types that are "different" (i.e., not types.Identical) yet end up with equal LinkString representations (which violates LinkString's contract). The easy fix is to just always use simple, exported, generated field names in the struct. This should allow further struct reuse across packages too, and shrink binary sizes slightly. For golang#62498. Fixes golang#62545. Change-Id: I9c973f5087bf228649a8f74f7dc1522d84a26b51 Reviewed-on: https://go-review.googlesource.com/c/go/+/527135 Auto-Submit: Matthew Dempsky <mdempsky@google.com> Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com> Reviewed-by: Keith Randall <khr@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> (cherry picked from commit e3ce312) Reviewed-on: https://go-review.googlesource.com/c/go/+/534916 Reviewed-by: Matthew Dempsky <mdempsky@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
…scall2 in code Currently, set_crosscall2 takes the address of crosscall2 without using the GOT, which, on some architectures, results in a PC-relative relocation (e.g. R_AARCH64_ADR_PREL_PG_HI21 on ARM64) to the crosscall2 symbol. But crosscall2 is dynamically exported, so the C linker thinks it may bind to a symbol from a different DSO. Some C linker may not like a PC-relative relocation to such a symbol. Using a local trampoline to avoid taking the address of a dynamically exported symbol. It may be possible to not dynamically export crosscall2. But this CL is safer for backport. Later we may remove the trampolines after unexport crosscall2, if they are not needed. Fixes golang#63509. Updates golang#62556. Change-Id: Id28457f65ef121d3f87d8189803abc65ed453283 Reviewed-on: https://go-review.googlesource.com/c/go/+/533535 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> (cherry picked from commit 872d718) Reviewed-on: https://go-review.googlesource.com/c/go/+/534915 Reviewed-by: David Chase <drchase@google.com>
…/http2 After CL 534295 was merged to fix a CVE it introduced an underflow when we try to decrement sc.curHandlers in handlerDone. Pull in a fix from x/net/http2: http2: fix underflow in http2 server push https://go-review.googlesource.com/c/net/+/535595 For golang#63511 Fixes golang#63560 Change-Id: I5c678ce7dcc53635f3ad5e4999857cb120dfc1ab GitHub-Last-Rev: 587ffa3 GitHub-Pull-Request: golang#63561 Reviewed-on: https://go-review.googlesource.com/c/go/+/535575 Run-TryBot: Mauri de Souza Meneguzzo <mauri870@gmail.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: David Chase <drchase@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit 0046c14) Reviewed-on: https://go-review.googlesource.com/c/go/+/537996 Reviewed-by: Cherry Mui <cherryyz@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
…Windows paths On Windows, A root local device path is a path which begins with \\?\ or \??\. A root local device path accesses the DosDevices object directory, and permits access to any file or device on the system. For example \??\C:\foo is equivalent to common C:\foo. The Clean, IsAbs, IsLocal, and VolumeName functions did not recognize root local device paths beginning with \??\. Clean could convert a rooted path such as \a\..\??\b into the root local device path \??\b. It will now convert this path into .\??\b. IsAbs now correctly reports paths beginning with \??\ as absolute. IsLocal now correctly reports paths beginning with \??\ as non-local. VolumeName now reports the \??\ prefix as a volume name. Join(`\`, `??`, `b`) could convert a seemingly innocent sequence of path elements into the root local device path \??\b. It will now convert this to \.\??\b. In addition, the IsLocal function did not correctly detect reserved names in some cases: - reserved names followed by spaces, such as "COM1 ". - "COM" or "LPT" followed by a superscript 1, 2, or 3. IsLocal now correctly reports these names as non-local. For golang#63713 Fixes golang#63715 Fixes CVE-2023-45283 Fixes CVE-2023-45284 Change-Id: I446674a58977adfa54de7267d716ac23ab496c54 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2040691 Reviewed-by: Roland Shoemaker <bracewell@google.com> Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2072596 Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/540276 Auto-Submit: Heschi Kreinick <heschi@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Change-Id: I3d607ba9f701a76a46f3ab3223fa83e5c517d285 Reviewed-on: https://go-review.googlesource.com/c/go/+/540517 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Gopher Robot <gobot@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
… rlimit_test.go Tests in rlimit_test.go exist to test the behavior of automatically bumping RLIMIT_NOFILE on Unix implemented in rlimit.go (issue golang#46279), with darwin-specific behavior split out into rlimit_darwin.go and the rest left empty in rlimit_stub.go. Since the behavior happens only on Unix, it doesn't make sense to test it on other platforms. Copy rlimit.go's 'unix' build constraint to rlimit_test.go to accomplish that. Leave out the simplification of the build constraint in rlimit_stub.go so that this CL remains a test-only fix. In particular, this fixes a problem where TestOpenFileLimit was failing in some environments when testing the wasip1/wasm port. The RLIMIT_NOFILE bumping behavior isn't implemented there, so the test was testing the environment and not the Go project. Updates golang#46279. For golang#61116. Fixes golang#63994. Change-Id: Ic993f9cfc021d4cda4fe3d7fed8e2e180f78a2ca Cq-Include-Trybots: luci.golang.try:go1.21-wasip1-wasm_wasmtime Reviewed-on: https://go-review.googlesource.com/c/go/+/539435 Reviewed-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com> Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> (cherry picked from commit b7cbcf0) Reviewed-on: https://go-review.googlesource.com/c/go/+/540615 Reviewed-by: Heschi Kreinick <heschi@google.com> Auto-Submit: Heschi Kreinick <heschi@google.com>
…in dead store elimination Update golang#63743 Change-Id: I163c6038c13d974dc0ca9f02144472bc05331826 Reviewed-on: https://go-review.googlesource.com/c/go/+/538595 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> Reviewed-by: Keith Randall <khr@google.com> (cherry picked from commit 43b57b8) Reviewed-on: https://go-review.googlesource.com/c/go/+/538857 Auto-Submit: Heschi Kreinick <heschi@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com>
…s after the nil check Have nil checks return a pointer that is known non-nil. Users of that pointer can use the result, ensuring that they are ordered after the nil check itself. The order dependence goes away after scheduling, when we've fixed an order. At that point we move uses back to the original pointer so it doesn't change regalloc any. This prevents pointer arithmetic on nil from being spilled to the stack and then observed by a stack scan. Fixes golang#63743 Change-Id: I1a5fa4f2e6d9000d672792b4f90dfc1b7b67f6ea Reviewed-on: https://go-review.googlesource.com/c/go/+/537775 Reviewed-by: David Chase <drchase@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Keith Randall <khr@google.com> (cherry picked from commit 962ccbe) Reviewed-on: https://go-review.googlesource.com/c/go/+/538717 Auto-Submit: Heschi Kreinick <heschi@google.com> Reviewed-by: Heschi Kreinick <heschi@google.com>
… splice to avoid inconsistency with O_NONBLOCK Fixes golang#63801 Updates golang#59041 Updates golang#63795 Details: golang#59041 (comment) Change-Id: Id3fc1df6d86b7c4cc383d09f9465fa8f4cc2a559 Reviewed-on: https://go-review.googlesource.com/c/go/+/536015 Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@google.com> (cherry picked from commit 40cdf69) Reviewed-on: https://go-review.googlesource.com/c/go/+/538117 Auto-Submit: Heschi Kreinick <heschi@google.com> Reviewed-by: Mauri de Souza Meneguzzo <mauri870@gmail.com> Reviewed-by: Heschi Kreinick <heschi@google.com>
…lar in Stat and Lstat Prior to CL 460595, Lstat reported most reparse points as regular files. However, reparse points can in general implement unusual behaviors (consider IO_REPARSE_TAG_AF_UNIX or IO_REPARSE_TAG_LX_CHR), and Windows allows arbitrary user-defined reparse points, so in general we must not assume that an unrecognized reparse tag represents a regular file; in CL 460595, we began marking them as irregular. As it turns out, the Data Deduplication service on Windows Server runs an Optimization job that turns regular files into reparse files with the tag IO_REPARSE_TAG_DEDUP. Those files still behave more-or-less like regular files, in that they have well-defined sizes and support random-access reads and writes, so most programs can treat them as regular files without difficulty. However, they are still reparse files: as a result, on servers with the Data Deduplication service enabled, files could arbitrarily change from “regular” to “irregular” without explicit user intervention. Since dedup files are converted in the background and otherwise behave like regular files, this change adds a special case to report DEDUP reparse points as regular. Fixes golang#63764. Updates golang#63429. No test because to my knowledge we don't have any Windows builders that have the deduplication service enabled, nor do we have a way to reliably guarantee the existence of an IO_REPARSE_TAG_DEDUP file. (In theory we could add a builder with the service enabled on a specific volume, write a test that encodes knowledge of that volume, and use the GO_BUILDER_NAME environment variable to run that test only on the specially-configured builders. However, I don't currently have the bandwidth to reconfigure the builders in this way, and given the simplicity of the change I think it is unlikely to regress accidentally.) Change-Id: I649e7ef0b67e3939a980339ce7ec6a20b31b23a1 Cq-Include-Trybots: luci.golang.try:go1.21-windows-amd64-longtest Reviewed-on: https://go-review.googlesource.com/c/go/+/538218 Reviewed-by: Alex Brainman <alex.brainman@gmail.com> Reviewed-by: David Chase <drchase@google.com> Auto-Submit: Heschi Kreinick <heschi@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
bradfitz
approved these changes
Nov 9, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull in
release-branch.go1.21up to ed817f1 which is 1.21.4 tag. Not pulling in the extra few commits on the branch after that tag.