version: come up with good default implementation

[bradfitz]

When we do tailscale binary releases, we stamp version info into the binaries with the Go linker.

But for people just using "go get" the default way, we don't have that info.

Since Go modules, we at least see the summary of all our deps (and Go's version itself; I'm running a devel build) automatically stamped into the binaries:

$ go version -m ~/bin/tailscaled
/home/bradfitz/bin/tailscaled: devel +e7f9e17b79 Tue Jan 28 22:08:43 2020 +0000
        path    tailscale.com/cmd/tailscaled
        mod     tailscale.com   (devel)
        dep     github.com/apenwarr/fixconsole  v0.0.0-20191012055117-5a9f6489cc29      h1:muXWUcay7DDy1/hEQWrYlBy+g0EuwT70sBHg65SeUc4=            
        dep     github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e      h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=    
        dep     github.com/google/go-cmp        v0.4.0  h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
        dep     github.com/klauspost/compress   v1.9.8  h1:VMAMUUOh+gaxKTMk+zqbjsSjsIcUcL/LF4o63i82QyA=
        dep     github.com/mdlayher/netlink     v1.1.0  h1:mpdLgm+brq10nI9zM1BpX1kpDbh3NLl3RSnVq6ZSkfg=
        dep     github.com/pborman/getopt       v0.0.0-20190409184431-ee0cd42419d3      h1:YtFkrqsMEj7YqpIhRteVxJxCeC3jJBieuLr0d4C4rSA=
        dep     github.com/tailscale/wireguard-go       v0.0.0-20200213180345-a7c4b7719b1d      h1:LVJovgZxbmPxtY6kJm4vwMtk0HpcNeI+vU2jB3T8M40=
        dep     golang.org/x/crypto     v0.0.0-20200210222208-86ce3cb69678      h1:wCWoJcFExDgyYx2m2hpHgwz8W3+FPdfldvIgzqDIhyg=
        dep     golang.org/x/net        v0.0.0-20200202094626-16171245cfb2      h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI=
        dep     golang.org/x/oauth2     v0.0.0-20200107190931-bf48bf16ab8d      h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
        dep     golang.org/x/sys        v0.0.0-20200217220822-9197077df867      h1:JoRuNIf+rpHl+VhScRQQvzbHed86tKkqwPMV34T8myw=
        dep     rsc.io/goversion        v1.2.0  h1:SPn+NLTiAG7w30IRK/DKp1BjvpWabYgxlLp/+kx5J8w=

We can get that at runtime (using https://godoc.org/rsc.io/goversion/version) but for better or worse, that doesn't include that the Git hash of the top-level module, only the deps.

So one thing we could do is add a fake dep to another module (like github.com/tailscale/version-horizon) that we have a bot auto-advance every $INTERVAL, so at least we have a rough date range of what binary people are running when they're running unofficial binaries. Or, if we're already having a bot do this, avoid that indirection and just have the bot auto-commit the latest date to version/version.go every $INTERVAL instead.

Low priority, but we'll probably want better data in the future.

/cc @apenwarr @crawshaw @danderson

[bradfitz]

If others want to see what's available, you can try:

diff --git a/version/version.go b/version/version.go
index 1bc61d6..3a2562e 100644
--- a/version/version.go
+++ b/version/version.go
@@ -7,5 +7,24 @@
 // Package version provides the version that the binary was built at.
 package version
 
-const LONG = "LONGVER-TODO"
-const SHORT = "SHORTVER-TODO"
+import (
+       "fmt"
+       "os"
+
+       "rsc.io/goversion/version"
+)
+
+var LONG = "VERSION-UNKNOWN"
+var SHORT = "VERSION-UNKNOWN"
+
+func init() {
+       n, err := os.Executable()
+       if err != nil {
+               return
+       }
+       v, err := version.ReadExe(n)
+       if err != nil {
+               println(err.Error())
+       }
+       println(fmt.Sprintf("Version: %+v\n", v))
+}

Then go run ./cmd/tailscaled -h.

[danderson]

I'm getting an increasing number of inquiries about this in the context of OSS packaging in distros. It's a problem that we can't ship sensible version info if you're building from just the OSS repo.

I'm going to take a swing at fixing this ahead of the Go compiler providing us more useful information, at least well enough that distro packagers can embed version info as they build, while we can continue to provide corp version info for the non-free programs.

[bradfitz]

What's your plan?

[danderson]

Rough plan, not tested out yet:

• Make a richer version type that embeds more information (short version per git describe, git commit hash, and an "is OSS version" bool that means "do the other fields refer to the OSS repo or the corp one?").
• Add a shell script at the root of the OSS repo that does what redo currently does in our corp repo - do a bunch of git introspection to generate version data. Distro packagers can be given documentation to run that when they build packages (and we can make the CLI complain at people when version info is missing). Doesn't solve the case of an individual building by themselves with go get, but I'm okay waiting on Go for that case. We can continue to do some ratcheting date-based fallback for that case if we really want to.
• Add tailscale.com/version.Set(v), to allow our non-oss binaries to override the OSS build info with their own version object. This lets the Windows and Apple builds provide build info from the corp repo, while still letting all our code uniformly use tailscale.com/version for version information.
• Our corp repo continues to use redo to generate its own private version info, and our non-oss binaries call version.Set early in execution.

Works in the following use cases:

• Us building packages from the corp repo works, because oss/cmd/* won't override the OSS version with the corp version info. Need a tiny bit of adjustment so that redo regenerates oss version info, trivial. Slight risk of version confusion over wireguard-go, since we do go.mod replace shenanigans, but in practice we keep the go.mod version and our replaced version in sync anyway, so should be fine.
• OSS distro packagers (e.g. Arch, Nix) already do complicated non-go-get type build processes, it's trivial for them to insert ./generate-version.sh before the compilation phase to get version info that exactly matches what we embed in packages we make.
• Us building the non-oss binaries (Windows, macOS, iOS) still works, because we have a corp version stamp and corp/cmd/* binaries override the OSS stamp with the corp one at startup.
• Richer version info means we can format versions like oss-0.96-123 or corp-0.96-123 (pls bikeshed exact format in PR later :) ), and avoid descriptor confusion when we're staring at stats. Having this data in a structured type means we can also build tooling that doesn't have to parse version strings.

That's the general shower thoughts I had this morning. The context is that I reviewed the initial Nix/NixOS derivations for Tailscale last night, which are going to ship to NixOS users with the crappy fallbackversion info. Separately this morning, an Arch user pointed out that some of my AUR packages violate AUR convention by shipping binaries, and I need to switch to building from source - which would force me to implement some "version for OSS" hack anyway. Given those, I want to solve that problem ONCE AND FOR ALL upstream, so I can give downstream distro packagers uniform instructions for how to build binaries with good version data.

[bradfitz]

So, another bespoke Makefile / make.bash / make.go for oss. (golang/go#37475)

That's fine for now, for people who want to use it.

I have a crazier idea for the go get users: look up the version from a network service at runtime based on the program taking a signature of its own binary (using debug/elf, etc). The network service would serve effectively a map[Signature]struct{GitCommit, CommitTime} that would be populated by a daemon repeatedly building each commit with a dozen latest/most populate versions of the compiler. I might even just build that as a non-Tailscale service on Cloud Run that anybody could use for free.

[danderson]

Pretty much, yeah. It's definitely not what I want to do, but the third-party distros need something to embed version information we can reason about for debugging.

A versioning service sounds interesting, though I question whether you can actually account for the variety of weirdery out there (e.g. all Nix binaries have altered library load paths and whatnots, so the compilers are likely to produce slightly different output). I think it's possible, but it'd be fiddly.

[Foxboron]

Yo,

This issue has been silent since March and assigned a low priority. But with the release of version 1.0 I think you need to reconsider.

Today if I want to download tailscale there are a few binaries I can select. But no signatures. That isn't a problem in theory, go binaries are reproducible. However there is no clear way to figure out how they are built. Looking at the built version

λ tailscale_1.0.5_amd64 » ./tailscale --version          
1.0.5-g31b5dec0a

I can checkout the 1.0.5 tag, and try build it with -trimpath. But the commit g31b5dec0a is not present in this repository. Even injecting the correct version into the binary gives me a different checksum of the binaries.

What are the major roadblocks for embedding version information, and streamlining the build process? It would help reproducible builds for your project and distro packagers (semi-relevant as I'm interested distributing tailscale in the [community] repo of Arch).

[bradfitz]

This got blocked on golang/go#37475 which is accepted but not yet implemented in upstream Go.

Our -g31b5dec0a suffix means "git commit" 31b5dec0a, but that's a git commit of our parent repo that has all our other repos (e.g. https://github.com/tailscale/tailscale-android, the closed-source iOS app) as git submodules.

I would suggest solutions but I'm not sure your goal. You mentioned several somewhat separate concerns.

Our builds don't contain anything closed-source mixed in from the parent repo other than the version string, so if you set that, use -trimpath, and build with the same version of Go that we do (https://github.com/tailscale/go, the commit of which we print out at start-up), then you should get the same output bytes.

For Arch builds, I'd just make the version be "1.0.5-g<git commit of tailscale/tailscale repo>"

[Foxboron]

I would suggest solutions but I'm not sure your goal. You mentioned several somewhat separate concerns.

My goal is to have tailscale reproducible. This is important both when distributing binaries with no signatures, and for packagers to have a streamlined build process. My impression was that this versioning problem was the reason why I can't have a make build in the project. But it seems like I was mistaken.

Our builds don't contain anything closed-source mixed in from the parent repo other than the version string, so if you set that, use -trimpath, and build with the same version of Go that we do (https://github.com/tailscale/go, the commit of which we print out at start-up), then you should get the same output bytes.

So I tried. I fetched https://github.com/tailscale/go/releases/tag/build-56db76510f9640d2ad652f206ae6e41c1a5d63ca which is what 1.0.5-g31b5dec0a should be built with (output of running tailscale). Ran ./src/make.bash in the go source. Tried building tailscale by inserting the correct version and running the following compilation options which is purely guessed as it's not dynamically compiled.

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 ~/Downloads/go/bin/go build -trimpath ./cmd/tailscale

λ ~ » sha256sum Downloads/tailscale_1.0.5_amd64/tailscale 
05ce75560631d6723a73fdac7a6711f8d5e142dad8bae7c2b14f8c13c61077b6  Downloads/tailscale_1.0.5_amd64/tailscale
λ ~ » sha256sum Git/prosjekter/Go/tailscale/tailscale 
4791c43c5f1f24f3177731cd93bbd0f56d764b03a27fea919a0465acfa9a6326  Git/prosjekter/Go/tailscale/tailscale
λ ~ » ./Git/prosjekter/Go/tailscale/tailscale --version
1.0.5-g31b5dec0a
λ ~ » ./Downloads/tailscale_1.0.5_amd64/tailscale --version
1.0.5-g31b5dec0a

I don't know how the bootstrapping process affects the results, and I'm unsure if it's feasibly without you redistributing the compiler itself (what guarantees does the go compiler have?). Please do point out if I have made any mistakes with this, I'm just guessing my way around at this point.

I think this is a problem, but I'm unsure if it's related to this issue specifically. Currently there doesn't seem to be a good way to prove how the distributed tailscale is being built.

[apenwarr]

I think we supply some specific build tags too nowadays, right?

[bradfitz]

Oh, we also set the tailscale_go build tag in our builds.

Relevant bits from our build system:

redo  out/x86_64-linux/oss/cmd/tailscale/tailscale (resumed)
#!/bin/sh -e
# Per-platform settings for the go compiler.
# Auto-generated by go.od. Do not edit!
S='/home/bradfitz/src/tailscale.io'
export GO='/home/bradfitz/.cache/tailscale-go/bin/go'
export CC='cc'
export CFLAGS=''
export GO_LDFLAGS='-extldflags ""'
export GOOS=''
export GOARCH=''
export GOARM=''
export CGO_ENABLED='0'

# Necessary because CI forces GOROOT and we need
# to override it back.
export GOROOT=
redo    (../../../../../oss/version/all)                                                                                                                                            
redo    (../../../../../oss/version/version.h)
redo    oss/version/describe.txt
redo    oss/version/describe.txt (done)                                                                                                                                             

redo  out/x86_64-linux/oss/cmd/tailscale/tailscale (done)                                                                                                                           

....

        '$GO' build \\
            -v -trimpath -tags=redo,tailscale_go \\
            -ldflags "\$GO_LDFLAGS" \\
            -o "\$t" \\
            "\$pkg"

[bradfitz]

(This is all pretty off topic for this issue, though.)

[Foxboron]

(Thanks,!I'm aware it's offtopic. I'm happy to take up this discussion wherever appropriate.)

[bradfitz]

If you like Slack, see https://github.com/tailscale/tailscale/wiki/Community for a link. Otherwise if you prefer GitHub, just create a new issue here? Perhaps titled "Document how to generate the same binaries Tailscale distributes"

[DentonGentry]

I think the main intent of this issue has been addressed by https://github.com/tailscale/tailscale/blob/main/build_dist.sh
Specific questions about reproducibility split out into #779.

[DentonGentry]

https://go-review.googlesource.com/c/go/+/353930 will embed the git hash into generated binaries. I think that will be sufficient to cover the remaining case of concern in this issue, of people building from source.

[bradfitz]

Waiting on Go 1.18 in couple weeks.