Print a more useful string then 'devel'

[oliv3r]

git describe --always --dirty is pretty amazing to help identify (un)tagged versions. It would be nice if shfmt would do this 'compile' time during release mode. I'm not familiar enough with go, but in C we tend to have a ./configure time generated version.h, which get the content at compile time. Doing a sed on the string would also work, but would make the file 'dirty', so better not that.

Especially when pulling the docker containers with the 'latest' tag, having identify-ability is quite useful, 'which version was used for this job, well version a.b.c-21-gdeadbeef is far more interesting then 'devel', even when the later statement is completely true :)

[mvdan]

I completely agree, but Go doesn't have a way to insert arbitrary code as part of its build process - and that is by design. Imagine if running go get github.com/foo/bar could scan your home directory.

Modules have introduced versioning to builds, but unfortunately, a local go build doesn't include such information just yet. That will hopefully get fixed soon; see golang/go#29814. That would definitely be the right solution.

Until then, we have two alternatives, but they both kinda stink:

• Adding a "build script" like build.sh or Makefile, and asking users to use that instead of go build directly. Many projects do this, but it's opt-in, and generally frowned upon in the Go ecosystem.
• Updating master every few commits to change what its version is. This is kinda sloppy, and only gives an approximate version.

We do have a third option, which is to try to build the docker images in a way that includes good version information. This wouldn't fix regular go get or go install methods, but at least it could fix the latest docker image to include something.

[kaey]

If you run GOBIN=$(pwd) GO111MODULE=on go get mvdan.cc/sh/v3/cmd/shfmt proper version information is recorded in a binary and you can look at it via

$ go version -m shfmt
shfmt: go1.14
        path    mvdan.cc/sh/v3/cmd/shfmt
        mod     mvdan.cc/sh/v3  v3.0.2  h1:NU+UpTZHRdIZ9igRo8zi/7+5/NpetYlhlyDhz1/AdMM=
        dep     github.com/pkg/diff     v0.0.0-20190930165518-531926345625      h1:b5m9ubdpxvfhiJnF64/W1rUTSUOzKHipjy5wOWsZCBM=
        dep     golang.org/x/crypto     v0.0.0-20191002192127-34f69633bfdc      h1:c0o/qxkaO2LF5t6fQrT4b5hzyggAkLLlCUjqfRxd8Q4=
        dep     golang.org/x/sys        v0.0.0-20191008105621-543471e840be      h1:QAcqgptGM8IQBC9K/RC4o+O9YmqEm0diQn9QmZw/0mU=
        dep     mvdan.cc/editorconfig   v0.1.1-0.20191109213504-890940e3f00e    h1:mpgBTmbRe4vLIQIWTPipFCM2P97pEnkFSthekRyxqfY=

or https://godoc.org/runtime/debug#ReadBuildInfo

Docker build can also be updated to use go get instead of depending on vcs.

[oliv3r]

So for the normal 'manual' build process (go build) I agree to wait until go sorts this upstream.
For the containers, the easiest way is to:
sed "s|devel|$(git describe --always --dirty)|g" version.go.in > version.go
but then I don't know, can you 'source' this version.go file only if it exists? So that version = "devel" if version.go didn't exist, but if it does, whatever was in version.go would be used?
That's go magic I am unfamiliar with :) (or go in general)
The other way, which for the pipeline is 'ok' as we know we won't ever have dirty files due to clean checkouts of the CI, is to:
sed -i 's| version = "devel"| version = "'"$(git describe --always)"'"|g' main.go With a slightly smarter regex for the dev (note that github comments strip tabs here, because they are rude :p)
The substitution is quite sensitive, and breakage won't be noticed (sed won't return an error if it can't substitute anything)

[mvdan]

@kaey that's not a good solution because it doesn't build what's actually in git. You're just building master. What if I want to build a tag, or a commit that hasn't been pushed to the remote? It could work for the Docker builds because they only happen to build master and tags which have been pushed out, but it's not a general solution.

@oliv3r a pre-compilation step for Docker is the third method I described earlier. It could work, though it would still be very hacky.

[kaey]

You're just building master

No. It builds whatever you specify after @ which can be a tag or a commit. Defaults to latest published tag, not master.

What if I want to build a tag, or a commit that hasn't been pushed to the remote?

You want to ship a binary without sources publicly available? Is it useful? For development you can still use a fallback to -ldflags -X

[mvdan]

No. It builds whatever you specify after @ which can be a tag or a commit. Defaults to latest published tag, not master.

Right - by master, I really just meant what's in the public git repository.

You want to ship a binary without sources publicly available? Is it useful? For development you can still use a fallback to -ldflags -X

You want the binaries to be consistent, ideally. Right now, the version is consistently static, and the build process is simple. Having two or three ways to build a binary is an option, but it's far from ideal.

If we decide to do anything here, I think the most sensible would be to somehow force the Go build system to embed module version information into the build. If that requires internet access it's a bummer, but I don't think there's a way around it today.

[kaey]

Having a way to set version string manually also gives you an ability to lie. You say it's tag v3.X.Y, but binary was actually built from later commit in release branch v3.X. Or worse, downstream patches - you receive a bug report just to find later that it was on a buggy patched version. This is not even a hypothetical problem, see top-level comment by souprock here https://news.ycombinator.com/item?id=19935648.

Go takes away that ability from you. Want to patch? Make your fork publicly available and add replace directive to go.mod.

[mvdan]

Yeah, we agree on what we want. I just want to avoid forcing a build to do a network roundtrip if one wants the binary to have meaningful version information. Though perhaps that's the best we can have right now.

[oliv3r]

@mvdan yeah, I was going with your pre-compilation step. Or maybe 'devsplaining' it ;)

It's ugly, it's hacky, but for the docker container, would call it acceptable.

What I was after I guess in my comment, if you have a preferred way to see it in a MR. I'd prefer to keep the --dirty flag, to truely mark the build, but the sed will always dirty the build, so it would have to be done in a version.go file that's not tracked.

[kaey]

I just want to avoid forcing a build to do a network roundtrip

You need to have your deps vendored for that ¯_(ツ)_/¯

[oliv3r]

Lets start small however, if you are building your own thing, well then you aught to be smart enough to go to your source tree and find things.
Pre-compiled dev packages should come from the CI, so can be pre-processed. Docker images, come from the CI/Hub, so can be pre-processed. Also These two are the cases where the version information is important. So lets focus on those for this ticket?

[mvdan]

Sure, if anyone wants to give this a go or send a PR, that's welcome.

[mvdan]

This is fixed for the time being. All published binary and docker artifacts should be properly versioned from now on.

The only gaps left at the moment are go get mvdan.cc/sh/v3/cmd/shfmt@master, and git clone https://github.com/mvdan/sh && cd sh && go install ./cmd/shfmt. Both will get the devel version.

If we moved to the module build info, the first would start getting good version information, but not the second. So I don't think it's hugely important that we move to this fancier method today, as it's still not foolproof.

golang/go#37475 seems to be getting traction now, so I think we might see some progress in the coming months.