Feature request: allow manually setting Mod Time

[veqryn]

Thank you for this package.

I currently use it to host a swagger.json file that contains the API documentation for my gRPC server (which has a REST->gRPC Gateway on it, hence the need for swagger).

We re-generate code from all of our protobuf files whenever they change, and whenever any mocked interfaces change (we just have a container that runs go generate against everything in the repo).

Because of this, the timestamps on our files (including the swagger.json files) are constantly changing, even though the contents of the files does not change.

This means that our vfsgen embedded doc files always show changes to the modTime values, even though nothing else changed.

I would like to disable this, and the easiest way would probably be to create an option to manually set the modTime to a specific value. I'm open to other idea too though.

thanks!

[dmitshur]

Hi @veqryn,

Thanks for the feature request. Fortunately, this is already possible to achieve without changes to vfsgen. vfsgen was designed to enable this kind of customization (and more) through interfaces.

Please see #31 (comment) and the code snippet inside. It should let you achieve what you want.

Edit: Since you want to set modtime to zero rather than modify it from zero, the Stat method would instead look more like this:

...

func (f modTimeFile) Stat() (os.FileInfo, error) {
    fi, err := f.File.Stat()
    if err != nil {
        return nil, err
    }
    return modTimeFileInfo{fi}, nil
}

type modTimeFileInfo struct {
    os.FileInfo
}

func (modTimeFileInfo) ModTime() time.Time {
    return time.Time{}
    // or any custom logic you'd like to implement for your needs
    // (perhaps use the modtime of the protobuf files you're generating from)
}

This is a common feature request, and so I plan to document it in the README so it's more visible. Issue #31 tracks that task.

[veqryn]

Hi @dmitshur, I'm just a little lost, but how does this work to prevent the generated file from having a different mod timestamp?

To clarify, I want the generated file to be the same each time I generate it (even if it has been modified/saved multiple times between), that way git stops telling me there are changes when the content has stayed the same.

[dmitshur]

Can you share how you're using vfsgen right now? If so, it'll be easier for me to show you what I mean. If not, I can try to explain anyway by coming up with an example.

[veqryn]

Sure.

1. I run vfsgen to generate embedded golang files.
2. Commit the generated embedded golang files to our repo.
3. Sometime later, I or someone else will recompile all of our various proto files. We have a lot, so we just recompile all of them, regardless of whether they have changed or not. Protoc is deterministic and has stable output, so if the contents of a proto file have not changed, then the outputted generated golang file from protoc will show no differences (ie: git diff shows nothing).
4. Re-run vfsgen to re-create the embedded golang files.
5. Go to commit to repo, and notice that vfs shows lots of timestamps have changed, even though the contents have not changed at all (ie: git diff shows lots of changed files). The timestamps have changed because protoc output identical but new files when it ran.

I would like it so that vfsgen had an option to overwrite the modtime when it generates files, in the actual files it generates, that way git doesn't show differences all the time when i go to commit those files.

(This has nothing to do with an http file server, as I don't use that at all in any step above.)

[veqryn]

My code looks like this:

Golang executable to generate embedded golang files (because I couldn't understand what https://github.com/shurcooL/vfsgen/blob/master/cmd/vfsgendev/main.go was doing or how to use it):

package main

import (
	"flag"
	"fmt"
	"net/http"

	"github.com/shurcooL/vfsgen"
)

func main() {

	dir := flag.String("dir", "", "The directory to recursively generate vfs / embedded-bindata for")
	outfile := flag.String("outfile", "", "The file path and name (include extension) to output the generated file")
	pkg := flag.String("pkg", "", "The package name to give the vfs file")
	tags := flag.String("tags", "", "The build tags to give the vfs generation")
	variable := flag.String("variable", "", "The variable name to give the vfs (start with a capital letter if you want it exported)")
	comment := flag.String("comment", "", "The comment to give the variable")

	flag.Parse()

	fmt.Printf("vfsgen for directory: %s; output to: %s; package name: %s; build tags: %s; variable name: %s; comment: %s\n", *dir, *outfile, *pkg, *tags, *variable, *comment)
	err := vfsgen.Generate(http.Dir(*dir), vfsgen.Options{

		// Filename of the generated Go code output (including extension)
		Filename: *outfile,

		// PackageName is the name of the package in the generated code
		PackageName: *pkg,

		// BuildTags are optional build tags to give the generated code
		BuildTags: *tags,

		// VariableName is the name of the http.FileSystem variable in the generated code
		VariableName: *variable,

		// VariableComment is the comment of the http.FileSystem variable in the generated code
		VariableComment: *comment,
	})

	if err != nil {
		panic(err)
	}
}

An example of my go generate statements:

// Generate golang protobuf/grpc/gateway and swagger docs
//go:generate protoc -I=./include -I=../vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I=../vendor/github.com/lyft -I=../vendor/github.com/grpc-ecosystem/grpc-gateway -I=./ecm/v2 --go_out=plugins=grpc:./ecm/v2/go --grpc-gateway_out=logtostderr=true:./ecm/v2/go --swagger_out=logtostderr=true:./ecm/v2/swagger --validate_out=lang=go:./ecm/v2/go ./ecm/v2/rp_ecm_v2.proto

// Generate embedded docs
// TODO: get vfsgen to ignore or skip if only the file timestamp has changed OR somehow overwrite the timestamp
//go:generate go run ./vfsgen/vfsgen.go --dir=./ecm/v2/swagger/ --outfile=./ecm/v2/embedded_docs/embedded_docs.go --pkg=embedded_docs --variable=Docs -comment "Docs statically implements an embedded virtual filesystem provided to vfsgen. To access the swagger file, use path: '/rp_ecm_v2.swagger.json'"

And then some code that is using the particular embedded golang file:

	// Open the embedded swagger json
	f, err := embedded_docs.Docs.Open("/rp_ecm_v2.swagger.json")
	// .... stuff ...

[dmitshur]

Thanks.

The relevant part in the snippet is here:

vfsgen.Generate(http.Dir(*dir), ...

That means you're currently using http.Dir as the http.FileSystem interface implementation that you provide to vfsgen.Generate.

Your original request was to be able to override the underlying filesystem (on disk) file mod times to be all zeros. So, you just need to wrap http.Dir(*dir) with the http.FileSystem wrapper I showed above. The wrapper will override file mod times to be zero, which is what vfsgen.Generate will then use.

It can look something like this:

package main

import (...)

func main() {
    ...
    // Override all file mod times to be zero using modTimeFS.
    var inputFS http.FileSystem = modTimeFS{
        fs: http.Dir(*dir),
    }
    err := vfsgen.Generate(inputFS, vfsgen.Options{
    ...
}

// modTimeFS is an http.FileSystem wrapper that modifies
// underlying fs such that all of its file mod times are set to zero.
type modTimeFS struct {
    fs http.FileSystem
}

func (fs modTimeFS) Open(name string) (http.File, error) {
    f, err := fs.fs.Open(name)
    if err != nil {
        return nil, err
    }
    return modTimeFile{f}, nil
}

type modTimeFile struct {
    http.File
}

func (f modTimeFile) Stat() (os.FileInfo, error) {
    fi, err := f.File.Stat()
    if err != nil {
        return nil, err
    }
    return modTimeFileInfo{fi}, nil
}

type modTimeFileInfo struct {
    os.FileInfo
}

func (modTimeFileInfo) ModTime() time.Time {
    return time.Time{}
    // or any custom logic you'd like to implement for your needs
    // (perhaps use the modtime of the protobuf files you're generating from)
}

[veqryn]

Ok, I will give this a try.

Could you also explain a bit of how https://github.com/shurcooL/vfsgen/blob/master/cmd/vfsgendev/main.go works, and how to properly use it? Can it be used for everything my script above is currently doing?

[dmitshur]

Have you read the documentation for vfsgendev at https://github.com/shurcooL/vfsgen#vfsgendev-usage? You may also need to read the section above, at https://github.com/shurcooL/vfsgen#go-generate-usage, since it’s being referenced.

Let me know if something there is unclear or if it doesn’t sufficiently explain what vfsgendev is doing.

You should be able to use it as long as you’re following a compatible workflow.