Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement kaitai.Struct common interface #30

Merged
merged 4 commits into from Mar 31, 2024
Merged

Conversation

GreyCat
Copy link
Member

@GreyCat GreyCat commented Mar 29, 2024

Analogous to KaitaiStruct in all other languages (like Java), this adds kaitai.Struct interface, mandating Kaitai_IO() method on all ksc-generated types in go.

kaitai/struct_test.go Outdated Show resolved Hide resolved
kaitai/struct_test.go Outdated Show resolved Hide resolved
Copy link
Member

@generalmimon generalmimon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to add a (relatively heavy) dependency github.com/stretchr/testify just to make literally 2 lines of test code slightly easier to write, but if you think it's not a problem, then it's not a problem.

@GreyCat
Copy link
Member Author

GreyCat commented Mar 29, 2024

@jchv @cugu If you're still around and interested in reviewing this — any strong objections?

Copy link
Contributor

@jchv jchv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Threw in my 2c. I don't have strong opinions but the general idea seems fine.

kaitai/struct.go Outdated
// This is deliberately named with a `Kaitai_` prefix and underscore to avoid conflicts
// with other methods that may result from the attributes in Kaitai Struct type, e.g.
// is a user will define attribute `i_o` this will conflict with the method `IO()`.
Kaitai_IO() *Stream
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. I think this is fairly unideal, though it's not obvious how one could do better without changing the attribute name mangling.

Other potential bikeshed paint colors: IO_(), KaitaiIO_()

Though maybe special casing kaitai_i_o in the compiler is not a bad idea. It's not like this is likely to come up in non-contrived examples anyways...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IO_()

This is my new favorite. I knew that if you start the identifier with an underscore _, then it doesn't start with a capital letter and won't be exported, but I didn't think of ending the identifier with an underscore _. It's simple and I suppose it potentially works better with autocompletion (assuming that only identifiers starting with what you type are shown) - in case a user would sometimes need to refer to _io, _parent or _root, they can start typing I, Par or Ro and will be offered IO_(), Parent_() and Root_() respectively, and they don't have to remember that there is a special Kaitai_ prefix.

Though maybe special casing kaitai_i_o in the compiler is not a bad idea. It's not like this is likely to come up in non-contrived examples anyways...

This may be true in practice, but I still advocate that there should be no "holes" in the language. There are already some holes in our Go implementation that I'm not happy about (e.g. the Read method - hello_world.go:18, which will clash with a user-specified instance called read), but at least I'd avoid adding more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also like the idea of IO_(). Just to get an Go expert opinion: how bad it will be from ecosystem perspective — e.g. it will generate code that will have to break varnames linting? How serious it is?

On the side note, to be honest, I won't mind breaking compat and turning Go Read method into Read_ for example (given that vast majority of other languages generates _read() or some similar).

kaitai/struct_test.go Show resolved Hide resolved
@generalmimon
Copy link
Member

generalmimon commented Mar 30, 2024

The golangci-lint CI job logs are interesting:

prepare environment
  Checking for go.mod: go.mod
  Cache Size: ~50 MB (52525062 B)
  /usr/bin/tar -xf /home/runner/work/_temp/0d91b02e-6eb7-4a83-802b-26ef68d84748/cache.tzst -P -C /home/runner/work/kaitai_struct_go_runtime/kaitai_struct_go_runtime --use-compress-program unzstd
  /usr/bin/tar: ../../../go/pkg/mod/golang.org/x/net@v0.6.0/go.sum: Cannot open: File exists
  Error: /usr/bin/tar: ../../../go/pkg/mod/golang.org/x/net@v0.6.0/proxy/proxy.go: Cannot open: File exists
  Error: /usr/bin/tar: ../../../go/pkg/mod/golang.org/x/net@v0.6.0/proxy/socks5.go: Cannot open: File exists
...
  /usr/bin/tar: ../../../go/pkg/mod/github.com/stretchr/testify@v1.9.0/.ci.gogenerate.sh: Cannot open: File exists
  /usr/bin/tar: ../../../go/pkg/mod/github.com/stretchr/testify@v1.9.0/LICENSE: Cannot open: File exists
  /usr/bin/tar: ../../../go/pkg/mod/github.com/stretchr/testify@v1.9.0/.ci.gofmt.sh: Cannot open: File exists
  /usr/bin/tar: ../../../go/pkg/mod/github.com/stretchr/testify@v1.9.0/MAINTAINERS.md: Cannot open: File exists
  /usr/bin/tar: Exiting with failure status due to previous errors
  Warning: Failed to restore: "/usr/bin/tar" failed with error: The process '/usr/bin/tar' failed with exit code 2
  Cache not found for input keys: golangci-lint.cache-2830-0bbfc5de4c89f725d225c1263d61ea48efb7b45b, golangci-lint.cache-2830-

This seems to be a common problem that many users of https://github.com/golangci/golangci-lint-action have run into, but I'm not 100% sure what's causing it and how to fix it. Apparently, it's a problem with restoring the cache done by golangci-lint-action.

Despite reading a decent portion of the related issues (but definitely not all, since there are many of them), I don't have a definitive answer. In this issue - golangci/golangci-lint-action#863 (comment) - it was suggested that the caching done by actions/setup-go@v5 clashes with the golangci-lint-action caching, so apparently, you're supposed to enable caching either on setup-go or golangci-lint-action but not both. However, I think we do that, with the cache: false on setup-go - .github/workflows/go.yml:36, although I'm not sure whether the enabled setup-go caching in the other job "Test" .github/workflows/go.yml:16-19 counts or not.

It's possible that the go mod download before running golangci-lint-action is the problem:

- run: go mod download
- name: Run golangci-lint
uses: golangci/golangci-lint-action@v4

... but I don't know. I only added it because I saw it in golangci/golangci-lint-action#902 (comment) and it reportedly fixed certain problems (but now after taking a closer look, they actually enabled the caching on setup-go and disabled it on golangci-lint-action, so the other way around).

Since all the offending files seem to have paths like ../../../go/pkg/mod/*, in the end we'll probably end up with skip-pkg-cache: true and move on (https://github.com/golangci/golangci-lint-action/tree/v4.0.0?tab=readme-ov-file#how-to-use):

        with:
          # ...
          version: v1.54
          # ...

          # Optional: if set to true, then the action won't cache or restore ~/go/pkg.
          # skip-pkg-cache: true

... but I'm still curious whether it's somehow our fault, or indeed a bug in the golangci-lint-action's caching implementation.

@GreyCat GreyCat merged commit 47c8778 into master Mar 31, 2024
2 checks passed
@GreyCat GreyCat deleted the add_kaitai_struct branch March 31, 2024 22:14
@GreyCat
Copy link
Member Author

GreyCat commented Mar 31, 2024

@generalmimon That's a very fair concern around lint, and indeed I agree it looks weird. Even if we're doing anything wrong, I would welcome golangci-lint-action/ to have some preventive diagnostics instead of just blindly dumping 3000+ errors.

Thanks for your thorough investigation :) Indeed, let's give a try with skip-pkg-cache.

@generalmimon
Copy link
Member

From our CI runs that we've seen so far, it only happens in pull requests - I haven't seen it in any of the runs on master.

Of all the runs on this PR, the first run doesn't have the problem:

Step Run golangci-lint

Run golangci/golangci-lint-action@v4
  with:
    version: v1.54
    args: -c .golangci.yml
    github-token: ***
    only-new-issues: false
    skip-cache: false
    skip-pkg-cache: false
    skip-build-cache: false
    install-mode: binary
prepare environment
  Checking for go.mod: go.mod
  Cache Size: ~10 MB (10145088 B)
  /usr/bin/tar -xf /home/runner/work/_temp/5cf113fc-9fbb-454a-ae8e-c4a984987a79/cache.tzst -P -C /home/runner/work/kaitai_struct_go_runtime/kaitai_struct_go_runtime --use-compress-program unzstd
  Cache restored successfully
  Restored cache for golangci-lint from key 'golangci-lint.cache-2830-0bbfc5de4c89f725d225c1263d61ea48efb7b45b' in 531ms
  Finding needed golangci-lint version...
  Requested golangci-lint 'v1.54', using 'v1.54.2', calculation took 27ms
  Installation mode: binary
  Installing golangci-lint binary v1.54.2...
  Downloading binary https://github.com/golangci/golangci-lint/releases/download/v1.54.2/golangci-lint-1.54.2-linux-amd64.tar.gz ...
  /usr/bin/tar xz --overwrite --warning=no-unknown-keyword --overwrite -C /home/runner -f /home/runner/work/_temp/863a4ebe-4f42-4937-a2cb-1d86da1a72ad
  Installed golangci-lint into /home/runner/golangci-lint-1.54.2-linux-amd64/golangci-lint in 397ms
  Prepared env in 956ms
run golangci-lint
  Running [/home/runner/golangci-lint-1.54.2-linux-amd64/golangci-lint run --out-format=github-actions -c .golangci.yml] in [] ...
  Received 10145088 of 10145088 (100.0%), 9.7 MBs/sec
  Error: File is not `gci`-ed with --skip-generated -s standard -s default (gci)
  Error: File is not `gofmt`-ed with `-s` (gofmt)
  Warning: var-naming: don't use underscores in Go names; method Kaitai_IO should be KaitaiIO (revive)
  Warning: var-naming: don't use underscores in Go names; method Kaitai_IO should be KaitaiIO (revive)
  Error: parameter *testing.T should be the first or after context.Context (thelper)
  
  Error: issues found
  Ran golangci-lint in 9005ms

Step Post Run golangci-lint

Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/kaitai_struct_go_runtime/kaitai_struct_go_runtime --files-from manifest.txt --use-compress-program zstdmt
Cache Size: ~50 MB (52525062 B)
Cache saved successfully
Saved cache for golangci-lint from paths '/home/runner/.cache/golangci-lint, /home/runner/.cache/go-build, /home/runner/go/pkg' in 3561ms

... but the second run and all the others have the problem:

Step Run golangci-lint

Run golangci/golangci-lint-action@v4
  with:
    version: v1.54
    args: -c .golangci.yml
    github-token: ***
    only-new-issues: false
    skip-cache: false
    skip-pkg-cache: false
    skip-build-cache: false
    install-mode: binary
prepare environment
  Checking for go.mod: go.mod
  Cache Size: ~50 MB (52525062 B)
  /usr/bin/tar -xf /home/runner/work/_temp/02bc5714-a1f3-497e-9ff9-ec390a26769b/cache.tzst -P -C /home/runner/work/kaitai_struct_go_runtime/kaitai_struct_go_runtime --use-compress-program unzstd
  /usr/bin/tar: ../../../go/pkg/mod/golang.org/x/net@v0.6.0/go.sum: Cannot open: File exists
  ...
  /usr/bin/tar: ../../../go/pkg/mod/github.com/stretchr/testify@v1.9.0/.ci.gofmt.sh: Cannot open: File exists
  /usr/bin/tar: ../../../go/pkg/mod/github.com/stretchr/testify@v1.9.0/MAINTAINERS.md: Cannot open: File exists
  /usr/bin/tar: Exiting with failure status due to previous errors
  Warning: Failed to restore: "/usr/bin/tar" failed with error: The process '/usr/bin/tar' failed with exit code 2
  Cache not found for input keys: golangci-lint.cache-2830-0bbfc5de4c89f725d225c1263d61ea48efb7b45b, golangci-lint.cache-2830-
  Finding needed golangci-lint version...
  Requested golangci-lint 'v1.54', using 'v1.54.2', calculation took 17ms
  Installation mode: binary
  Installing golangci-lint binary v1.54.2...
  Downloading binary https://github.com/golangci/golangci-lint/releases/download/v1.54.2/golangci-lint-1.54.2-linux-amd64.tar.gz ...
  /usr/bin/tar xz --overwrite --warning=no-unknown-keyword --overwrite -C /home/runner -f /home/runner/work/_temp/b9c31a39-a734-490c-9e29-e7b2f492ccd2
  Installed golangci-lint into /home/runner/golangci-lint-1.54.2-linux-amd64/golangci-lint in 479ms
  Prepared env in 2073ms
run golangci-lint
  Running [/home/runner/golangci-lint-1.54.2-linux-amd64/golangci-lint run --out-format=github-actions -c .golangci.yml] in [] ...
  Error: File is not `gci`-ed with --skip-generated -s standard -s default (gci)
  Error: File is not `gofmt`-ed with `-s` (gofmt)
  Error: var-naming: don't use underscores in Go names; method Kaitai_IO should be KaitaiIO (revive)
  Error: var-naming: don't use underscores in Go names; method Kaitai_IO should be KaitaiIO (revive)
  Error: parameter *testing.T should be the first or after context.Context (thelper)
  
  Error: issues found
  Ran golangci-lint in 1014ms

Step Post Run golangci-lint

Post job cleanup.
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/kaitai_struct_go_runtime/kaitai_struct_go_runtime --files-from manifest.txt --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key golangci-lint.cache-2830-0bbfc5de4c89f725d225c1263d61ea48efb7b45b, another job may be creating this cache. More details: Cache already exists. Scope: refs/pull/30/merge, Key: golangci-lint.cache-2830-0bbfc5de4c89f725d225c1263d61ea48efb7b45b, Version: 0f79570b790d0689c186c2d92be94cb909971c6643d398173c3c5a0ecd5cf601
Saved cache for golangci-lint from paths '/home/runner/.cache/golangci-lint, /home/runner/.cache/go-build, /home/runner/go/pkg' in 767ms

@generalmimon
Copy link
Member

@GreyCat FYI, there has been a new major version v5.0.0 of golangci-lint-action recently that looks like it should resolve the aforementioned /usr/bin/tar: ...: Cannot open: File exists issues:

Changes

The PR message of golangci/golangci-lint-action#1024 claims to fix issues like golangci/golangci-lint-action#863 and golangci/golangci-lint-action#807, which both refer to the same /usr/bin/tar: ...: Cannot open: File exists errors.

So hopefully we can resolve the cache-related problems by updating to golangci/golangci-lint-action@v5 (we're currently using @v4, see .github/workflows/go.yml:41) without having to worry about skip-pkg-cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants