Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent decoding #474

Open
mash opened this issue Oct 9, 2023 · 0 comments
Open

Concurrent decoding #474

mash opened this issue Oct 9, 2023 · 0 comments

Comments

@mash
Copy link

mash commented Oct 9, 2023

Hi, thank you for making this package, loading JSON files is now much faster.

Though we're seeing following SIGSEGV occasionally.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x13f2ad2]

goroutine 638 [running]:
github.com/goccy/go-json/internal/decoder.(*structDecoder).DecodeStream(0x0, 0xc001d02420, 0xc0163a2000?, 0xc001102000)
        /root/server/vendor/github.com/goccy/go-json/internal/decoder/struct.go:696 +0x3b2
github.com/goccy/go-json.(*Decoder).DecodeWithOption(0xc000c99548, {0x1c392e0, 0xc001102000}, {0x0, 0x0, 0x6122203a2265646f?})
        /root/server/vendor/github.com/goccy/go-json/decode.go:233 +0xe6
github.com/goccy/go-json.(*Decoder).Decode(...)
        /root/server/vendor/github.com/goccy/go-json/decode.go:199
github.com/org/server/models.ReadFile({0xc001a8e0c0, 0x2c})
        /root/server/models/repo.go:381 +0x399
github.com/org/server/models.(*Repo).visit(0xc0017ad220, {0xc000def800, 0x13}, {0xc001a8e0c0, 0x2c}, {0xc001a8e081?, 0xb?})
        /root/server/models/repo.go:479 +0xc5
github.com/org/server/models.(*Repo).load.func1({0xc001a8e0c0?, 0xc000db43a0?}, {0x252c1a8?, 0xc00018c820?})
        /root/server/models/repo.go:606 +0x45
github.com/saracen/walker.(*walker).walk(0xc001e171c0, {0xc000db43a0, 0x20}, {0x252c1a8, 0xc00018c820})
        /root/server/vendor/github.com/saracen/walker/walker.go:76 +0x85
github.com/saracen/walker.(*walker).readdir(0xc001e171c0?, {0xc000db43a0, 0x20})
        /root/server/vendor/github.com/saracen/walker/walker_unix.go:46 +0x3b2
github.com/saracen/walker.(*walker).gowalk(0xc001e171c0, {0xc000db43a0, 0x20})
        /root/server/vendor/github.com/saracen/walker/walker.go:118 +0x25
github.com/saracen/walker.(*walker).walk.func1()
        /root/server/vendor/github.com/saracen/walker/walker.go:103 +0x1f
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /root/server/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 633
        /root/server/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0x96
FAIL    github.com/org/server/cli/cmd/repo       1.564s

We're using github.com/saracen/walker to read a lot of JSON files in parallel,
and seems to be hitting the CompileToGetDecoder race condition.

Is there a workaround or fix to this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant