Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible memory leak when opening a new DB by the commit buffer #585

Open
doron-cohen opened this issue May 8, 2024 · 0 comments
Open

Comments

@doron-cohen
Copy link

Describe the bug
Hi! We are using NutsDB as a temporary KV store on disk. We create the database, use it for a short period (few seconds up to a few minutes) and close it.
After running for a few hours we started to notice the memory growing constantly across runs. We ran a profiler to detect the issue and it shows createNewBufferWithSize is the cause:
Screenshot 2024-05-08 at 11 55 45

I setup a quick test to benchmark and take a look inside:

func BenchmarkOpen(b *testing.B) {
	for i := 0; i < b.N; i++ {
		dbDir, err := os.MkdirTemp("", "state_********")
		require.NoError(b, err)
		defer os.RemoveAll(dbDir)

		db, err := nutsdb.Open(
			nutsdb.DefaultOptions,
			nutsdb.WithDir(dbDir),
		)
		require.NoError(b, err)

		err = db.Close()
		require.NoError(b, err)
	}
}

Note that I don't even run any transactions here. I am confused to why would it even leak. In our real code we do run some operations on init.

Ran it:

❯ go test -bench '^BenchmarkOpen$' -benchmem -memprofile memprofile.out package_name
/dirsync/core/state
goos: darwin
goarch: arm64
pkg: talon-sec.com/cloud-apps/apps/services/device-service/internal/modules/dirsync/core/state
BenchmarkOpen-10             338           4909304 ns/op         4270787 B/op        644 allocs/op
PASS
ok      package_name       4.137s

Looking at the mem profile I could see it does grow a lot only be running it 10 times:

❯ go tool pprof memprofile.out
(pprof) list Open
Total: 6.43GB
ROUTINE ======================== github.com/nutsdb/nutsdb.Open in /Users/docohen/.local/share/go/pkg/mod/github.com/nutsdb/nutsdb@v1.0.4/db.go
  512.09kB     6.42GB (flat, cum) 99.75% of Total
  512.09kB   512.09kB    123:func Open(options Options, ops ...Option) (*DB, error) {
         .          .    124:   opts := &options
         .          .    125:   for _, do := range ops {
         .          .    126:           do(opts)
         .          .    127:   }
         .     6.42GB    128:   return open(*opts)
         .          .    129:}
         .          .    130:
         .          .    131:// Update executes a function within a managed read/write transaction.
         .          .    132:func (db *DB) Update(fn func(tx *Tx) error) error {
         .          .    133:   if fn == nil {

Adding nutsdb.WithGCWhenClose(true) helps but this is still a problem:

(pprof) list Open
Total: 1.75GB
ROUTINE ======================== github.com/nutsdb/nutsdb.Open in /Users/docohen/.local/share/go/pkg/mod/github.com/nutsdb/nutsdb@v1.0.4/db.go
         0     1.75GB (flat, cum) 99.53% of Total
         .          .    123:func Open(options Options, ops ...Option) (*DB, error) {
         .          .    124:   opts := &options
         .          .    125:   for _, do := range ops {
         .          .    126:           do(opts)
         .          .    127:   }
         .     1.75GB    128:   return open(*opts)
         .          .    129:}
         .          .    130:
         .          .    131:// Update executes a function within a managed read/write transaction.
         .          .    132:func (db *DB) Update(fn func(tx *Tx) error) error {
         .          .    133:   if fn == nil {

Only adding nutsdb.WithCommitBufferSize(0) reduces it to a normal size:

(pprof) list Open
Total: 47.19MB
ROUTINE ======================== github.com/nutsdb/nutsdb.Open in /Users/docohen/.local/share/go/pkg/mod/github.com/nutsdb/nutsdb@v1.0.4/db.go
         0    33.54MB (flat, cum) 71.08% of Total
         .          .    123:func Open(options Options, ops ...Option) (*DB, error) {
         .          .    124:   opts := &options
         .          .    125:   for _, do := range ops {
         .          .    126:           do(opts)
         .          .    127:   }
         .    33.54MB    128:   return open(*opts)
         .          .    129:}
         .          .    130:
         .          .    131:// Update executes a function within a managed read/write transaction.
         .          .    132:func (db *DB) Update(fn func(tx *Tx) error) error {
         .          .    133:   if fn == nil {

To Reproduce
Steps to reproduce the behavior(Be specific!):
Use the test and commands above.

Give sample code if you can.

Expected behavior
A clear and concise description of what you expected to happen.

What actually happens
A clear and concise description of what actually happens.

Screenshots
If applicable, add screenshots to help explain your problem.

please complete the following information :

  • OS: Locally on a mac. In production under Alpine linux in a container.
  • NutsDB Version v1.0.4

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant