Issues with BadgerDB

[werbenhu]

Regarding issues with BadgerDB:

1. The difference between badgerhold and using Badger/v4 is essentially the difference between Badger/v1 and Badger/v4, as Badgerhold utilizes V1.

2. When using github.com/dgraph-io/badger and Badger/v2, inserting 1 million data entries consumes over 2 GB of disk space. Even after deleting all data and calling GC, there is still around 1 GB of residual disk space left. I haven't been able to figure out why this is happening. Additionally, inserting 1 million data entries is very slow.

3. With Badger/v3 and Badger/v4, inserting data is extremely fast. For the same 1 million data entries, disk usage is only between 200-300 MB. Insertion speed is also very fast. However, both versions suffer from the issue of GC not working properly. After deleting all data and calling GC, there is no reduction in disk space. You can refer to this issue: GC doesn't seem to run.

4. Considering the current situation, I think it might be best to temporarily use Badger/v4. I am still exploring whether there are alternative solutions. @thedevop @mochi-co @dgduncan I would like to hear your opinions on this matter.

5. The following is the code I tested:

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"syscall"

	// "github.com/dgraph-io/badger"
	// "github.com/dgraph-io/badger/v2"
	// "github.com/dgraph-io/badger/v3"
	"github.com/dgraph-io/badger/v4"
	"github.com/gin-gonic/gin"
)

const (
	PATH  = ".badger"
	RATIO = 0.5
)

var db *badger.DB

func insert(c *gin.Context) {
	prefix := c.Query("prefix")
	size := c.Query("size")

	cnt, _ := strconv.Atoi(size)

	fmt.Printf("insert cnt:%d, prefix:%s\n", cnt, prefix)

	for i := 1; i <= cnt; i++ {
		err := db.Update(func(txn *badger.Txn) error {
			bs := make([]byte, 1024)
			for j := 0; j < 1024; j++ {
				bs[j] = byte(i)
			}
			err := txn.Set([]byte(prefix+strconv.Itoa(i)), bs)
			return err
		})
		if err != nil {
			c.JSON(http.StatusOK, gin.H{
				"err": err.Error(),
			})
			return
		}
	}

	c.JSON(http.StatusOK, gin.H{
		"message": "success",
	})
}

func count(c *gin.Context) {
	var count int
	err := db.View(func(txn *badger.Txn) error {
		opts := badger.DefaultIteratorOptions
		opts.PrefetchValues = false // We don't need to prefetch values, only keys
		it := txn.NewIterator(opts)
		defer it.Close()

		// Iterate over each item and count them
		for it.Rewind(); it.Valid(); it.Next() {
			count++
		}

		return nil
	})

	if err != nil {
		c.JSON(http.StatusOK, gin.H{
			"err": err.Error(),
		})
		return
	}

	c.JSON(http.StatusOK, gin.H{
		"count": count,
	})
}

func delete(c *gin.Context) {
	prefix := c.Query("prefix")
	size := c.Query("size")

	cnt, _ := strconv.Atoi(size)
	fmt.Printf("delete cnt:%d, prefix:%s\n", cnt, prefix)
	for i := 1; i <= cnt; i++ {
		err := db.Update(func(txn *badger.Txn) error {
			err := txn.Delete([]byte(prefix + strconv.Itoa(i)))
			return err
		})
		if err != nil {
			c.JSON(http.StatusOK, gin.H{
				"err": err.Error(),
			})
			return
		}
	}

	c.JSON(http.StatusOK, gin.H{
		"message": "success",
	})
}

func find(c *gin.Context) {
	key := c.Query("key")

	var bs []byte
	err := db.View(func(txn *badger.Txn) error {
		item, err := txn.Get([]byte(key))
		if err != nil {
			return err
		}
		bs, err = item.ValueCopy(nil)
		return err
	})
	if err != nil {
		c.JSON(http.StatusOK, gin.H{
			"err": err.Error(),
		})
		return
	}

	c.JSON(http.StatusOK, gin.H{
		"message": "success",
		"bs":      bs,
	})
}

func gc(c *gin.Context) {
again:
	fmt.Printf("GC starting...\n")
	err := db.RunValueLogGC(RATIO)
	if err == nil {
		goto again
	}

	fmt.Printf("GC end err:%s.\n", err.Error())
	c.JSON(http.StatusOK, gin.H{
		"message": "success",
	})
}

func main() {

	sigs := make(chan os.Signal, 1)
	done := make(chan bool, 1)
	signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
	go func() {
		<-sigs
		done <- true
	}()

	var err error
	opts := badger.DefaultOptions(".badger")
	opts.ValueLogFileSize = 100 << 20

	db, err = badger.Open(opts)
	if err != nil {
		log.Fatal(err)
	}

	r := gin.Default()
	r.GET("/insert", insert)
	r.GET("/delete", delete)
	r.GET("/find", find)
	r.GET("/count", count)
	r.GET("/gc", gc)
	go r.Run(":9000")

	<-done
	fmt.Printf("Server done\n")
	db.Close()
}

###
GET http://localhost:9000/insert?size=1000000&prefix=a

###
GET http://localhost:9000/delete?size=1000000&prefix=a

###
GET http://localhost:9000/gc

###
GET http://localhost:9000/count

[mochi-co]

Thanks for your research into this @werbenhu, it's very comprehensive!

"With Badger/v3 and Badger/v4, inserting data is extremely fast. [...] However, both versions suffer from the issue of GC not working properly."
"I think it might be best to temporarily use Badger/v4."

Did you mean temporarily use Badger/v1 ?

[werbenhu]

@mochi-co I personally prefer using V4 as V1 consumes a very large amount of disk space. Although the issue with GC not working in V4, it utilizes a much smaller amount of disk space.

Perhaps there might be some inaccuracies in my test results. You all can also test it in your own environments.

[mochi-co]

@werbenhu Ah, I understand what you mean. Although it's not ideal, this sounds to me to be the best compromise for now since most complaints stem from the use of disk space. A 10x reduction in disk usage should be beneficial even with the GC issues.

[werbenhu]

@mochi-co I've submitted a PR #376, everyone please take a look first. It would be best if we could find a better solution (such as Badger community fixing this GC issue tomorrow😁).

[werbenhu]

Here at https://github.com/smallnest/kvbench, there are many key-value data implementations written in pure Go. Perhaps we can select one of them, a stable option, and implement a storage hook for it as a built-in persistent database for users to choose from.

[werbenhu]

I've found Pebble to be so perfect. Inserting 1 million data entries using the NoSync option only takes 2 seconds, with disk usage only at 80 MB. Searching for data by key prefix from within the 1 million entries is also very fast. I've decided to make time to implement the Pebble hook.

Here's the code I tested:

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"syscall"

	"github.com/cockroachdb/pebble"
	"github.com/gin-gonic/gin"
)

const (
	PATH  = ".pebble"
	RATIO = 0.5
)

var db *pebble.DB

func insert(c *gin.Context) {
	prefix := c.Query("prefix")
	size := c.Query("size")

	cnt, _ := strconv.Atoi(size)

	fmt.Printf("insert cnt:%d, prefix:%s\n", cnt, prefix)

	for i := 1; i <= cnt; i++ {
		bs := make([]byte, 1024)
		for j := 0; j < 1024; j++ {
			bs[j] = byte(i)
		}

		err := db.Set([]byte(prefix+strconv.Itoa(i)), bs, pebble.NoSync)
		if err != nil {
			c.JSON(http.StatusOK, gin.H{
				"err": err.Error(),
			})
			return
		}
	}

	c.JSON(http.StatusOK, gin.H{
		"message": "success",
	})
}

func count(c *gin.Context) {

	prefix := c.Query("prefix")

	var count int
	var iter *pebble.Iterator

	if len(prefix) == 0 {
		iter, _ = db.NewIter(nil)
	} else {
		iter, _ = db.NewIter(&pebble.IterOptions{
			LowerBound: []byte(prefix),
			UpperBound: keyUpperBound([]byte(prefix)),
		})
	}

	for iter.First(); iter.Valid(); iter.Next() {
		count++
	}

	if err := iter.Close(); err != nil {
		log.Fatal(err)
	}

	c.JSON(http.StatusOK, gin.H{
		"count": count,
	})
}

func delete(c *gin.Context) {
	prefix := c.Query("prefix")
	size := c.Query("size")

	cnt, _ := strconv.Atoi(size)
	fmt.Printf("delete cnt:%d, prefix:%s\n", cnt, prefix)
	for i := 1; i <= cnt; i++ {
		err := db.Delete([]byte(prefix+strconv.Itoa(i)), pebble.NoSync)
		if err != nil {
			log.Fatal(err)
		}

		if err != nil {
			c.JSON(http.StatusOK, gin.H{
				"err": err.Error(),
			})
			return
		}
	}

	c.JSON(http.StatusOK, gin.H{
		"message": "success",
	})
}

func find(c *gin.Context) {
	key := c.Query("key")

	value, closer, err := db.Get([]byte(key))
	if err != nil {
		c.JSON(http.StatusOK, gin.H{
			"err": err.Error(),
		})
		return
	}
	defer closer.Close()

	c.JSON(http.StatusOK, gin.H{
		"message": "success",
		"bs":      value,
	})
}

func keyUpperBound(b []byte) []byte {
	end := make([]byte, len(b))
	copy(end, b)
	for i := len(end) - 1; i >= 0; i-- {
		end[i] = end[i] + 1
		if end[i] != 0 {
			return end[:i+1]
		}
	}
	return nil // no upper-bound
}

func main() {

	sigs := make(chan os.Signal, 1)
	done := make(chan bool, 1)
	signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
	go func() {
		<-sigs
		done <- true
	}()

	var err error
	db, err = pebble.Open(PATH, &pebble.Options{})
	if err != nil {
		log.Fatal(err)
	}

	r := gin.Default()
	r.GET("/insert", insert)
	r.GET("/delete", delete)
	r.GET("/find", find)
	r.GET("/count", count)
	go r.Run(":9000")

	<-done
	fmt.Printf("Server done\n")
	if err := db.Close(); err != nil {
		fmt.Printf("close db err:%s\n", err.Error())
	}
}

###
GET http://localhost:9000/insert?size=1000000&prefix=werben
###
GET http://localhost:9000/insert?size=10&prefix=mochi
###
GET http://localhost:9000/find?key=werben999
###
GET http://localhost:9000/count
###
GET http://localhost:9000/count?prefix=mochi
###
GET http://localhost:9000/delete?size=1000000&prefix=werben

[dgduncan]

I will test this later on this weekend. I have not used Pebble before but I trust almost anything that comes out of CockroachDB.

It really seems like BadgerDB is almost more trouble than it is worth.

[mochi-co]

I agree with @dgduncan @werbenhu regarding Pebble. For what it's worth, I only implemented Badger as it was touted as the successor to Bolt at the time. I didn't put any extensive thought or usage research into it. Absolutely happy to deprecate it and integrate Pebble as the preferred storage mechanism.

[werbenhu]

@dgduncan I don't have any time this weekend. It would be great if you could implement this hook if you have time.

[werbenhu]

@OpenJarvisAI I don't think this is an issue. Based on my previous tests, 1 million data entries were synchronized to the disk very quickly. Even if you use Redis, Redis's RDB persistence also has the same issue. You should know that there is no perfect solution, only the most suitable one. We provide several persistence options, and you can choose one according to your own usage scenarios.

[luohao123]

Hello, does it now have a built-in (or very easy config) way to use it?

I think offline messages cache is very needed for mqtt server if we use it and when server was down, the messages which is not sent should be reload !

[OpenJarvisAI]

does it can use now?

[werbenhu]

Refer to #378 .

[werbenhu]

@luohao123 I correct what I said before. Data under the NoSync mode won't be lost, and the WAL (Write Ahead Log) of LMS ensures that even in the event of a crash, data won't be lost.

[luohao123]

Hello, what's the status now?

[werbenhu]

Refer to #378 .

[werbenhu]

@mochi-co @dgduncan PR #378 has added a persistence hook based on cockroachdb/pebble. Please review it.

[OpenJarvisAI]

@mochi-co any updates? Would like to see any local test on it