Need advice on production deployment

[GoodManWEN]

Hi everyone, I came across this amazing library some time ago and I'm trying to use it in a production environment soon. I've got some questions that I haven't found answered in the documentation, so I'm going to seek help from the community.

My usage scenarios are:

1. Approximately 500G of data in total (uncompressed)
2. Two existing machines with 64GB RAM and 2TB storage
3. The usage scenarios are almost all random reads and writes, with a read/write volume of approximately 100/1
4. The service have a relatively high number of qps but not to the limit of what a single machine can load. (Peaks can reach 300-500 requests per second)
5. We want to optimise the service with the aim of minimising response latency.
6. We want the data service take some memory for caching to increase performance if possible.
7. We would like to see a more robust solution for securing data, things such as crash safe policy.
8. As we have two machines and one is currently able to handle peak loads, although we want to set up a 1-master-1-slave cluster, we only want one of the two to provide service. The other one we only want to do for backup purposes. For example, we synchronise every 6 hours, which prevents our data from suffering an unacceptable crash in the event of an accident.

Suggestions on any of the above points are very welcome. Because we know that although this service has a similar API to redis, the architecture is on a completely different underlying layer. In many cases, this makes it difficult to apply our experience. Also because the community is not yet rich in information, there are many questions we cannot fully confirm the answers to.

P.S. Can anyone tell me where I can find documents related to the multi-threaded read and write design of this library. Since the original single-threaded model of Redis is completely changed, I'm afraid we'll have different expectations for dirty reads/writes cases.

[git-hulk]

@GoodManWEN For improving the read performance, you can increase the blob cache size in kvrocks.conf. And it's unnecessary to leave one instance as a stand-by instance, you can use the backup mechanism periodically if want to prevent data corruption.

[GoodManWEN]

Yes, because I only want to back up for security purposes and a spare instance is not necessary. Can you tell me more about the backup mechanism? Think that's exactly what I need, but I'm not sure how I should do the backup and restore

[git-hulk]

Kvrocks has the bgsave command like Redis which would create a database checkpoint in the backup dir(default path is dir/backup). And the restore is as simple as moving this dir to the db dir.

[GoodManWEN]

Superb, Then if I have backups weekly it might solve my needs. But this looks like a full database output, is there an incremental way to update the backup?

[PragmaTwice]

Maybe you can find most of the storage performance tuning in the RocksDB wiki, e.g. https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning

Although we are happy to maintain some more user-friendly documents about tuning by our own.

[GoodManWEN]

Thanks for reply, I read it carefully and it was very helpful. I think in my usage case adjusting all the options except the Bloom filter would help improve performance. There is something about write caching that I don't understand very well, I'm not quite sure at what level this caching occurs. I'm concerned about whether if there's case where small writes remain in the cache for a long time without being written to the hard drive. (Comparing this to Redis' RDB process, I can get a clear idea of the conditions under which the cache will be refreshed). Any updates in the documentation would be very helpful.

[niumcdao]

I execute bgsave command and I not found some *.sst file in backup dir. Why?

[PragmaTwice]

It's not a blocking command (I'm not sure if there will be a blocking version in the future).

You can check info persistence for whether the process triggered by the bgsave command is finished.