Consider a different DB type for hop-node

[shanefontaine]

What DB do we want to use, key-value or relational?

The biggest question to solve here is will the DB we choose be a limiting factor somehow? For example, will we only be able to process x TPS because we chose one DB but could process x*2 TPS because we chose another. My gut says that the true limiting factor will be network calls unrelated to the DB, but I’m not 100% sure. What constraints are we going to hit as it relates to decentralized network bonder operations?

Other Notes

• If key-value, do not use leveldb. It has been unmaintained for years at this point. Research what to use.
• The concept of getTransfersFromX should not exist. We need to avoid arbitrarily running OOM and this would do it (i.e. no matter what time we use for X someone can always technically cause us to go OOM by sending, for example).

Thought Experiment

Note

💡 For each case below, consider the case where Hop volume is on par with, or exceeding, that of the Arbitrum quest.

• If network latency will always be the limiting factor for user-facing operations, then probably relational.
• If DB read/write speed will be a limiting factor for user-facing operations, then probably use key-value.
• Will DB size be a bigger issue than it is in V1 since the network is decentralized? If so, do we need to optimize for size so that, for example, we don’t have 1k bonders needing 100GB each?
  • Pruning, architecture, or something else might be the correct answer here instead of DB type, but worth considering.
• Will memory be an issue if we get high tx volume like Arbitrum quests? If so, do we need to optimize for size so that, for example, we don’t have 1k bonders needing 16GB each?
  • Other factors will likely be the correct answer here instead of DB type, but worth considering.
• Will IOPS be a limiting factor in any way?

Reference comments

"My thinking on this is to use a relation DB as main db and key/value db for cache, since the events the bonder stores are relational to other events. For relational dbs, postgres is the recommended db to use due to it’s robustness/features and open sourceness."

If we use PG, figure out a better way to do version control or debugging

• Don’t use long strings to avoid manual errors that aren’t caught by linters/IDE/etc. They should be easy to test without hand-written strings.
  - Example with the ? ? ? and the large multi-line strings.
  - SQL statements
• Avoid copy/paste duplication like this commit (I think this is in the code, though, not PG)