Flat storage tool #11013
Flat storage tool #11013
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #11013 +/- ##
==========================================
- Coverage 71.33% 71.21% -0.12%
==========================================
Files 760 761 +1
Lines 152301 152530 +229
Branches 152301 152530 +229
==========================================
- Hits 108637 108631 -6
- Misses 39197 39426 +229
- Partials 4467 4473 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
|
||
| // This DB can only write to write_db | ||
| // When reading, it first reads from wirte_db, then amended_db, then base_db | ||
| let mixed_db = MixedDB::new( |
There was a problem hiding this comment.
confirming
- we are creating a mixed db where we are writing to the custom recovery database we are trying to create with the flat storage?
- we are reading from the split storage db but with one tiny change where we are manually setting the final height to our custom value?
|
Confirming the flat head is inclusive of the data of the block height for which we have flat storage created. nearcore/chain/chain/src/resharding.rs Lines 496 to 499 in 10b8725 |
|
Taking a look at the implementation of Considering resharding we would like the |
| .set_ser(DBCol::FlatStorageStatus, &shard_uid.to_bytes(), &FlatStorageStatus::Empty) | ||
| .expect("Unable to set FS status"); | ||
| store_update.commit().expect("Unable to change FS status"); | ||
|
|
||
| let mut creator = | ||
| FlatStorageShardCreator::new(shard_uid, tip.height - 1, epoch_manager, rw_hot_runtime); |
There was a problem hiding this comment.
There's some weird logic checking if specific deltas exist for the start_height parameter upto final_head. I would probably try to set this to tip.height instead of tip.height - 1 to begin with, although @Longarithm or @pugachAG might have a better idea why this has a -1
There was a problem hiding this comment.
Maybe for more safety... It is easier to move head forward later than backwards (see MoveFlatHeadMode::Back). tip.height also works.
|
This tool was used to build FS for testnet and mainnet with final height set to the last block before the switch to 5 shards. Neard command
This is the neard command that fails with this binary This is a problem for us, as the current approach requires us to build FS pre-resharding for faster performance. We also need a way to verify FS for optional height (current verify tool works only for final height). We maybe can do the same kind of modification as my draft PR, but I am not sure that this is a valid solution. |
This PR is for demonstration purposes only, and should not be merged as is.
I plan to use
flat-storage inittool to recreate a flat storage from a historical height. But this tool uses final height right now.Finding and changing every usage of
BlockMiscin flat storage initialisation seemed like too much. So before init I "overwrite" BlockMisc column.This is done through
MixedDB. This struct is also used to write all the changes in the separate db.MixedDBhas read-only database, and read-write database. All writes go to the read-write db, but reading is done sequentially. Read priority is decided during db creation.So, to "overwrite" base db, I need to combine it with read-only db that contains amendments, and choose read priority as
ReadDBFirst.To write something to another db, I need to use it as a read-write db, and choose read priority as
WriteDBFirst(as i assume that we should give priority to a fresher data).I have some questions still:
height + 1orheight?verifycommand and run it after the FlatStorage creation? Will it help us to find any bugs?