feat: add chain-initiator tool to use mainnet snapshot in localnet #3435

snobbee · 2023-12-07T13:44:49Z

Here is how to run the chain-initiator tool:

$ go run ./scripts/chain-initiator/... https://snapshots.polkachu.com/snapshots/sifchain/sifchain_15048938.tar.lz4 https://github.com/Sifchain/sifnode/releases/download/v1.3.0-beta/sifnoded-v1.3.0-beta-darwin-arm64 https://github.com/Sifchain/sifnode/releases/download/v1.3.1-beta/sifnoded-v1.3.1-beta-darwin-arm64 --home /tmp/node --skip-snapshot --skip-chain-init
2023/12/11 10:42:26 skipping snapshot retrieval
2023/12/11 10:42:26 skipping chain init
2023/12/11 10:42:31 Old binary path: /tmp/sifnoded-1.3.0-beta and version: 1.3.0-beta
2023/12/11 10:42:38 New binary path: /tmp/sifnoded-1.3.1-beta and version: 1.3.1-beta
2023/12/11 10:42:38 Waiting for node to start...
[...]
2023/12/11 10:42:43 Node is running.
2023/12/11 10:42:44 Current Block Height: 15049014
2023/12/11 10:42:44 Waiting for next block height...
[...]
10:42AM INF executed block height=15049015 module=state num_invalid_txs=0 num_valid_txs=0
10:42AM INF commit synced commit=436F6D6D697449447B5B3830203133392032343720323535203132392034392032322031363720353320313832203137342031313720313135203636203620313637203232302031353020313630203138342031393120313339203233322039302032313720313933203237203933203234332031363920313436203232365D3A4535413133377D
[...]
2023/12/11 10:42:49 New Block Height: 15049015
[...]
2023/12/11 10:42:51 Submitted upgrade proposal: 1.3.1-beta, upgrade block height: 15049020
[...]
10:42AM INF received complete proposal block hash=3718AD754A13DF012AFEBB89397BBA7909F997AE4A8D4978EB31F68B1FC3F8CF height=15049017 module=consensus
[...]
2023/12/11 10:42:56 Voted on upgrade proposal: 149
[...]
2023/12/11 10:43:01 Waiting for block height 15049020 ...
[...]
2023/12/11 10:43:06 Waiting for block height 15049020 ...
[...]
10:43AM ERR UPGRADE "1.3.1-beta" NEEDED at height: 15049020: 
10:43AM ERR CONSENSUS FAILURE!!! err="UPGRADE \"1.3.1-beta\" NEEDED at height: 15049020: " module=consensus stack="goroutine 101 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x64\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:732 +0x44\npanic({0x102c860e0, 0x1405fc772f0})\n\truntime/panic.go:884 +0x204\ngithub.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{_, _}, _, {_, _}, {_, _}, _, {_, _}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.16/x/upgrade/abci.go:71 +0xc38\ngithub.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.16/x/upgrade/module.go:130\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x102fa8418, 0x140000520b0}, {0x102fb6650, 0x14060464080}, {{0xb, 0x0}, {0x14055635f90, 0xa}, 0xe5a13c, ...}, ...}, ...)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.16/types/module/module.go:491 +0x188\ngithub.com/Sifchain/sifnode/app.(*SifchainApp).BeginBlocker(...)\n\tgithub.com/Sifchain/sifnode/app/app.go:721\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0x14060439f80, 0x20, 0x20}, {{0xb, 0x0}, {0x14055635f90, 0xa}, 0xe5a13c, {0x1bb798a8, ...}, ...}, ...})\n\tgithub.com/cosmos/cosmos-sdk@v0.45.16/baseapp/abci.go:177 +0x768\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0x14060439f80, 0x20, 0x20}, {{0xb, 0x0}, {0x14055635f90, 0xa}, 0xe5a13c, {0x1bb798a8, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.27/abci/client/local_client.go:280 +0x108\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0x14060439f80, 0x20, 0x20}, {{0xb, 0x0}, {0x14055635f90, 0xa}, 0xe5a13c, {0x1bb798a8, ...}, ...}, ...})\n\tgithub.com/tendermint/tendermint@v0.34.27/proxy/app_conn.go:81 +0x4c\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x102fa9288?, 0x1400012f440}, {0x102fb0500, 0x1400120afb0}, 0x140005d03c0, {0x102fb7c20, 0x14000ca2090}, 0xe5a13b?)\n\tgithub.com/tendermint/tendermint@v0.34.27/state/execution.go:307 +0x3a8\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0x140014a3b30, 0x7}}, {0x140014a3b40, 0xa}, 0xe5a0eb, 0xe5a13b, {{0x1405dcd1100, ...}, ...}, ...}, ...)\n\tgithub.com/tendermint/tendermint@v0.34.27/state/execution.go:140 +0xf0\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0x14000226000, 0xe5a13c)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1661 +0x99c\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0x14000226000, 0xe5a13c)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1570 +0x28c\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1505 +0xa0\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0x14000226000, 0xe5a13c, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1543 +0xb90\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0x14000226000, 0x1405fb70280, {0x0, 0x0})\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:2164 +0x1698\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0x14000226000, 0x1405fb70280, {0x0?, 0x100ff0e90?})\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:1962 +0x28\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0x14000226000, {{0x102f8de00, 0x1405fc54468}, {0x0, 0x0}})\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:861 +0x37c\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0x14000226000, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:788 +0x350\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.27/consensus/state.go:379 +0xf4\n"
10:43AM INF service stop impl={"Logger":{}} module=consensus msg={} wal=/tmp/node/data/cs.wal/wal
10:43AM INF service stop impl={"Dir":"/tmp/node/data/cs.wal","Head":{"ID":"4BeLzGkSMXWx:/tmp/node/data/cs.wal/wal","Path":"/tmp/node/data/cs.wal/wal"},"ID":"group:4BeLzGkSMXWx:/tmp/node/data/cs.wal/wal","Logger":{}} module=consensus msg={} wal=/tmp/node/data/cs.wal/wal
2023/12/11 10:43:11 Block height 15049020 reached
2023/12/11 10:43:11 Process killed successfully
2023/12/11 10:43:11 Process was killed: signal: killed
2023/12/11 10:43:16 Waiting for node to start...
[...]
10:43AM INF applying upgrade "1.3.1-beta" at height: 15049020
10:43AM INF Running upgrade handler for 1.3.1-beta
[...]
2023/12/11 10:43:21 Node is running.
2023/12/11 10:43:21 Current Block Height: 15049020
2023/12/11 10:43:21 Waiting for next block height...
[...]
10:43AM INF executed block height=15049021 module=state num_invalid_txs=0 num_valid_txs=0
10:43AM INF commit synced commit=436F6D6D697449447B5B3739203739203534203135203335203134203236203134342031313120323439203133362032382035322031352032343420323432203936203234352036392031303820323434203138372031313020323331203234362031363220313938203139322032353520313131203934203133335D3A4535413133447D
[...]
^C2023/12/11 10:43:42 Process killed successfully

The command requires 3 arguments:

a snapshot url that can be retrieved by snapshot provider such as this one https://polkachu.com/tendermint_snapshots/sifchain (example of URL: https://snapshots.polkachu.com/snapshots/sifchain/sifchain_15048938.tar.lz4)
a binary URL from the github release section to the current/old version of the binary we want to upgrade from
a binary URL from the github release section to the new version of the binary we want to upgrade to

What it does?

retrieve old and new binaries from github release section
init chain
retrieve snapshot file and extract into home path
export genesis file
re-init chain
add validator key
add genesis account
generate genesis transaction
collect genesis transactions
validate the resulting genesis file
read the genesis file
read the mainnet genesis file
add the validator account and balance to the existing account and balance states
reset distribution, staking and slashing states and discount the module balances
assign the initial genesis file genutil state to the mainnet genutil state
save the new genesis file
start the chain
wait for chain readiness
query latest proposal id and calculate next proposal id
submit new software upgrade proposal for upgrading to new binary version
vote for new proposal
waiting for upgrade block height until chain panic
stop the process
start the chain with the new version of the binary

codecov · 2023-12-07T13:49:50Z

Codecov Report

Merging #3435 (8970c55) into master (0c63583) will increase coverage by 1.00%.
Report is 14 commits behind head on master.
The diff coverage is 58.58%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3435      +/-   ##
==========================================
+ Coverage   43.35%   44.35%   +1.00%     
==========================================
  Files         166      183      +17     
  Lines       15895    16736     +841     
==========================================
+ Hits         6891     7423     +532     
- Misses       8591     8868     +277     
- Partials      413      445      +32

Files	Coverage Δ
app/app.go	`94.42% <100.00%> (+4.31%)`	⬆️
cmd/sifnoded/cmd/ibc-diag.go	`15.74% <ø> (ø)`
x/clp/keeper/executors.go	`59.12% <100.00%> (+0.60%)`	⬆️
x/clp/types/codec.go	`100.00% <100.00%> (ø)`
x/clp/types/key_rewards_bucket.go	`100.00% <100.00%> (ø)`
x/clp/types/keys.go	`85.36% <100.00%> (+2.03%)`	⬆️
x/clp/types/querier.go	`100.00% <ø> (ø)`
x/clp/types/types.go	`97.14% <100.00%> (+0.26%)`	⬆️
x/dispensation/types/types.go	`100.00% <100.00%> (ø)`
x/epochs/keeper/abci.go	`100.00% <100.00%> (ø)`
... and 26 more

... and 2 files with indirect coverage changes

…there

timlind · 2023-12-11T13:37:32Z

scripts/chain-initiator/types.go

+	tokenregistrytypes.GenesisState
+
+	Registry Registry `json:"registry"`
+}


Aren't these types in the repo already? Sifgen is also encoding them, can that be used?

@timlind I am using them here at line 117 but the problem is that for some structures they contain numbers that the JSON parser does not understand unless you replace the type of those fields by json.Number and same goes for Duration type to string. that’s why we rely on those extra definitions to override the types of those fields.

pgoos

LGTM, great work 🚀

feat: add chain-initiator tool to use mainnet snapshot in localnet

ba28cf7

snobbee added 8 commits December 8, 2023 05:56

test: linter

50cc47d

feat: chain-initiator uses snapshot url and export genesis file from …

de70a64

…there

test: lint

18bf261

feat: chain-initiator include submit software upgrade proposal logic

4f0d1c8

fix: improved stability

7707aed

feat: improved chain initiator

7d4bcbe

test: fix lint

78d08fe

feat: binary url can be remote url or local path

26c73f4

timlind reviewed Dec 11, 2023

View reviewed changes

timlind approved these changes Dec 13, 2023

View reviewed changes

pgoos approved these changes Dec 13, 2023

View reviewed changes

Merge branch 'master' into feat/chain-initiator

8970c55

snobbee merged commit 0698422 into master Dec 13, 2023
6 of 7 checks passed

snobbee deleted the feat/chain-initiator branch December 13, 2023 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add chain-initiator tool to use mainnet snapshot in localnet #3435

feat: add chain-initiator tool to use mainnet snapshot in localnet #3435

snobbee commented Dec 7, 2023 •

edited

codecov bot commented Dec 7, 2023 •

edited

timlind Dec 11, 2023

snobbee Dec 12, 2023

pgoos left a comment

feat: add chain-initiator tool to use mainnet snapshot in localnet #3435

feat: add chain-initiator tool to use mainnet snapshot in localnet #3435

Conversation

snobbee commented Dec 7, 2023 • edited

codecov bot commented Dec 7, 2023 • edited

Codecov Report

timlind Dec 11, 2023

Choose a reason for hiding this comment

snobbee Dec 12, 2023

Choose a reason for hiding this comment

pgoos left a comment

Choose a reason for hiding this comment

snobbee commented Dec 7, 2023 •

edited

codecov bot commented Dec 7, 2023 •

edited