New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add database analyse-gas-usage command #10240
Commits on Nov 23, 2023
-
feat: add database analyse-gas-usage command
Add a command which will allow to analyse gas usage on the blockchain. The command can be executed by running: ```bash ./neard database analyse-gas-usage ``` The command will look at gas used in all of the blocks, shards and accounts, and print out an analysis report. This commit just adds the command, the implementation will be added in the following commits.
Configuration menu - View commit details
-
Copy full SHA for 7aa1745 - Browse repository at this point
Copy the full SHA 7aa1745View commit details -
Make it possible to analyse the last N blocks
Add a flag that can be used to specify how many latest blocks should be analysed using the analyse-gas-usage command. Now running: ```bash ./neard database analyse-gas-usage --last-blocks 1000 ``` Will go over the last 1000 blocks in the blockchain and analyse them.
Configuration menu - View commit details
-
Copy full SHA for 4eb31c8 - Browse repository at this point
Copy the full SHA 4eb31c8View commit details -
For each block we want to gather information about how much gas was used in every shard and account. Gas is used when transactions and receipts are executed, and every such execution produces an ExecutionOutcome, which is saved in the database. We can read all ExecutionOutcomes originating from a given shard from the database, and get all of the needed information from there. Every ExecutionOutcome contains the AccountId of the executor and the amount of gas that was burned during the execution. It's everything that is needed for analysing gas usage. Usage from all blocks is merged into a single instace of GasUsageStats and the interesting pieces of information are displayed for the user.
Configuration menu - View commit details
-
Copy full SHA for 67967f5 - Browse repository at this point
Copy the full SHA 67967f5View commit details -
Make it possible to analyse a range of block heights
Add a flag that can be used to specify a range of blocks to be analysed. Now running: ```bash ./neard database analyse-gas-usage --from-block-height 120 --to-block-height 130 ``` Will analyse 11 blocks with heights in range of [120, 130] The logic of converting command line arguments to block iterators has been extracted to the block_iterators module. In the future it'll be possible to reuse it for another command that wants to analyse some subset of blocks.
Configuration menu - View commit details
-
Copy full SHA for 202df41 - Browse repository at this point
Copy the full SHA 202df41View commit details -
Calculate the optimal split for each shard
For each shard, calculate the optimal account that the shard could be split at, so that gas usage on both sides of the split is similar. An exact split isn't always possible, because it's possible for one account to consume 40% of all gas, but the function tries its best to make it fair. Information about optimal splits is displayed for the user.
Configuration menu - View commit details
-
Copy full SHA for fe6875d - Browse repository at this point
Copy the full SHA fe6875dView commit details -
Find 10 biggest accounts by gas usage
Find the accounts that consume the most gas and display them in the analysis. This information is interesting, because it might turn out that one account consumes 50% of all gas.
Configuration menu - View commit details
-
Copy full SHA for b7d4709 - Browse repository at this point
Copy the full SHA b7d4709View commit details -
Configuration menu - View commit details
-
Copy full SHA for 555b947 - Browse repository at this point
Copy the full SHA 555b947View commit details
Commits on Nov 24, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 0e8c59a - Browse repository at this point
Copy the full SHA 0e8c59aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 473747f - Browse repository at this point
Copy the full SHA 473747fView commit details -
fix: fix calculation of used_gas_total in GasUsageInShard::merge()
GasUsageInShard::merge() merges results from two instances of GasUsageInShard into one. The merge is performed by adding information about each account from the other shard. Then used_gas_total is updated as well. However this is the wrong thing to do. Calling `add_used_gas` already increases `used_gas_total`, so doing it again doubles the total gas, which is incorrect. Fix the bug by removing the line that doubles the gas.
Configuration menu - View commit details
-
Copy full SHA for 2919011 - Browse repository at this point
Copy the full SHA 2919011View commit details -
Remove GasUsageInShard::used_gas_total
The field GasUsageInShard::used_gas_total keeps the total amount of gas used in a shard. The problem is that keeping it in sync with the rest of the struct requires special care and is prone to bugs. A mistake in this logic has already caused one bug in calculating shard splits. It's safer to get rid of the field and just calculate the total amount of used gas when needed. It doesn't take that long to calculate this value from scratch.
Configuration menu - View commit details
-
Copy full SHA for f087966 - Browse repository at this point
Copy the full SHA f087966View commit details -
Improve GasUsageInShard::calculate_split()
The previous implementation of calculate_split() worked okayish, but it had some faults. The first fault was that it split the shard into two halves where the left one consistend of accounts smaller than the boundary account. This doesn't match the logic that NEAR uses for boundary accounts. In NEAR the left half includes the boundary account, i.e it's <= boundary_acc, not < boundary_acc. The new function uses the same division as NEAR: (left <= split_account), (right > split_account). The second fault was that the old function looked for a split where gas_left >= gas_right. It isn't easy to prove that this is the optimal one - why is the left half supposed to be bigger? Let's implement it in a way that is easier to understand. The new calculate_split() looks for a split that minimizes the difference between the two halves. This makes sense, we want the halves to be as similar as possible. The commit also adds unit tests to make sure that calculate_split() works correctly.
Configuration menu - View commit details
-
Copy full SHA for f01c80e - Browse repository at this point
Copy the full SHA f01c80eView commit details -
Configuration menu - View commit details
-
Copy full SHA for d168c5d - Browse repository at this point
Copy the full SHA d168c5dView commit details -
Configuration menu - View commit details
-
Copy full SHA for b9469dd - Browse repository at this point
Copy the full SHA b9469ddView commit details -
Configuration menu - View commit details
-
Copy full SHA for 367e8f3 - Browse repository at this point
Copy the full SHA 367e8f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for c1641b2 - Browse repository at this point
Copy the full SHA c1641b2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 902728c - Browse repository at this point
Copy the full SHA 902728cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 81b328a - Browse repository at this point
Copy the full SHA 81b328aView commit details
Commits on Nov 29, 2023
-
fix: left half of shard split should be less than split_account
When splitting a shard into two halves the shard is divided into two on the split_account (boundary_account). The split should look like this: left < boundary_account right >= boundary_account Previously I misunderstood the semantics and implemented the split as if it was left <= boundary and right > boundary. Fix it.
Configuration menu - View commit details
-
Copy full SHA for 9abd600 - Browse repository at this point
Copy the full SHA 9abd600View commit details -
feat: display gas usage of all accounts as the percentage of shard total
Previously gas usage of some accounts was displayed as percentage of shard split half, but this was hard to reason about. Let's change it to a percentage of total gas usage on a shard.
Configuration menu - View commit details
-
Copy full SHA for a36ea54 - Browse repository at this point
Copy the full SHA a36ea54View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8d13264 - Browse repository at this point
Copy the full SHA 8d13264View commit details -
Configuration menu - View commit details
-
Copy full SHA for 913f1f8 - Browse repository at this point
Copy the full SHA 913f1f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 645a5a8 - Browse repository at this point
Copy the full SHA 645a5a8View commit details -
I guess VSCode froze and didn't run format on save :C
Configuration menu - View commit details
-
Copy full SHA for 95e43b9 - Browse repository at this point
Copy the full SHA 95e43b9View commit details