Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add database analyse-gas-usage command #10240

Merged
merged 24 commits into from Nov 30, 2023

Commits on Nov 23, 2023

  1. feat: add database analyse-gas-usage command

    Add a command which will allow to analyse gas usage on the blockchain.
    The command can be executed by running:
    ```bash
    ./neard database analyse-gas-usage
    ```
    The command will look at gas used in all of the blocks,
    shards and accounts, and print out an analysis report.
    
    This commit just adds the command, the implementation will
    be added in the following commits.
    jancionear committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    7aa1745 View commit details
    Browse the repository at this point in the history
  2. Make it possible to analyse the last N blocks

    Add a flag that can be used to specify how many latest blocks
    should be analysed using the analyse-gas-usage command.
    
    Now running:
    ```bash
    ./neard database analyse-gas-usage --last-blocks 1000
    ```
    
    Will go over the last 1000 blocks in the blockchain
    and analyse them.
    jancionear committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    4eb31c8 View commit details
    Browse the repository at this point in the history
  3. Implement analysing gas usage

    For each block we want to gather information about how much gas was
    used in every shard and account.
    
    Gas is used when transactions and receipts are executed, and every
    such execution produces an ExecutionOutcome, which is saved in the
    database.
    We can read all ExecutionOutcomes originating from a given shard
    from the database, and get all of the needed information from there.
    Every ExecutionOutcome contains the AccountId of the executor
    and the amount of gas that was burned during the execution.
    It's everything that is needed for analysing gas usage.
    
    Usage from all blocks is merged into a single instace of GasUsageStats
    and the interesting pieces of information are displayed for the user.
    jancionear committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    67967f5 View commit details
    Browse the repository at this point in the history
  4. Make it possible to analyse a range of block heights

    Add a flag that can be used to specify a range of blocks
    to be analysed.
    
    Now running:
    ```bash
    ./neard database analyse-gas-usage --from-block-height 120 --to-block-height 130
    ```
    
    Will analyse 11 blocks with heights in range of [120, 130]
    
    The logic of converting command line arguments to block iterators
    has been extracted to the block_iterators module. In the future
    it'll be possible to reuse it for another command that wants to
    analyse some subset of blocks.
    jancionear committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    202df41 View commit details
    Browse the repository at this point in the history
  5. Calculate the optimal split for each shard

    For each shard, calculate the optimal account that the shard could
    be split at, so that gas usage on both sides of the split is similar.
    An exact split isn't always possible, because it's possible for one account
    to consume 40% of all gas, but the function tries its best to make it fair.
    
    Information about optimal splits is displayed for the user.
    jancionear committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    fe6875d View commit details
    Browse the repository at this point in the history
  6. Find 10 biggest accounts by gas usage

    Find the accounts that consume the most gas and display
    them in the analysis. This information is interesting,
    because it might turn out that one account consumes 50%
    of all gas.
    jancionear committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    b7d4709 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    555b947 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. Configuration menu
    Copy the full SHA
    0e8c59a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    473747f View commit details
    Browse the repository at this point in the history
  3. fix: fix calculation of used_gas_total in GasUsageInShard::merge()

    GasUsageInShard::merge() merges results from two instances of GasUsageInShard into one.
    The merge is performed by adding information about each account from the other shard.
    Then used_gas_total is updated as well. However this is the wrong thing to do.
    Calling `add_used_gas` already increases `used_gas_total`, so doing it again
    doubles the total gas, which is incorrect.
    
    Fix the bug by removing the line that doubles the gas.
    jancionear committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    2919011 View commit details
    Browse the repository at this point in the history
  4. Remove GasUsageInShard::used_gas_total

    The field GasUsageInShard::used_gas_total keeps the total amount of
    gas used in a shard. The problem is that keeping it in sync with the
    rest of the struct requires special care and is prone to bugs.
    A mistake in this logic has already caused one bug in calculating
    shard splits. It's safer to get rid of the field and just calculate
    the total amount of used gas when needed. It doesn't take that long
    to calculate this value from scratch.
    jancionear committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    f087966 View commit details
    Browse the repository at this point in the history
  5. Improve GasUsageInShard::calculate_split()

    The previous implementation of calculate_split() worked okayish,
    but it had some faults.
    
    The first fault was that it split the shard into two halves where the left
    one consistend of accounts smaller than the boundary account. This doesn't match
    the logic that NEAR uses for boundary accounts. In NEAR the left half includes
    the boundary account, i.e it's <= boundary_acc, not < boundary_acc.
    The new function uses the same division as NEAR: (left <= split_account), (right > split_account).
    
    The second fault was that the old function looked for a split where gas_left >= gas_right.
    It isn't easy to prove that this is the optimal one - why is the left half supposed to be bigger?
    Let's implement it in a way that is easier to understand. The new calculate_split()
    looks for a split that minimizes the difference between the two halves. This makes sense,
    we want the halves to be as similar as possible.
    
    The commit also adds unit tests to make sure that calculate_split() works correctly.
    jancionear committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    f01c80e View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d168c5d View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    b9469dd View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    367e8f3 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    c1641b2 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    902728c View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    81b328a View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. fix: left half of shard split should be less than split_account

    When splitting a shard into two halves the shard is divided
    into two on the split_account (boundary_account).
    The split should look like this:
    left < boundary_account
    right >= boundary_account
    
    Previously I misunderstood the semantics and implemented the split
    as if it was left <= boundary and right > boundary. Fix it.
    jancionear committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    9abd600 View commit details
    Browse the repository at this point in the history
  2. feat: display gas usage of all accounts as the percentage of shard total

    Previously gas usage of some accounts was displayed as percentage
    of shard split half, but this was hard to reason about.
    Let's change it to a percentage of total gas usage on a shard.
    jancionear committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    a36ea54 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8d13264 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    913f1f8 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    645a5a8 View commit details
    Browse the repository at this point in the history
  6. fix: fix formatiing

    I guess VSCode froze and didn't run format on save :C
    jancionear committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    95e43b9 View commit details
    Browse the repository at this point in the history