[META] Revisit gas price strategies

[konradkonrad]

Abstract

In the course of the scenario player runs on goerli we encountered a lot of problems with too low gas price estimations, when the gas prices in the network fluctuate a lot. The naive assumption is, that the web3-py provided gas price strategy for fast should still work in those cases, but apparently it does not. On top of that, we don't have any strategy for speeding up "stuck" transfers that were already sent. We should

• at least make sure we are using consistent gas price strategies across the board or have valid justifications where we deviate from that
• study the failure modes of the strategies currently in use in more detail
• explore mechanisms to speed up important transactions, if they are "stuck" (i.e. take "longer than expected" to be mined)

Motivation

Raiden does not expose interactive gas price adjustments -- it also is not straight forward to introduce such a feature, because on chain transactions are most of the time the worst-case scenario and need to happen automatically (e.g. "publish a secret", "update the non closing balance proof"). Also, signed transactions do not have an expiry. Failure to have a transaction mined before a deadline can lead to actual token loss or at least extra cost for the gas wasted on the too-late transaction plus service cost. There is also a usability component to this: if transactions are stuck due to a too low gas price, the channels can stay in an unusable state for long periods of time.
There is also the other side: paying too high gas prices takes away some of the monetary benefits of using L2 for token transfers. Therefore optimizations of the employed gas price strategy have a strong impact on the viability of Raiden.

Other projects

While this ticket is concerned with the python implementation of the Raiden client, gas price strategies are important, too, for

• the monitoring service
• the scenario player

Backwards Compatibility

This does not have implications for backwards compatibility.

Solution

• provide feedback for the transactions in a notification like manner to the web ui
• provide an endpoint to top up transactions based on the old gas price
• raiden client must be able to replace a transaction and wait for the new one
  • reuse the same nonce
  • replace waits on the old transaction
• provide the functionality in the web ui to top up pending transactions

Internal design / implementation

Current state

Transactions are "implicitly" executed via either

• The RaidenEventHandler as the result of events dispatched from state transitions via the state machine.
• The RestAPI as the result of a user action (e.g. channel open)

In both cases the resulting transactions are not visible / accessible from outside the executing functions and therefore can't be interacted with.

Proposed solution (July 2021)

To allow transactions to be both visible to the user as well as replaceable, cancelable etc. a TransactionHandler (or similar name) should be added that takes over the handling of transactions.

This handler should support the following actions:

• Submit and wait on a tx (independent on the actual tx hash)
• List currently pending tx
• Replace a tx
  Either:
  • With an identical one with increased gas price
  • With a dummy tx that will fail (to allow for canceling txs)

The transaction handler should be accessible via e.g. the RaidenService and be used in all places where we currently directly submit transactions.

Pseudo interface:

• TransactionHandler
  • .submit(tx: TransactionEstimated) -> TransactionMined
  • .__iter__() -> Iterator[<SomeTxIdThatIsNotBasedOnTheActualHash>]
  • .increase_gas_price(tx_id: <SomeTxIdThatIsNotBasedOnTheActualHash>, new_gas_price: int)
  • .cancel(tx_id: <SomeTxIdThatIsNotBasedOnTheActualHash>)

(Alternatively instead of using opaque ids an object could be returned that has .increase_gas_price() and .cancel() methods).

In the next step automated handling of gas price increases can be added on top.

[konradkonrad]

@ezdac feel free to add anything I missed above!

[ezdac]

Some considerations for a manually implemented gas-price strategy:

Constraints

One problem in calculating gas-prices is that of node-locality: we can only observe the current gas-prices in our mempool and don't have a global view of the network's txpool. A better gas-oracle would aggregate the txpool of several nodes, but we don't want to rely on external services too much.
Historical prices are available to our node, but of course past performance doesn't indicate future performance.
We will never be able to 100% guarantee transactions going through.

Goals

Transactions are supposed to be rare in Raiden and there are 2 (mostly adverse) dimensions to optimise for - economic feasibility and block-inclusion time.

I would consider us having two kinds of transactions:

Non-Time critical transactions:

• channel opening, deposit, etc.

Here timing might not be important, and a user very likely favours economic feasibility over time.

Settlement transactions

• updateNonClosingBalanceProof
• withdraw, settle, close - user-dependent on opportunity cost by locked funds

Those transactions are mostly time critical and for the challenge period it is crucial to get tx's in the blockchain in order to not lose funds.

Possible solution:

• as a "default" use time-based historical gas-price estimators, for simplicity e.g. based on the web3 construct_time_based_gas_price_strategy, which can be parameterised for sample-size and inclusion-probability (but does not consider fluctuation magnitude)
• for transactions that require finality at some future block x = current_block + y, make the gas price strategy dependent on x - current_block , where the closer the function gets to +0, the higher the gas price will be.

If initial tx sends will fail (being stuck in the txpool for longer than expected), a follow-up transaction with significatly higher gas price should be made.
Since it is possible that multiple transactions will get mined, the total cumulative gas-cost sent must not be higher than the economic value the transaction is representing (e.g. balances in a channel).
This means a balance has to be found between:

• frequent top-ups with small gas increment - keeps total gas-cost down
• infrequent top-ups with high gas increment - increases inclusion probability

For a schematic representation of the idea:

• y axis: total cumulative gas-cost
• x axis: delta-block = current_block - initial-transaction-block
• vertical lines represent tx-sends at the calculated gas-price
• the horizontal line ("tx-value") is the value the tx represents for the node in Wei

This curve initially targets the normal time based gas-price strategy, and scales up the importance and thus gas-price the closer we get to the target block. It assumes that we observe high upwards fluctuation in gas-price that make the historical, running average based strategy unfeasible for future predictions.
We don't care too much about overpaying a transaction at this point.

Problems:

• MEV, tx's could be taken hostage by the miners and gas-price bumped (this is a general problem)
• if done wrong, we could overpay the tx significantly

[fredo]

Since it is possible that multiple transactions will get mined, the total cumulative gas-cost sent must not be higher than the economic value the transaction is representing (e.g. balances in a channel).

What are multiple transactions regarding one channel? Like UpdateNonClosingBalanceProof and Settle?

[konradkonrad]

Settlement transactions

updateNonClosingBalanceProof
withdraw, settle, close - user-dependent on opportunity cost by locked funds

Those transactions are mostly time critical and for the challenge period it is crucial to get tx's in the blockchain in order to not lose funds.

Don't forget the SecretRegistry!

[konradkonrad]

• We should also have a way to display transactions in flight
• We should expose functionality to "speed up" transactions in flight by API call

[ezdac]

In general, I believe solving this gas-induced finality problem is really hard (if not impossible), and there is potential that if we're not careful with this we overspend.

We should expose functionality to "speed up" transactions in flight by API call

That would be a good idea for a first step in a multi-step process - providing a manual endpoint where the user can decide how much the tx is worth to him

What are multiple transactions regarding one channel? Like UpdateNonClosingBalanceProof and Settle?
It might be a good optimisation to consider information from all our recently pending transactions in getting a better gas-price estimator, but this could get complex really fast.

What I meant is "topup" transactions, that represent the same action we want to take, but (hopefully) supersede the older, pending transaction. There is the possibility that both the pending AND the topup transaction will get included in a block, which is not good for the economic feasibility of the action. This is especially true if we are e.g. close to the cutoff block in the curve above, and the stakes in gas are high.
The worst case scenario would be:

• we topup transactions frequently, so there are a lot of tx's for the same "action" inflight
• gas-price drop rapidly, and a node is willing to put all tx in the block
• like that we could pay more gas-cost for the action than the action is actually worth (e.g. balance dispute)

So the sum of gas-cost for all inflight tx for a particular action may never be higher than the action is worth

[ezdac]

I wonder how all the front-running bots do this - there must be quite some research and competing algorithms in this field

[fredo]

What I meant is "topup" transactions, that represent the same action we want to take, but (hopefully) supersede the older, pending transaction. There is the possibility that both the pending AND the topup transaction will get included in a block, which is not good for the economic feasibility of the action. This is especially true if we are e.g. close to the cutoff block in the curve above, and the stakes in gas are high.

This is impossible. Top up transactions share the same (Ethereum) nonce. Basically, it is the same transaction with the same nonce but a higher gas price. Once, any of these transactions gets included into a block, all other transactions become invalid as they use an already used nonce. Thus including two transactions from the same address with the same nonce into a block, makes the whole block invalid. Thus, economic feasability should not be affected by this.

[fredo]

I wonder how all the front-running bots do this - there must be quite some research and competing algorithms in this field

To my knowledge, these bots simply scan the transaction pool of pending transactions and base their price on the findings in txpool. typically frontrunning bots want to frontrun a specific transaction, so they only need to put their transaction in with the higher gas price (often seen with only higher by one WEI).

[ezdac]

it is the same transaction with the same nonce but a higher gas price

Mh, I misunderstood something conceptually here severely. True, we use the same nonce here and don't increase it.
That makes things way easier

[fredo]

One other thing which comes to my mind.

Do we have taken into account that EIP-1559 is scheduled for mid July which will have a big impact on the whole gas pricing mechanism? If I got my head around it correctly, it will much more reliable to have a certain price at which you can be sure that your transaction gets included into the block. Only the worst case scenario will lead to the same model as we have it now (which should happen very rarely).
Block size can increase dynamically up to 25 million (if I remember correctly) and the protocol's equilibrium is to have the block always half full.

[ezdac]

Just skimmed EIP-1559, here is a quick and rough TL;DR for the parts relevant to our discussion:

• the base_fee_per_gas is deterministically calculated by the parent blocks gas usage (as @fredo said optimising for 50% utilised blocks @ 12.5 million gas) - this will get burned
• you can additionally tip miners with the priority_fee for tx ordering / higher probability for block-inclusion
• transactions can specify a max_fee_per_gas, so that the effective gas-price for a transaction is capped as of:

# priority fee is capped because the base fee is filled first
priority_fee_per_gas = min(transaction.max_priority_fee_per_gas, transaction.max_fee_per_gas - block.base_fee_per_gas)
# signer pays both the priority fee and the base fee
effective_gas_price = priority_fee_per_gas + block.base_fee_per_gas

• Legacy transactions will still be compatible, but don't benefit from the new pricing mechanism, because
  gas_price = base_fee_per_gas + priority_fee_per_gas (I believe), so that full amount of the gas_price is consumed.

The overall problem we try to solve remains still the same, although it might be less likely that we have to topup transactions in order to get them included before a target block.

For our gas-strategy this would potentially mean:

• the first transaction will include no or a very small max_priority_fee_per_gas,
• we wait some amount of time, and determine wether the tx was included
• if it was:
  • we're done 👍
• if not:
  • for following transactions we then would only topup the max_priority_fee_per_gas as a function of block-height until the cutoff block
• For all tx's, we cap the max_fee_per_gas with a value that is based on the aforementioned tx-value in order to avoid the very rare edgecase of paying more gas than the tx is worth

Probably it should be something like:

where:

• f is the max_priority_fee_per_gas
• x is the current block
• x0 is our target block (e.g. end of settlement period)
• a can be a parameter to scale the y value around our first transaction, so this e.g depends on the settlement-time

The price-capping around x0-x=0 will be done via the max_priority_fee_per_gas (tbd)

[fredo]

Decision by the team:

As a first step, expose an endpoint where a user can top up manually transactions and show transactions. This is not best UX but a user always can make transactions not starve.

Afterwards we can automise and optimize.

Additionally, we wait for environment change with EIP-1559