Bridges V2 - dynamic lanes #4427

bkontur · 2024-05-10T12:49:31Z

Migrated from the branch bridges-v2-migration/

TBD: more description when migration is done

Closes: paritytech/parity-bridges-common#2976

TODO

search and fix for comment: // TODO:(bridges-v2) - most of that stuff was introduced with free header execution: https://github.com/paritytech/polkadot-sdk/pull/4102 - more info in the comment bellow

* Revert "Revert new client (#2153)" This reverts commit 17cf7b596170419a1e9b9d39d7e329acdb1a87de. * add comment re cache to CachingClient (#2162) * do not wait for reader when sending items (#2163) * Use unsync cache + async-aware sync primitives (#2164) * use unsync cache + async-aware sync primitives * clippy * trigger CI * Revert "trigger CI" This reverts commit b5c1405492409b85abf552ba0b089c000824f722.

…ble (#2206) * Avoid using extra background task for TransactionTracker * Add docs

* final check before transaction submit * pass pre-encoded transaction to validate

* read best + best finalized headers from subscription * spelling * clippy

bkontur · 2024-05-23T09:15:57Z

@svyatonik Slava, could you please help with first TODO, just search in the code:
// TODO:(bridges-v2) - most of that stuff was introduced with free header execution: https://github.com/paritytech/polkadot-sdk/pull/4102 in the code base.

I was able to merge/port everything, but I was not sure about CheckAndBoostBridgeGrandpaTransactions and CheckAndBoostBridgeParachainsTransactions.

Relates to commit: 30355e2
Original PR: paritytech/parity-bridges-common#2584

Either PR to this PR or just directly push to this PR is ok, it's up to you :)

I didn't fix cumulus runtimes yet, I will do that when all commits are ported, so you don't need to fix them now, anyway they don't compile yet. For testing, if this command works, then it is ok for now:

cargo test -p bp-header-chain -p bp-messages -p bp-parachains -p bp-polkadot-core -p bp-relayers -p bp-runtime -p bp-test-utils -p bp-xcm-bridge-hub -p bp-xcm-bridge-hub-router -p bp-beefy  -p bp-westend -p bp-rococo -p bp-polkadot-bulletin -p bp-polkadot -p bp-kusama -p bp-bridge-hub-westend -p bp-bridge-hub-rococo -p bp-bridge-hub-cumulus -p bp-bridge-hub-kusama -p bp-bridge-hub-polkadot  -p bridge-runtime-common  -p relay-substrate-client -p substrate-relay-helper -p finality-relay -p messages-relay -p parachains-relay -p relay-utils -p equivocation-detector  -p pallet-bridge-beefy -p pallet-bridge-grandpa -p pallet-bridge-messages -p pallet-bridge-parachains -p pallet-bridge-relayers -p pallet-xcm-bridge-hub -p pallet-xcm-bridge-hub-router  --features runtime-benchmarks,try-runtime,integrity-test

svyatonik · 2024-05-23T13:02:12Z

@svyatonik Slava, could you please help with first TODO, just search in the code: // TODO:(bridges-v2) - most of that stuff was introduced with free header execution: https://github.com/paritytech/polkadot-sdk/pull/4102 in the code base.

I was able to merge/port everything, but I was not sure about CheckAndBoostBridgeGrandpaTransactions and CheckAndBoostBridgeParachainsTransactions.

Relates to commit: 30355e2 Original PR: paritytech/parity-bridges-common#2584

Either PR to this PR or just directly push to this PR is ok, it's up to you :)

I didn't fix cumulus runtimes yet, I will do that when all commits are ported, so you don't need to fix them now, anyway they don't compile yet. For testing, if this command works, then it is ok for now:
cargo test -p bp-header-chain -p bp-messages -p bp-parachains -p bp-polkadot-core -p bp-relayers -p bp-runtime -p bp-test-utils -p bp-xcm-bridge-hub -p bp-xcm-bridge-hub-router -p bp-beefy  -p bp-westend -p bp-rococo -p bp-polkadot-bulletin -p bp-polkadot -p bp-kusama -p bp-bridge-hub-westend -p bp-bridge-hub-rococo -p bp-bridge-hub-cumulus -p bp-bridge-hub-kusama -p bp-bridge-hub-polkadot  -p bridge-runtime-common  -p relay-substrate-client -p substrate-relay-helper -p finality-relay -p messages-relay -p parachains-relay -p relay-utils -p equivocation-detector  -p pallet-bridge-beefy -p pallet-bridge-grandpa -p pallet-bridge-messages -p pallet-bridge-parachains -p pallet-bridge-relayers -p pallet-xcm-bridge-hub -p pallet-xcm-bridge-hub-router  --features runtime-benchmarks,try-runtime,integrity-test

@bkontur Pushed commits that solve the mess with extensions

bkontur · 2024-05-23T13:09:25Z

@svyatonik Slava, could you please help with first TODO, just search in the code: // TODO:(bridges-v2) - most of that stuff was introduced with free header execution: https://github.com/paritytech/polkadot-sdk/pull/4102 in the code base.
I was able to merge/port everything, but I was not sure about CheckAndBoostBridgeGrandpaTransactions and CheckAndBoostBridgeParachainsTransactions.
Relates to commit: 30355e2 Original PR: paritytech/parity-bridges-common#2584
Either PR to this PR or just directly push to this PR is ok, it's up to you :)
I didn't fix cumulus runtimes yet, I will do that when all commits are ported, so you don't need to fix them now, anyway they don't compile yet. For testing, if this command works, then it is ok for now:
cargo test -p bp-header-chain -p bp-messages -p bp-parachains -p bp-polkadot-core -p bp-relayers -p bp-runtime -p bp-test-utils -p bp-xcm-bridge-hub -p bp-xcm-bridge-hub-router -p bp-beefy  -p bp-westend -p bp-rococo -p bp-polkadot-bulletin -p bp-polkadot -p bp-kusama -p bp-bridge-hub-westend -p bp-bridge-hub-rococo -p bp-bridge-hub-cumulus -p bp-bridge-hub-kusama -p bp-bridge-hub-polkadot  -p bridge-runtime-common  -p relay-substrate-client -p substrate-relay-helper -p finality-relay -p messages-relay -p parachains-relay -p relay-utils -p equivocation-detector  -p pallet-bridge-beefy -p pallet-bridge-grandpa -p pallet-bridge-messages -p pallet-bridge-parachains -p pallet-bridge-relayers -p pallet-xcm-bridge-hub -p pallet-xcm-bridge-hub-router  --features runtime-benchmarks,try-runtime,integrity-test
@bkontur Pushed commits that solve the mess with extensions

Slava, perfect, thank you, I will check and continue tomorrow

…ts-and-nits

* Define Chain::STATE_VERSION * Add vec_db module * Use VecDb instead of StorageProof for message delivery * Make sure that the TrustedVecDb is sorted * Address review comments * Run benchmarks on parent commit * Run benchmarks with new code * Fix test * Fix code review comments

* moved FromBridgedChainMessagesDeliveryProof to bp-messages * spelling * fix benchmarks compilation

* moved FromBridgedChainMessagesProof to bp-messages * fmt

* Use compact proofs for parachains * Remove StorageProofChecker * Cleanup Reuse messages proof generation from messages pallet (#2195) * reuse messages proof generation from messages pallet * incorrect merge fixed remaining TODOs after messages pallet Config refactoring (#2196) Return UntrustedVecDb from prove_storage() (#2197) Storage proofs related renamings (#2198) * Storage proofs related renamings * Leftovers * StorageSize -> StorageProofSize

* add test for maximal message dispatch weight * fixed weight computations - use kb for both size-related benchmarks * fix typo in weight formula * decreased maximal message size to 64Kb

* add LaneState enum and a field to lanes data * use OptionQuery in InboundLanes and OutboundLanes * ensure that lanes are opened when accessing them * an option to specify opened lanes in genesis config + fixed tests * added PR reference to TODO * fix failing benchmarks * spelling * get_or_init_data -> data

* Box messages proof argument in receive_messages_proof call * fix benchmark tests

* change LaneId underlying type from [u8; 4] to H256 * fixed typo * added some tests * spelling * started fixing testnets * uncommented call size test * changed RewardsAccountParams encoding + added values separator when computing LaneId * review suggestions

* StorageProofSize -> StorageSize * Rename benchmarks * StorageSize -> UnverifiedStorageProofParams * Fix clippy * Fix priority boost

* some useful stuff like LanesManager * clippy * more clippy * Error::LanesManager * {in, out}bound_lane -> active_{in, out}bound_lane * merge two impl blocks in one * fmt

* update async-trait to 0.1.69 * more clippy fixes

* Updated version according to Cumulus * Updated docs

* allow delivery confirmations on closed outbound lanes * fix benchmarks compilation

…nsactions (#2259)

)

* some code grooming: enable warn(missing_docs) for all piblic crates + added missing documentation + removed obsolete clippy/deny workarounds * removed strange allow + added comment related to other allow * removed incorrect_clone_impl_on_copy_type which is unknown to CI clippy

* XCM over bridge pallet (won't ever be merged to this repo): initial commit * try both bridge-id and lane-id * flush * flush * flush * flush * first prototype * started working on xcm-over-bridge tests * proper open_bridge_works test * more open_bridge tests * request_bridge_closure tests * tests for close_bridge * report_misbehavior tests * renaming xcm-over-bridge ad xcm-bridge-hub: we'll have two similar pallets for dynamic lanes/fees support. One will be deployed at the bridge hub (xcm-bridge-hub) and another one (xcm-bridge-hub-router) at sibling/parent chains to send messages to remote network over the bridge hub * added couple of TODOs * moved BridgeQueuesState here + TODO for implementing report_bridge_queues_state + fmt * fixing TODOs * fixing TODOs * moved bridge locations to primitives * added a couple of TODOs * ref issue from TODO * clippy and spelling * fix tests * another TODOs * typo * LaneState -> force_close_bridge * start_closing_the_bridge -> start_closing_bridge * removed Closing bridge state, so now we only have the Opened -> optional temporary Closed state -> None * spelling * started prototyping suspend+resume on misbehavior * continue prototyping * more tests for open_bridge * more tests for close_bridge * more tests for report_misbehavior * started tests for resume_misbehaving_bridges * implemented tests for resume_misbehaving_bridges * remove UnsuspendedChannelWithMisbehavingOwner because now, when all bridges are resumed at once + we assume that the inbound XCM channel is suspended immediately it is no longer actual * added TODO * add comment re misbehavior reporter if he's associated with the bridge owner * ignored clippy * fmt * use versioned XCM structs in public interfaces and storage + Box XCM locations where possible * spelling * removed TODO in favor of paritytech/parity-bridges-common#2257 * LocalChannelManager -> LocalXcmChannelManager * removed code specific to #2233, because it isn't the only option now * removemisbehavior mentions * also temporary (?) remove BridgesByLocalOrigin because the storage format will likely change to be able to resume bridges from the on_iniitalize/on_idle * AllowedOpenBridgeOrigin -> OpenBridgeOrigin * - TooManyBridgesForLocalOrigin * use bridge_destination_relative_location in args * better impl of strip_low_level_junctions * get rid of xcm_into_latest * clippy * added #![warn(missing_docs)] to new crates

) * implement ExportXcm and MessageDispatch for pallet-xcm-bridge-hub * spelling

… pallet-xcm-bridge-hub (#2261) Xcm bridge hub router v2 (backport to master branch) (#2312) * copy new pallet (palle-xcm-bridge-hub-router) from dynamic-fees-v1 branch * added remaining traces of pallet-xcm-bridge-hub-router * added comment about sharing delivery fee factor between all bridges, opened by this chain * spelling * clippy Implement additional require primitives for dynamic fees directly for pallet-xcm-bridge-hub (#2261) * added backoff mechanism to inbound bridge queue * impl backpressure in the XcmBlobHaulerAdapter * leave TODOs * BridgeMessageProcessor prototype * another TODO * Revert "also temporary (?) remove BridgesByLocalOrigin because the storage format will likely change to be able to resume bridges from the on_iniitalize/on_idle" This reverts commit bdd7ae11a8942b58c5db6ac6d4e7922aa28cece4. * prototype for QueuePausedQuery * implement ExportXcm and MessageDispatch for pallet-xcm-bridge-hub * spelling * flush * small comments to myself * more backports from dynamic-fees-v1 * use new pallet as exporter and dispatcher in Millau * use new pallet as exporter and dispatcher in Rialto * use new pallet as exporter and dispatcher in RialtoParachain * flush * fix remaining compilation issues * warnings + fmt * fix tests * LocalXcmChannelManager * change lane ids * it works! * remove bp-xcm-bridge-hub-router and use LocalXcmChannelManager everywhere * removed commented code * cleaning up * cleaning up * cleaning up * - separated BridgeId and LaneId - BridgeId now uses versioned universal locations - added missing stuff to exporter.rs * OnMessagesDelivered is back * start using bp-xcm-bridge-hub as OnMessagesDelivered * cleaning up * spelling * fix stupid issues * Backport latest relevant dynamic fees changes from v1 to v2 (#2372) * backport latest relevant dynamic fees changes from v1 to v2 * fix comment Added remaining unit tests for pallet-xcm-bridge-hub (#2499) * added backoff mechanism to inbound bridge queue * impl backpressure in the XcmBlobHaulerAdapter * leave TODOs * BridgeMessageProcessor prototype * another TODO * Revert "also temporary (?) remove BridgesByLocalOrigin because the storage format will likely change to be able to resume bridges from the on_iniitalize/on_idle" This reverts commit bdd7ae11a8942b58c5db6ac6d4e7922aa28cece4. * prototype for QueuePausedQuery * implement ExportXcm and MessageDispatch for pallet-xcm-bridge-hub * spelling * flush * small comments to myself * more backports from dynamic-fees-v1 * use new pallet as exporter and dispatcher in Millau * use new pallet as exporter and dispatcher in Rialto * use new pallet as exporter and dispatcher in RialtoParachain * flush * fix remaining compilation issues * warnings + fmt * fix tests * LocalXcmChannelManager * change lane ids * it works! * remove bp-xcm-bridge-hub-router and use LocalXcmChannelManager everywhere * removed commented code * cleaning up * cleaning up * cleaning up * - separated BridgeId and LaneId - BridgeId now uses versioned universal locations - added missing stuff to exporter.rs * OnMessagesDelivered is back * start using bp-xcm-bridge-hub as OnMessagesDelivered * cleaning up * spelling * fix stupid issues * added remaining unit tests for pallet-xcm-bridge-hub fixed benchmarks (#2504) Remove pallet_xcm_bridge_hub::SuspendedBridges (#2505) * remove pallet_xcm_bridge_hub::SuspendedBridges * apply review suggestions

…-messages (#2580)

* moved all call-related definitions to primitives * moving refund extension to relayers pallet * cleaning up * fix compilation + spelling

paritytech-cicd-pr · 2024-05-30T10:23:38Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6349884

bkontur self-assigned this May 13, 2024

bkontur force-pushed the bko-bridges-v2 branch from 9551535 to 6426d31 Compare May 14, 2024 08:27

bkontur mentioned this pull request May 14, 2024

Companion for Bridges V2 - relay client part paritytech/parity-bridges-common#2988

Draft

3 tasks

bkontur force-pushed the bko-bridges-v2 branch 6 times, most recently from a206bd7 to e6ea0cc Compare May 17, 2024 07:33

bkontur mentioned this pull request May 17, 2024

Bridges - changes for Bridges V2 - relay client part #4494

Open

2 tasks

bkontur force-pushed the bko-bridges-v2 branch 5 times, most recently from bae73d2 to ec0ce63 Compare May 22, 2024 23:12

svyatonik and others added 6 commits May 23, 2024 10:34

Avoid using extra background task for polling subscription when possi…

411de12

…ble (#2206) * Avoid using extra background task for TransactionTracker * Add docs

Polish the Subscription API (#2208)

daaa24c

Final check before transaction submit (#2351)

e38f989

* final check before transaction submit * pass pre-encoded transaction to validate

fix grandpa benchmarks (#2577)

fb9d1c8

Get best and finalized head updates from subscription (#2166)

6a021c8

* read best + best finalized headers from subscription * spelling * clippy

bkontur force-pushed the bko-bridges-v2 branch from ec0ce63 to 30355e2 Compare May 23, 2024 08:38

bkontur and others added 5 commits May 30, 2024 11:10

Merge remote-tracking branch 'origin/master' into bko-bridges-backpor…

fcbf025

…ts-and-nits

Merge remote-tracking branch 'origin/master' into bko-bridges-backpor…

6cbab96

…ts-and-nits

Moved FromBridgedChainMessagesDeliveryProof to bp-messages (#2169)

f7b819a

* moved FromBridgedChainMessagesDeliveryProof to bp-messages * spelling * fix benchmarks compilation

Moved FromBridgedChainMessagesProof to bp-messages (#2170)

cc3a1e4

* moved FromBridgedChainMessagesProof to bp-messages * fmt

serban300 and others added 28 commits May 30, 2024 12:15

prune messages from confirmation tx, not from the on_idle (#2211)

616900a

Fix maximal message dispatch weight (#2219)

046c617

* add test for maximal message dispatch weight * fixed weight computations - use kb for both size-related benchmarks * fix typo in weight formula * decreased maximal message size to 64Kb

Box messages proof argument in receive_messages_proof call (#2222)

d26eded

* Box messages proof argument in receive_messages_proof call * fix benchmark tests

bump Substrate, Polkadot and Cumulus (#2223)

245ae81

Add test method for growing storage proof size (#2224)

c5fb7e6

fix some nightly errors (#2225)

7db7606

Renamings (#2226)

cff071b

* StorageProofSize -> StorageSize * Rename benchmarks * StorageSize -> UnverifiedStorageProofParams * Fix clippy * Fix priority boost

Introduce MessagesLaneManager (#2227)

664fe09

* some useful stuff like LanesManager * clippy * more clippy * Error::LanesManager * {in, out}bound_lane -> active_{in, out}bound_lane * merge two impl blocks in one * fmt

Fix clippy (#2240)

441be0b

* update async-trait to 0.1.69 * more clippy fixes

Backport from staging (#2243)

0acdd64

* Updated version according to Cumulus * Updated docs

Allow delivery confirmations on closed outbound lanes (#2257)

545e7c2

* allow delivery confirmations on closed outbound lanes * fix benchmarks compilation

remove internal code that pretends to support cross-lane delivery tra…

2af54af

…nsactions (#2259)

introduce partial bp-xcm-bridge-hub - just LocalXcmChannelManager (#2265

d0f0681

)

Implement ExportXcm and MessageDispatch for pallet-xcm-bridge-hub (#2285

1a810a8

) * implement ExportXcm and MessageDispatch for pallet-xcm-bridge-hub * spelling

emit events from bridge hub router (#2760)

7ea7bcc

use saturating_len where possible (#2316)

d1fa271

move messages_call_ext.rs from bridge-runtime-common to pallet-bridge…

4ade73c

…-messages (#2580)

moved all call-related definitions to primitives (#2583)

f519e49

Move refund extension to relayers pallet (#2584)

2bd8a0d

* moved all call-related definitions to primitives * moving refund extension to relayers pallet * cleaning up * fix compilation + spelling

fix compilation

2108641

remove priority_calculator.rs

9e55cce

fix extensions

75ee05d

bkontur force-pushed the bko-bridges-v2 branch from feb2434 to 75ee05d Compare May 30, 2024 10:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bridges V2 - dynamic lanes #4427

Bridges V2 - dynamic lanes #4427

bkontur commented May 10, 2024 •

edited

bkontur commented May 23, 2024

svyatonik commented May 23, 2024

bkontur commented May 23, 2024

paritytech-cicd-pr commented May 30, 2024

Bridges V2 - dynamic lanes #4427

Are you sure you want to change the base?

Bridges V2 - dynamic lanes #4427

Conversation

bkontur commented May 10, 2024 • edited

TODO

bkontur commented May 23, 2024

svyatonik commented May 23, 2024

bkontur commented May 23, 2024

paritytech-cicd-pr commented May 30, 2024

bkontur commented May 10, 2024 •

edited