feat: chain sync projection primitives #520

mkazlauskas · 2022-11-13T18:29:54Z

Context

In order to create a database that is optimized for our provider queries, we'll need to project events from Chain Sync protocol

Proposed Solution

See README of the new projection package and the last commit (cc4bcea)

Important Changes Introduced

See the commit log.

Follow-up work

projection:

Create an abstraction for "store projectors" and implementation for stake pools and stake keys
Implement postgres store/repository for stake pools and stake keys

ogmios:

Complete the Ogmios transaction to core mapping. This PR only added mapping of certificates, the rest of transaction fields are currently not mapped.
Create a source observable from Ogmios ChainSync client. Also, create an abstraction to initialize evtCache$ for withRolledBackEvents

golden-test-generator:

Write tests for existing functionality
Start Ogmios ChainSync from some block (not necessarily from origin)

rhyslbw

Looks really nice so far @mkazlauskas!

8a41a2f seems like a fix.
Let's drop cae91f0 given there's no implementation to give us test coverage.
I think we should be keeping the projection package free from Ogmios implementations, as it makes sense for those to be scoped to the ogmios package.

yarn-project.nix

packages/core/src/Provider/Provider.ts

packages/projection/package.json

packages/projection/src/operators/withRolledBackEvents.ts

mkazlauskas · 2022-11-14T14:59:58Z

Addressed requested changes and added 1 of my own: b548aec replaced blockHeight with slot in CertificatePointer because this type is very similar to the pointer used in 'pointer addresses', and it uses slots. Better keep them aligned.

Ready for re-review.

packages/golden-test-generator/src/ChainSync/chainSync.ts

packages/util-rxjs/test/blockingWithLatestFrom.test.ts

mirceahasegan

My current review is a bit shallow, but it looks good to me 👽.
I would prefer a more in-depth re-review after the squash.

rhyslbw

Now would be a good time to remove the ignore for the GTG package if it's able to be made compliant without to much effort.

packages/wallet/package.json

packages/projection/package.json

packages/projection/README.md

packages/core/src/CardanoNode/types/CardanoNode.ts

packages/golden-test-generator/src/index.ts

Rename the existing 'get-blocks' command to 'chain-sync' add extend it to support simulating rollbacks. It can still be used to simply get blocks without specifying any rollbacks, but command name and the resulting data structure has changed.

… to use toSerializableObj util add metadata for supplied blockHeights option export getBlockHeaderAndHash

tsc was implicitly casting to an invalid type in some situations

implement coreToCsl genesis key delegation conversion

feat(core): add addressNetworkId and createRewardAccount utils BREAKING CHANGE: rename block types - CompactBlock -> BlockInfo - Block -> ExtendedBlockInfo BREAKING CHANGE: hoist ogmiosToCore to ogmios package

translating the entire block is a lot more expensive than just header

…f ogmios rename 'chain-sync' command to 'chain-sync-events'

feat(util-rxjs): add ObservableType<T> type util feat(core): add Chain Sync types

rhyslbw

Stellar work @mkazlauskas 🚀

mirceahasegan

Great work! 🚀

mkazlauskas requested review from rhyslbw and a team November 13, 2022 18:29

mkazlauskas force-pushed the feat/generalized-block-handler branch from f390ac1 to cc4bcea Compare November 13, 2022 20:57

rhyslbw requested changes Nov 14, 2022

View reviewed changes

yarn-project.nix Outdated Show resolved Hide resolved

packages/core/src/Provider/Provider.ts Show resolved Hide resolved

packages/projection/package.json Outdated Show resolved Hide resolved

packages/projection/src/operators/withRolledBackEvents.ts Outdated Show resolved Hide resolved

mkazlauskas force-pushed the feat/generalized-block-handler branch 2 times, most recently from ba7b405 to b548aec Compare November 14, 2022 14:48

mkazlauskas requested review from rhyslbw and a team November 14, 2022 15:00

mirceahasegan reviewed Nov 14, 2022

View reviewed changes

packages/golden-test-generator/src/ChainSync/chainSync.ts Outdated Show resolved Hide resolved

mirceahasegan reviewed Nov 14, 2022

View reviewed changes

packages/util-rxjs/test/blockingWithLatestFrom.test.ts Outdated Show resolved Hide resolved

mkazlauskas force-pushed the feat/generalized-block-handler branch from 31bd0d3 to 4524a62 Compare November 15, 2022 15:51

mkazlauskas requested a review from mirceahasegan November 15, 2022 15:52

mirceahasegan previously approved these changes Nov 16, 2022

View reviewed changes

mkazlauskas dismissed mirceahasegan’s stale review via 0205fbe November 16, 2022 14:33

mkazlauskas force-pushed the feat/generalized-block-handler branch from 4524a62 to 0205fbe Compare November 16, 2022 14:33

mkazlauskas requested a review from mirceahasegan November 16, 2022 14:33

mkazlauskas force-pushed the feat/generalized-block-handler branch from 0205fbe to 35c22a4 Compare November 16, 2022 14:41

rhyslbw requested changes Nov 17, 2022

View reviewed changes

rhyslbw previously approved these changes Nov 18, 2022

View reviewed changes

mkazlauskas added 10 commits November 18, 2022 09:20

fix(golden-test-generator): correct path of package.json

aa02fec

feat(golden-test-generator): add block range support to chain-sync

66626a3

fix(golden-test-generator)!: preserve BigInts by replacing serializer…

d951df2

… to use toSerializableObj util add metadata for supplied blockHeights option export getBlockHeaderAndHash

feat(util-rxjs): add blockingWithLatestFrom and toEmpty operators

d2e84a9

refactor(core): make 28 and 32 byte hash types non-generic

a7f90b7

tsc was implicitly casting to an invalid type in some situations

fix(core): correct hash types in genesis key delegation certificate

10d1f2b

implement coreToCsl genesis key delegation conversion

feat!: implement ogmiosToCore certificates mapping

aef2e8d

feat(core): add addressNetworkId and createRewardAccount utils BREAKING CHANGE: rename block types - CompactBlock -> BlockInfo - Block -> ExtendedBlockInfo BREAKING CHANGE: hoist ogmiosToCore to ogmios package

fix(ogmios): map prevHash='genesis' to undefined

ad47bbb

feat(ogmios): add ogmiosToCore.blockHeader

08fc7dc

translating the entire block is a lot more expensive than just header

mkazlauskas added 6 commits November 18, 2022 09:20

feat(ogmios): add ogmiosToCore.genesis

c48f6d4

chore: bump ogmios to 5.5.7

44f7cc8

feat(golden-test-generator)!: export core chain-sync events instead o…

2df226c

…f ogmios rename 'chain-sync' command to 'chain-sync-events'

feat: initial projection implementation

8a93d8d

feat(util-rxjs): add ObservableType<T> type util feat(core): add Chain Sync types

docs: update example webpack config path

bd8f950

build: remove golden-test-generator from .eslintignore

6df9255

mkazlauskas dismissed rhyslbw’s stale review via 6df9255 November 18, 2022 07:26

mkazlauskas force-pushed the feat/generalized-block-handler branch from 881751f to 6df9255 Compare November 18, 2022 07:26

rhyslbw self-requested a review November 18, 2022 07:38

rhyslbw approved these changes Nov 18, 2022

View reviewed changes

mirceahasegan approved these changes Nov 18, 2022

View reviewed changes

rhyslbw merged commit 91b0fc2 into master Nov 18, 2022

rhyslbw deleted the feat/generalized-block-handler branch November 18, 2022 09:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: chain sync projection primitives #520

feat: chain sync projection primitives #520

mkazlauskas commented Nov 13, 2022 •

edited

rhyslbw left a comment

mkazlauskas commented Nov 14, 2022

mirceahasegan left a comment •

edited

rhyslbw left a comment

rhyslbw left a comment

mirceahasegan left a comment

feat: chain sync projection primitives #520

feat: chain sync projection primitives #520

Conversation

mkazlauskas commented Nov 13, 2022 • edited

Context

Proposed Solution

Important Changes Introduced

Follow-up work

rhyslbw left a comment

Choose a reason for hiding this comment

mkazlauskas commented Nov 14, 2022

mirceahasegan left a comment • edited

Choose a reason for hiding this comment

rhyslbw left a comment

Choose a reason for hiding this comment

rhyslbw left a comment

Choose a reason for hiding this comment

mirceahasegan left a comment

Choose a reason for hiding this comment

mkazlauskas commented Nov 13, 2022 •

edited

mirceahasegan left a comment •

edited