FFI communication between native and meta chain

[wafflespeanut]

What would you like to be added:

As an alternative to JSON RPC communication (#80), I propose invoking functions through FFI. Metachain (MC) can be compiled as a staticlib, which then gets linked to defid (NC). The PoC for this approach exists in libmc branch in metachain and ain repository. CLI args from NC will be passed to MC, which will be bootstrapped and then defid will follow.

Why is this needed:

This way, we can eliminate the security concerns for running MC as an independent process and communicating through RPC (#45), which also reduces TCP overhead and avoids us packaging and running two binaries which will most probably exist in the same machine. #94 wouldn't have any adverse impact on this decision as the source of the bottleneck will still be in defid. It should be noted that MC and NC will not have any knowledge of what the other is doing and neither will one manage the other. However, when MC fails and exits, it will shutdown defid as well (at this point).

As for e2e testing, we can still expose RPC from MC, because essentially they'll be the same FFI calls.

cc @prasannavl @fuxingloh @DieHard073055

[defichain-bot]

@wafflespeanut: Thanks for opening an issue, it is currently awaiting triage.

The triage/accepted label can be added by foundation members by writing /triage accepted in a comment.

Details

I am a bot created to help the DeFiCh developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[defichain-bot]

@wafflespeanut: There are no 'area' labels on this issue. Adding an appropriate label will greatly expedite the process for us. You can add as many area as you see fit. If you are unsure what to do you can ignore this!

You can add area labels by leaving a /area comment.

Details

I am a bot created to help the DeFiCh developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[fuxingloh]

As an alternative to JSON RPC communication (#80), I propose invoking functions through FFI. Metachain (MC) can be compiled as a staticlib, which then gets linked to defid (NC). The PoC for this approach exists in libmc branch in ain repository. CLI args from NC will be passed to MC, which will be bootstrapped and then defid will follow.

I think it's sound given if both options are provided with a shared Interfacing for the sake of testing with the ability to force defid to switch between prepackaged binary or via a provided RPC URL --metachain_url=.... #97 is also a very important related issue where in the future or now, we might not want to "force" all defid clients to be mining for NC and MC, given MC has different performance and storage requirements for masternode operators.

This way, we can eliminate the security concerns for running MC as an independent process and communicating through RPC (#45),

Although #45 is important, it isn't sane for masternode operators to operate on a host machine they can't trust. It's more of an additional security good practice IMO, as I don't think they should not be operating their masternode over the network.

which also reduces TCP overhead and avoids us packaging and running two binaries which will most probably exist in the same machine. #94 wouldn't have any adverse impact on this decision as the source of the bottleneck will still be in defid.

From the current literature on our threading model, I think #94 requires a renewal of our existing threading model for MC integration rather than the bottleneck of integration.

[wafflespeanut]

I think it's sound given if both options are provided with a shared Interfacing for the sake of testing with the ability to force defid to switch between prepackaged binary or via a provided RPC URL --metachain_url=....

While I'm okay with supporting RPC for the purpose of testing, the whole aim of using FFI is to prepackage the binaries and to not worry about having RPC calls in production or needing to secure them, but if we're planning to have it in the near future anyway, then I don't think the FFI way serves any purpose. @prasannavl Need your opinion here.

#97 is also a very important related issue where in the future or now, we might not want to "force" all defid clients to be mining for NC and MC, given MC has different performance and storage requirements for masternode operators.

Is the performance and storage requirements the only constraint for not being able to run MC alongside NC?

Although #45 is important, it isn't sane for masternode operators to operate on a host machine they can't trust.

This is not specific to MC though, right? Wouldn't it apply to NC as well?

[fuxingloh]

While I'm okay with supporting RPC for the purpose of testing, the whole aim of using FFI is to prepackage the binaries and to not worry about having RPC calls in production or needing to secure them, but if we're planning to have it in the near future anyway, then I don't think the FFI way serves any purpose. @prasannavl Need your opinion here.

Yup, DMC is the first part of a series of changes to modulize the DeFiChain blockchain into several components that were initially proposed in DFIP 2111-B. DMC is the first of many execution planes we want to integrate into a shared network plane. The goal is to modularize the blockchain to make upgrading easier. It was explained in a video on a generative approach to adding more modularized components with a shared network plane.

Is the performance and storage requirements the only constraint for not being able to run MC alongside NC?

Part of it. With DMC embedded, the "server profile" and server maintenance's operational flow must be changed if we force defid to be prepackaged. Several operators such as exchanges, individuals, and node operators would have to reconfigure their setup to fit the runtime requirements running defid. It's not simply a plug-and-play since these must be pre-communicated, tested, and operationalized.

This is not specific to MC though, right? Wouldn't it apply to NC as well?

Yup, it would; it's not specific to MC or NC, just running servers or validators in general. IMO, additional security cushions more for the uninitiated server maintainers.

[wafflespeanut]

Btw, @mambisi has brought it to my attention going through the FFI route will not be as easy as it sounds because we won't get the context from substrate outside of wasm environment. Until we have a clear direction there, this is how we've currently decided to go:

graph

subgraph DeFiCh/ain
nc[Native chain consensus]
nb[Native chain bootstrap]
nb --> nc
nb --" FFI "--> mb
nc --" FFI "--> nrc
nb --" FFI "--> lock

subgraph "DeFiCh/libain-rs"
pt["Protobuf spec"]
ngc["gRPC client"]
nrc["RPC client"]

pt --> ngc
pt --> nrc
end

subgraph DeFiCh/metachain
mb[Metachain bootstrap]
mc[Metachain consensus]
mprc[RPC server]
lock[Random number agreement]

mprc --> lock
lock --> mprc
mb --> mc
mprc --> mc
nrc --" JSON RPC "--> mprc
end

end

• Metachain build will also emit a static library in addition to the executable. This gets linked to defid. When defid starts, it has the option to boot up metachain (and arguments following a certain flag will be passed to it). When metachain terminates, it can issue a shutdown request to defid.
• When metachain is under defid's management, both will agree upon a large random number which will then be set alongside the RPC arguments. This way, any incoming request (to those consensus RPC endpoints) not coming from defid will be dropped.
• The consensus mechanism will still be through RPC (addressed in JSON-RPC communication between Native Chain and Meta Chain #80). FFI will be used only to boot up metachain and agree upon the random number. E2E tests will work normally on both metachain side and ain side because there won't be any breaking changes.

[fuxingloh]

Btw, @mambisi has brought it to my attention going through the FFI route will not be as easy as it sounds because we won't get the context from substrate outside of wasm environment. Until we have a clear direction there, this is how we've currently decided to go:

Yup sounds good, I'm afraid of that as well.

• Metachain build will also emit a static library in addition to the executable. This gets linked to defid. When defid starts, it has the option to boot up metachain (and arguments following a certain flag will be passed to it). When metachain terminates, it can issue a shutdown request to defid.

Yup, this is great; it's important that it's optional. #97 (comment)

Does this also mean instead of starting up MetaChain, we could also provide an URL with a port number?

• When metachain is under defid's management, both will agree upon a large random number which will then be set alongside the RPC arguments. This way, any incoming request (to those consensus RPC endpoints) not coming from defid will be dropped.

Could we use JWT instead if it's not too troublesome? Minor, I'm also very comfortable with this design for now, but it might require changes. Let's see, it's not important now.

• The consensus mechanism will still be through RPC (addressed in JSON-RPC communication between Native Chain and Meta Chain #80). FFI will be used only to boot up metachain and agree upon the random number. E2E tests will work normally on both metachain side and ain side because there won't be any breaking changes.

Sounds good.

[wafflespeanut]

Does this also mean instead of starting up MetaChain, we could also provide an URL with a port number?

I didn't have this in mind, but it shouldn't be hard to implement with the current design.

Could we use JWT instead if it's not too troublesome? Minor, I'm also very comfortable with this design for now, but it might require changes. Let's see, it's not important now.

Agreed, let's get both defid and metachain running in a single node. I'm sure we can improve the mechanism later on.