6th Testnet Deployment

[tegefaulkes]

Specification

2023 August 31st target for 6th testnet deployment.

It's time to bring together all the work done for PK since last year's 5th Testnet Deployment in October 2022. This is 11 month's worth of work!

• QUIC replacing uTP for agent service
• JSON RPC replacing GRPC
• CLI extracted out of Polykey to Polykey-CLI
• MDNS for local discovery
• NodeGraph segregation between local and remote node addresses
• IPv6 Support (Agent and Client service)
• Websockets replacing GRPC for client service
• Hole punching via QUIC dialing and server to client hole punching
• Signalling via seed nodes using fire and forget procedures
• Switching to polykey.com as the main domain instead of polykey.io (polykey.io redirects to polykey.com)
• Adapting our Polykey-Infrastructure to work the new Polykey-CLI container images
• CLI bundling optimising for pkg
• Timeouts, contexts, abort signals, decorators
• ESM migration - cannot do this atm due Migrate to ESM (and get dynamic await) TypeScript-Demo-Lib#32 (comment)
• Switching to Node v18.15 or higher
• Vault operations should all be working (including uploads to the agent)
• Test memory leaks as per Investigate memory leaks in code and tests #542 - especially during testing, but also factoring out the complicated parts of PK should also help in identifying these issues

Additional context

• Testnet Node Deployment (testnet.polykey.io) #194 (comment) - 1st to 4th
• Testnet Node Deployment (testnet.polykey.io) #194 (comment) - 5th testnet deployment

Tasks

1. See all epic listed tasks

[CMCDragonkai]

Take the spec from the whiteboard as I may be rubbing it out soon.

[CMCDragonkai]

This is in the wrong repo. This should be in Polykey not Polykey-Docs.

[CMCDragonkai]

I just added MatrixAI/Polykey-CLI#71 to this epic. That was left over from the previous testnet deployment. We will need to address (and revisit) all of those problems here too.

[CMCDragonkai]

Additional context connected to commentary about the 1st to 5th testnet deployment.

[CMCDragonkai]

Remember to install the zenhub extension to see the epic related information @amydevs @tegefaulkes.

[amydevs]

Remember to install the zenhub extension to see the epic related information @amydevs @tegefaulkes.

have it installed already

[CMCDragonkai]

@tegefaulkes I can still see a bunch of TODOs left in the codebase in Polykey:

[nix-shell:~/Projects/Polykey/src]$ ag 'TODO'
vaults/VaultOps.ts
15:// TODO: tests
52:// TODO: tests
99:// TODO: tests
138:// TODO: tests
194:// TODO: tests
250:// TODO: tests

nodes/NodeConnectionManager.ts
573:      // TODO: This needs to throw if none were established.
597:    // TODO: do we bother with a concurrency limit for now? It's simple to use a semaphore.
598:    // TODO: if all connections fail then this needs to throw. Or does it? Do we just report the allSettled result?
679:    // TODO: A connection can fail while awaiting lock. We should abort early in this case.

nodes/errors.ts
3:// TODO: Some errors may need to be removed here, TBD in stage 2 agent migration

nodes/NodeManager.ts
514:  // TODO: make cancellable

schema/Schema.ts
204:      // TODO: to be implemented

discovery/Discovery.ts
315:    // TODO: for now, the chain data is treated as a 'disjoint' set of
325:    // TODO: there is no deterministic iteration order of keys in a record.

git/utils.ts
556:    // TODO: In the future, we may want to have an API where we return a

utils/utils.ts
97: * TODO: When TS 4.9.x is released, change this to `o is object`.
432:// TODO: remove this, quick hack to allow errors to jump the network

agent/handlers/vaultsGitInfoGet.ts
29:    _ctx: ContextTimed, // TODO: use

These should all be resolved or setup as issues.

[tegefaulkes]

Nodes and agent domains are relevant to the migration work still, the others are older than the migration work.

There may be some FIXMEs as well.

But yes, most of theses need to be addressed within this epic.

[CMCDragonkai]

I removed MatrixAI/js-ws#3 and #495 from this epic as we won't have time.

However those 2 will need to be scheduled for 7th testnet deployment. That's when WS and RPC would be fully factored out and utilised by all parts of PK and PKE.

[CMCDragonkai]

The nodes domain will need to be refactored in accordance with all the lessons from MatrixAI/js-quic#53.

[CMCDragonkai]

New plan. This all works concurrently too.

1. @tegefaulkes and myself finish js-quic to integrate into PK: Events Refactoring (integrating js-events) js-quic#53
   • Benching
   • Testing
   • Performance
   • Fast shutdowns
2. That leads to finalising up the configuration consolidation Consolidating Configuration Entropy and Nodes and Client Services Refactoring #552.
3. Simultaneously that leads a events refactoring and integration that I think is already happening in Consolidating Configuration Entropy and Nodes and Client Services Refactoring #552.
4. Simultaneously this leads to some refactoring of the nodes domain to be clearer too.
5. Finally getting back to NAT Signalling (Hole Punching) should be Fire & Forget, Coalesced, and Secured with Signatures #555 where we need to decide how asynchronous background work is done, and I'm still in favour of putting it into the tasks domain because I want all async background jobs to be managed there no matter how big or small, and we can later focus optimisation work around the tasks domain to separately deal with in-memory tasks or persistent tasks (it could be an optional flag to determine the difference). But 100% no floating promises.
6. @amydevs finishes up js-ws (Extract src/websockets from Polykey to js-ws js-ws#5) in accordance with event structure and TLS verification structure similar to js-quic. Atm, there will be no browser support yet, as that's only necessary for testnet 7. So both client and server are just node-only.
   • Important to ensure that the TLS system is the same as js-quic. Make sure you can supply client certificates and server certificates optionally. Note that unlike js-quic, server certs is not mandatory. Make sure the verifyPeer option works the same way as js-quic though!!
7. @amydevs helps @addievo finish off js-rpc (Factor out RPC from Polykey js-rpc#8). There are still a few issues to verify and confirm and test that it works properly.
   • Whitespace removal - possible change required in jsonstreamparser. See Message parser should be able to support arbitrary whitespace such as '\n', '\t', '\r', and ' ' within and between messages juanjoDiaz/streamparser-json#35 - I suspect this requires going into the source code and modifying the tokeniser so it tokenises a whitespace token based on an array of candidate characters. Extend the separator option to have an array of potential strings instead of a single string like ['\n', '\t', '\r']. Alternatively, allow separator to instead be a regular expression. Remember that the tokeniser should be capable of greedily eating up all the separators into 1 token.
   • Error filtering - this needs to be driven by the user. The user must be able to supply toError and fromError functions that not only performs the serialisation/deserialisation work, but also is capable of doing the error data filtering, as it is important that the certain information is blanked out or deleted before sending over to untrusted peers. This cannot be hardcoded into js-rpc, it has to be something that the user must create and supply. I suggest 2 parameters: toError/fromError and corresponding replacer and reviver (see JSON.parse and JSON.stringify MDN docs) in order to signal to the user to supply both... Consider optionality and also conformance with how PK currently does things.
   • Concurrent timeouts - deadlines and timeouts applied on both the caller and handler needs to be considered, and tests for both behaviours must be created that demonstrate what happens when both timeout at the same time. Note that if the caller times out first, it would have its own exception and ignore the server's error. And if the server times out first then the call would timeout. Document exactly how to ensure that the server either has a hard timeout that represents the "maximum" time the handler is allowed to run, and how the client's timeout becomes the minimum time. If the handler specifies a greater time, then the client's timeout will timeout first, then it should abort on the server handler. If the server handler specifies a lesser time, then the server will timeout first and abort the client. Both client and server can have infinite timeouts by default.
   • Error codes - JSON RPC does have error codes, these should use JSON RPC's code table first, and then specify a code for "application level errors"
     
     Notice that there are predesigned error codes first. And then if it's an application error, where an exception has occurred. We would optionally take the code property of the exception object if it exists. Consider how this would work with toError and fromError. More details in RPC Raw Stream Errors (distinguish between transport errors and application errors) js-rpc#3
8. Now after both are ready @amydevs and @addievo work together on a single PR to Polykey to integrate both js-ws and js-rpc into PK. This should coincide with Consolidating ClientService and js-ws and js-rpc integration #560 with some consolidation of both the client domain and agent and/or nodes domain. Which means moving client related functionality into the same director.
9. At this point @addievo can split off to work on PKE focusing on integration of js-ws and js-rpc and focusing on getting the pulumi stack dynamic and setting up for testnet 7.
10. @amydevs then instead you focus on getting js-mdns integrated into PK, refactor according to the events model, and then get it integrated into our nodes domain. Add additional tests to enable multiple PK nodes on the same computer/network to discover each other, focus on nodegraph expansion MDNS integration and NodeGraph structure expansion #537. @tegefaulkes can aid the specification and review of this work.
11. PK CLI packaging - @tegefaulkes and myself will be focusing on robustness and stability and packaging - that means Remove @matrixai/quic and @matrixai/id and separate error tree Polykey-CLI#20 and ci: merge staging to master Polykey-CLI#4.
12. Deployment to the new testnet. Assuming @addievo has finished the pulumi code in PKE, it could be possible to make use of the pulumi stack on bootup of PKE for the testnet 6, thus removing the existing AWS sdk script based infrastructure.

Critical path is 1, 2, 3, 4, 5, 11.

[CMCDragonkai]

Regarding standardisation of TLS between QUIC and WS:

Clients always request a server certificate.

Server can ask the client for a certificate.

Standardise between QUIC and WS TLS behaviour and configuration.

In Node TLS we have 3 options:

• requestCert - server responsds with an ask for a client cert
• rejectUnauthorized - true/false
• checkServerIdentity - if undefined, runs after rejectUnauthorized (unless rejectUnauthorized is false), otherwise also runs if this is set

QUIC (both sides):
  requestCert: true
  verifyPeer: true/false
  verifyCallback: undefined
    verifyPeer: verifyPeer
  verifyCallback: () => {}
    verifyPeer: false
    custom TLS

WS (both sides):
  requestCert: true
  verifyPeer: true
    rejectUnauthorized: true
    checkServerIdentity: undefined
  verifyPeer: false
    rejectUnauthorized: false
    checkServerIdentity: undefined
    IGNORE verifyCallback
  verifyCallback: undefined
    verifyPeer: verifyPeer
    checkServerIdentity: undefined
  verifyCallback: () => {}
    ONLY works if verifyPeer true
    rejectUnauthorized: false
    checkServerIdentity: undefined

Therefore checkServerIdentity is always undefined.

To standardise the TLSVerifyCallback.

We are going to change our types:

type TLSVerifyCallback = (
  certs: Array<Uint8Array>,
  ca: Array<Uint8Array>
) => PromiseLike<void>;

Both are arrays of DER buffers.

This means the derToPEM isn't used for certs.

But this.config.ca does require to be converted to DER buffers, it can be done ONCE at the beginning/construction of the QUICConnection at the WebSocketConnection (because the HTTPS server may have a new config).

The this.config is shared between all the QUICConnection. If the TLS was updated. Then each QUICConnection is using the newly updated config.

However when new QUICConnection occurs, we create a new quiche config based off the shared QUICConfig:

    const quicheConfig = buildQuicheConfig(config);

This means if you ask for this.config.ca you always get the latest this.config.ca. You don't get the original ca that was plugged into the QUICConnection.

Remember QUICServer.updateConfig, I may update the verifyCallback as well. Existing connections has to re-use the old verifyCallback as well. Because that could still be valid for those old connections.

Therefore new QUICConnection should do this:

      config: {...this.config},

[CMCDragonkai]

Testnet 6 could test ipv6 support too to cap off #400.

[tegefaulkes]

Two new issues at

• nodes status command should include connection and graph stats Polykey-CLI#36
• Fix and re-enable any disabled tests Polykey-CLI#37

[CMCDragonkai]

@tegefaulkes make sure to focus on getting #555 in as well as well as cleaning up any remaining issues in the review/qa column in this epic.

[CMCDragonkai]

Still waiting on #555 and #584 @tegefaulkes and @amydevs. If there are platform specific problems, please create issues separately, and so we can focus on just Linux-based Testnet 6.

[CMCDragonkai]

Connecting to the testnet node 3.106.15.126 shows that it currently has multiple certificates. This shouldn't be the case because we have not executed any sort of key renewal or rotation. @tegefaulkes please investigate why it has more than 1 certificate? Is it possible that it is using some pre-existing state? But then it shouldn't add certificates that isn't part of its root key chain.

[nix-shell:~/Projects/Polykey-CLI]$ ./dist/polykey.js agent status --client-host 3.106.15.126 --client-port 1315 --node-id v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70
✔ Please enter the password … ********************
status	"LIVE"
pid	1
nodeId	"v7v9ptvcdbdf8p4upok3prpmu3938ns8v4g45dib7sm5hqvvehv70"
clientHost	"0.0.0.0"
clientPort	1315
agentHost	"0.0.0.0"
agentPort	1314
publicKeyJWK	{"alg":"EdDSA","kty":"OKP","crv":"Ed25519","x":"P9Oe_Y1bXoyT2cUHnebeGkaL8R8kCFbJZ-WLHX_uj84","ext":true,"key_ops":["verify"]}
certChainPEM	"-----BEGIN CERTIFICATE-----\nMIIC0jCCAoSgAwIBAgIQBlMVYk8ecAC7Ni7+Rza2YjAFBgMrZXAwQDE+MDwGA1UE\nAxM1djd2OXB0dmNkYmRmOHA0dXBvazNwcnBtdTM5Mzhuczh2NGc0NW
RpYjdzbTVo\ncXZ2ZWh2NzAwHhcNMjMxMDE5MTYxNTMyWhcNMjQxMDE4MTYxNTMyWjBAMT4wPAYD\nVQQDEzV2N3Y5cHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3Nt\nNWhxdnZlaHY3MDAqMAUGAytlcAMhAD/Tnv2NW16Mk9nFB53m3hpGi/EfJAhWyWfl\nix1/7o/Oo4IBkjCCAY4wDAYDVR0TBAUwAwEB/zALBgNVHQ8EBAMCAf4wRQYDVR0l\nBD4wPAYIKwYBBQUHAwEGCCsGAQUFBwMCBggrBgEFBQcDAwYIKwYBBQUHAwQGCCsG\nAQUFBwMIBggrBgEFBQcDCTCBlgYDVR0RBIGOMIGLgjV2N3Y5cHR2Y2RiZGY4cDR1\ncG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3MIY6cGs6Ly92N3Y5\ncHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3\nMIcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAdBgNVHQ4EFgQUc1eACDFkBPX3ayGV\nWLkeCbK9yC0wHwYLKwYBBAGDvk8CAgEEEBYOMS4yLjEtYWxwaGEuMTQwUQYLKwYB\nBAGDvk8CAgIEQgRAq7lP0qSpl0gGMw9toyMQ/BAkBqJC0ZmvYcvB51AV2Qc5aUZB\nr3HNcUTStm/SVDiymTVEj/hn1STUQuVg+ZLFDzAFBgMrZXADQQDyV7R7HECFhNBF\nhVEvWYS0bnPQ8ULT9FvlIhvGNv9a9D0GxkepojM+EsoNE5Ogg7SvGaG07RNkpvx6\nIVD7Nr0N\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIC0jCCAoSgAwIBAgIQBlMNtLJ+cAC0rE7qLgAxbjAFBgMrZXAwQDE+MDwGA1UE\nAxM1djd2OXB0dmNkYmRmOHA0dXBvazNwcnBtdTM5Mzhuczh2NGc0NWRpYjdzbTVo\ncXZ2ZWh2NzAwHhcNMjMxMDE5MDczMTIzWhcNMjQxMDE4MDczMTIzWjBAMT4wPAYD\nVQQDEzV2N3Y5cHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3Nt\nNWhxdnZlaHY3MDAqMAUGAytlcAMhAD/Tnv2NW16Mk9nFB53m3hpGi/EfJAhWyWfl\nix1/7o/Oo4IBkjCCAY4wDAYDVR0TBAUwAwEB/zALBgNVHQ8EBAMCAf4wRQYDVR0l\nBD4wPAYIKwYBBQUHAwEGCCsGAQUFBwMCBggrBgEFBQcDAwYIKwYBBQUHAwQGCCsG\nAQUFBwMIBggrBgEFBQcDCTCBlgYDVR0RBIGOMIGLgjV2N3Y5cHR2Y2RiZGY4cDR1\ncG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3MIY6cGs6Ly92N3Y5\ncHR2Y2RiZGY4cDR1cG9rM3BycG11MzkzOG5zOHY0ZzQ1ZGliN3NtNWhxdnZlaHY3\nMIcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAdBgNVHQ4EFgQUc1eACDFkBPX3ayGV\nWLkeCbK9yC0wHwYLKwYBBAGDvk8CAgEEEBYOMS4yLjEtYWxwaGEuMTQwUQYLKwYB\nBAGDvk8CAgIEQgRAfoWjgh9OAgYxQ3cLdMu4eI3M7nY/mXf/YwZsImrJ+e5T8H5Z\nAZj/ivoR+jmbziEEP5VP26szsFBqNM4Hf2FXBjAFBgMrZXADQQDeQsqAB8GU64Tj\nbt7uxtk9Iqk8g8SwSlsSpJY8Crf6zqQ2bBSX/h2eer2h5WbGXhtFeHA/urEg9lco\nwHRKFH8L\n-----END CERTIFICATE-----\n"

We could take the PEM, put into a file, and use step certificate inspect to find out what these certificates are as a sanity check.

[CMCDragonkai]

There's evidence of a memory leak in the testnet nodes:

Private Zenhub Image

To deal with this, we need more observability. So that means we need the commands list out all connections. We need commands what tasks are still running or about to be run. We need to be open up the PK internal state via the commands.

• Connections
• Streams
• All other resources that you can find out

We do need a consistent tool to understand what the allocations.

Perhaps we could do this over the client service. Live debugging. Could be done dynamically instead of having to have it run on the beginning. This means not using the --inspect flag at the beginning, but instead enabling on the fly through the client service. Potentially proxying any data over the websocket, or not.

https://nodejs.org/en/docs/guides/debugging-getting-started#enabling-remote-debugging-scenarios

[CMCDragonkai]

I've moved a few issues out to CLI beta launch instead. We should aim to have all the issues done here today.

Then we have all the bugs and weird things to look at above, but they can be turned into their own issues.

If this epic is done, we can run the CLI demo again, this time with 2 nodes on 2 different networks that have LAN.

I can do one from my place here, and another node has to be done from the office. Since switching to ABB, office is not a CGNAT location anymore. So hole punching should work from office to the other site.

[CMCDragonkai]

@addievo can you start updating the PK CLI to use the new Polykey - signature update for all await PolykeyClient.createPolykeyClient.

[tegefaulkes]

There's evidence of a memory leak in the testnet nodes:

To deal with this, we need more observability. So that means we need the commands list out all connections. We need commands what tasks are still running or about to be run. We need to be open up the PK internal state via the commands.

• Connections
• Streams
• All other resources that you can find out

We do need a consistent tool to understand what the allocations.

Perhaps we could do this over the client service. Live debugging. Could be done dynamically instead of having to have it run on the beginning. This means not using the --inspect flag at the beginning, but instead enabling on the fly through the client service. Potentially proxying any data over the websocket, or not.

https://nodejs.org/en/docs/guides/debugging-getting-started#enabling-remote-debugging-scenarios

Do I need to create a new issue to track this?

[CMCDragonkai]

#597 is created for tracking the problem of multiple certificates.

[CMCDragonkai]

#598 has been created to track the memory leak issue, as well as potential future ways of understanding and remote debugging.

[amydevs]

MatrixAI/js-quic#72
https://chat.openai.com/share/0f4f29c8-4552-4a30-b089-d116a76dc7ab

The last address, (a link-local ipv6 address) will not work as of now, as sending to it will require a scope-id, in this case being wlp0s20f3. The finalised address with scope-id will look like fe80::1cc7:8829:3d67:240d%wlp0s20f3. The Rust std::net library will error if the address starts with fe80 and does not contain a scope %wlp0s20f3 and even if the parser succeeded, the underlying socket code would fail sending to an IPv6 link-local address without a scope. However, js-quic currently hangs when an IPv6 address contains a scope-id. This means that we cannot connect to nodes over link-local interfaces for now. Which would only really prevent node connection where a router does not exist to assign Network Interfaces globally routable IPv6 addresses such as the second last one shown below.

2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 50:28:4a:a8:0f:75 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.86/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp0s20f3
       valid_lft 53829sec preferred_lft 53829sec
    inet6 2600:1700:70:7780::22/128 scope global dynamic noprefixroute 
       valid_lft 3414sec preferred_lft 3414sec
    inet6 2600:1700:70:7780:ebb8:d731:bf5b:7ac1/64 scope global temporary dynamic 
       valid_lft 3585sec preferred_lft 3585sec
    inet6 2600:1700:70:7780:f37d:6baf:ab6c:ad24/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3585sec preferred_lft 3585sec
    inet6 fe80::1cc7:8829:3d67:240d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[CMCDragonkai]

I created #599 to track a testnet.polykey.com status page.

[CMCDragonkai]

The last major feature to integrate is #584.

@tegefaulkes please help review it.

That's it!

I'm moving the rest into MatrixAI/Polykey-CLI#40. And then it's going to be about bug fixing, stability, and CLI UX.

[CMCDragonkai]

We are going to do another demo, and get all the CLI output to the video team.

[tegefaulkes]

We just got two nodes to connect to each other through nats with the testnet. Celebrate!

We've also found some issues to address.

1. Nodes Get All is throwing an error
2. We are getting a weird output format for nodes connections what is the 0 and what is this NA for?
3. I think nodes connections is better suited for dictionary format - list format or table format (can you check which ones done robustly)
4. Sometimes the connections is being reformatted as ipv4 mapped ipv6 addresses, how is this possible why is done?
5. The output changes sometimes, to just join the NA after the ip address, so no port is available
6. Summary - the nodes connections command is just very non-deterministic
7. Secret leak is occurring on agent service, there should be no authorization metadata see --format=json
8. What's usageCount

These things need to be looked into and new issues created if needed.

[tegefaulkes]

There is too much information in the agent status command. the public key JWK and certChainPEM can be removed. We can get this through other commands.

[tegefaulkes]

new issue at MatrixAI/Polykey-CLI#44

I'll make a new issue about hole punching working only one way right now.

[tegefaulkes]

new issue for hole punching at #605

[CMCDragonkai]

All major feature issues is done. I'm closing this. Bugs and stability will go into the CLI beta epic MatrixAI/Polykey-CLI#40