IPC communication between agent and client (CLI/GUI)

[robert-cronin]

The next problem to figure out is how the client and agent, it looks like there are a couple of ways to achieve this:

• Via gRPC
• Via NodeJS net module which has support for windows and unix

Discovery between the agent and the client also needs to be figured out, here are some ideas:

• Rely on a shared path (this shared path must be user-specific) like /run/user/1000/polykey/socket (this only exists on Linux, find equivalent on Mac/Windows).
• Rely on a unix domain socket which is also on a shared path that is user-specific. This relies on a shared path existing.

Using UDS is really about getting access to the OS security capabilities to securing the comms. The nodejs TLS module can also do UDS as its an abstraction over the net module, you just have to specify the path in tls.connect. I'm just not sure about the server and tls.

[robert-cronin]

It turns out node grpc doe not support UDS: grpc/grpc-node#258 so I am going with the nodejs net module and a proto file/protobufjs for the message format

[CMCDragonkai]

Does grpc-node generate marshaling code given proto files?

[CMCDragonkai]

@robert-cronin can you update this issue indicating that UDS does work, and how you achieved it.

[CMCDragonkai]

Clarification with grpc-node https://github.com/grpc/grpc-node/blob/master/PACKAGE-COMPARISON.md it seems to mean that it's not pure JS. Which means potential issues when deploying this on nativescript.

[CMCDragonkai]

Oh I see, grpc-node actually distributes 2 packages. One of them requires C and the other is pure JS. The C one might be able to be compiled to the nativescript platforms. But the pure JS one would be safer, but is missing only a few features. Do we need those and which one are you using?

[robert-cronin]

Does grpc-node generate marshaling code given proto files?

protobufjs has the ability to generate JavaScript code that encodes and decodes protocol buffers. It also generates typescript definition files. These are the commands I am using for a proto folder defined at the root directory:

# Creates the JavaScript file
pbjs -t static-module -w commonjs -o proto/js/compiled.js proto/*.proto

# Generates the corresponding type definition files
pbts -o proto/js/compiled.d.ts proto/js/compiled.js

[robert-cronin]

@robert-cronin can you update this issue indicating that UDS does work, and how you achieved it.

Yeah so it was actually quite simple. The nodejs net module allows one to specify a file path in net.createServer(...).listen(socketFilePath) and it takes care of creating the socket file. Closing the server also removes the generated socket file.

[robert-cronin]

Oh I see, grpc-node actually distributes 2 packages. One of them requires C and the other is pure JS. The C one might be able to be compiled to the nativescript platforms. But the pure JS one would be safer, but is missing only a few features. Do we need those and which one are you using?

Yeah I was originally using grpc-js but discovered there was no way to synchronously start the server and let it dynamically choose and available port. In grpc you can do it by const port = server.bind('0.0.0.0:0') but in grpc-js the bind function doesn't return anything. The only way to do this is const port = await server.bindAsync('0.0.0.0:0') and so its impossible to follow RAII. I think the only way we will be able to use grpc-js is if we create stop and start functions for the server so they can be async. I don't mind this because there might be plenty of reasons why the user wants to control the starting and stopping of the secure server.

[CMCDragonkai]

Make sure this is documented or auto generated by webpack?

…

On 10 July 2020 13:05:33 GMT+10:00, Robbie Cronin ***@***.***> wrote: > Does grpc-node generate marshaling code given proto files? `protobufjs` has the ability to generate JavaScript code that encodes and decodes protocol buffers. It also generates typescript definition files. These are the commands I am using for a `proto` folder defined at the root directory: ``` # Creates the JavaScript file pbjs -t static-module -w commonjs -o proto/js/compiled.js proto/*.proto # Generates the corresponding type definition files pbts -o proto/js/compiled.d.ts proto/js/compiled.js ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #50 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[robert-cronin]

I am storing the proto files in a proto directory at the project root level and the script to generate it is stored in scripts/compile_proto.sh and is also an npm script in package.json. I will put something into the readme about this.

[robert-cronin]

Just for future reference, it turns out the only way to import grpc-js is import * as grpc from'@grpc/grpc-js

[CMCDragonkai]

We don't need to synchronously start and stop the server. You can still do RAII with async await. Just have a async main function. See loja-api's server entry point.

…

On 10 July 2020 13:14:17 GMT+10:00, Robbie Cronin ***@***.***> wrote: > Oh I see, grpc-node actually distributes 2 packages. One of them requires C and the other is pure JS. The C one might be able to be compiled to the nativescript platforms. But the pure JS one would be safer, but is missing only a few features. Do we need those and which one are you using? Yeah I was originally using `grpc-js` but discovered there was no way to synchronously start the server and let it dynamically choose and available port. In `grpc` you can do it by `const port = server.bind('0.0.0.0:0')` but in `grpc-js` the `bind` function doesn't return anything. The only way to do this is `const port = await server.bindAsync('0.0.0.0:0')` and so its impossible to follow RAII. I think the only way we will be able to use `grpc-js` is if we create `stop` and `start` functions for the server so they can be async. I don't mind this because there might be plenty of reasons why the user wants to control the starting and stopping of the secure server. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #50 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[robert-cronin]

Not sure what you mean, does loja have a dedicated Server class with a normal constructor and then a main function that acts like an init? I couldn't find anything like this.

If that's the case, I'm not sure what would be the difference to having a serverStart function?

I think there is still a way we can follow RAII, we just start the server asynchronously in the constructor. It's just not guaranteed to be started by the time the class is finished initialising.

[CMCDragonkai]

Like this (index.ts):

import fs from 'fs';
import { Pool } from 'pg';
import Container from '@loja/server/bootstrap';
import router from '@loja/server/router';

(async () => {
  let db;
  try {
    db = new Pool();
    console.log(`Connected to database`);
    const container = Container(db);
    fs.mkdirSync(container['config']['LOJA_UPLOAD_DIR'], { recursive: true });
    console.log(`Upload directory set to ${container['config']['LOJA_UPLOAD_DIR']}`);
    const app = router(container);
    await app.listen(container['config']['LOJA_PORT'], container['config']['LOJA_HOST']);
    console.log(
      `Loja listening on port ${container['config']['LOJA_HOST']}:${container['config']['LOJA_PORT']}`
    );
  } catch (e) {
    if (db != null) db.end();
  }
})();

Notice the try and catch.

Also the main function is basically an IIFE.

[robert-cronin]

oh okay, I did notice this file, but thought it wasn't relevant to PeerManager since it is supposed to be initialised as a class instance and not just a function call. I think especially from the point of view of Polykey, the sub components should be composable and not just functions to call.

[CMCDragonkai]

Oh right, so that's an issue. Some classes are not well "encapsulated" since they rely on "session" side-effects. For these things I usually abstract them out with dependency injection so that they are forced to be constructed at the top-level context. Thus that main function above, you see I am forced to create the database pool object there.

[CMCDragonkai]

So if we want to continue using gRPC between client and agent and not just agent to agent, then we would like to have gRPC support UDS because UDS inherits OS permissions whereas localhost can be sniffed by any program on the machine even executed by other users. However at the moment the node implementation of gRPC does not support UDS. It is possible to support if these 2 issues were to be resolved:

• Unable to connect to Unix socket using node-grpc client grpc/grpc-node#258
• Support libuv pipes for custom socket on platforms with UDS grpc/grpc#18435 - this got closed

Therefore the only solution is to use mTLS between agent and client to secure communications locally.

This can allow clients to exist on a different system to call into a PK agent. So it does have its own advantages.