Unable to connect to Unix socket using node-grpc client

[gcarling]

I am trying to communicate with a server that is listening on a unix socket - if I use TCP I have no issues, but after much effort have been unable to get it working with the unix socket. Code is

    const BIND_SOCKET = '/tmp/collector.sock';
    const BIND_TCP = ':9832'; // works fine if I use this for client + server
    const PROTO_PATH = './Collector.proto';
    const collector = grpc.load({ root: __dirname, file: PROTO_PATH }).collectorv1;

    this.client = new collector.Collector(BIND_SOCKET, grpc.credentials.createInsecure());

I have tried many variations on the socket, such as unix:/tmp/collector.sock and many others I have found online. However none of them work and my write call simply hangs forever, regardless of whether the server is active and listening on the port.

I am using grpc version 1.10.1

[murgatroid99]

I am pretty sure the node library does not currently support unix domain sockets.

[gcarling]

When I was debugging this I got up to here - line 485 of client.js

this.$channel = new grpc.Channel(address, credentials, options);

in this context, grpc has been required like this -

var grpc = require('./grpc_extension');

My understanding was there was some binary outside of JS that was being used to make the actual connection. This leads me to believe that any behavior supported by that library should be supported here? Maybe I am not fully understanding, but my point is that it does not appear as though the node client is actually the one making the connection. I could be wrong.

[gcarling]

@murgatroid99 Can you confirm for me that node-grpc uses the C++ library under the hood? I saw this issue (grpc/grpc#6504) which confirms for me that the C++ version supports unix sockets. Not sure if you are using this library or a different C++ binary as the base for connection, in any event hoping you can give me some info on that so I can debug further on my own. This is pretty critical to our performance so I am willing to put time into this, even open a PR if necessary.

[nicolasnoble]

So yes, node-grpc uses the grpc-core library under the hood. However, for nodejs integration, it uses the libuv codepath, which doesn't have uds integration. Adding uds to node-grpc would mean changing the name resolver of this file to have the same hack as this other file, and then we'd need to plumb the whole concept of libuv pipes into it.

This isn't trivial, that is.

[gcarling]

OK - thank you very much for the explanation. If at any point supporting unix sockets appears on your roadmap please let me know! I will check this issue once in a while to see if there are any updates. I can submit a formal feature request if that is useful to you.

[nicolasnoble]

I think we can consider this issue to be an actual feature request :-) Also it might be easier for us to implement this using the pure-javascript version of the library.

[jkryl]

Soon there will be anniversary of this ticket :-) I started to work on this feature. Note that if all what you care about is the client side, then one can use grpc-js package (grpc implementation written fully in JS). However grpc server is not implemented in grpc-js, so for that part, the grpc c lib must be fixed. I have submitted a PR for that one: grpc/grpc#18435 .

if that goes well and gets accepted, I will submit a PR for grpc-node as well, which will enable the feature.

[faern]

Ping on this issue. Having UDS support would be much appreciated. For now I will only need the client half and will try the grpc-js package, but it clearly states that it is a beta level release at the time of writing this.

[jkryl]

@faern : if you can leave a comment in grpc/grpc#18435 that you would like this feature implemented, that might raise the priority of the feature in eyes of grpc maintainers. It's not moving anywhere currently. thanks

[faern]

There are zero examples or howtos that I can see for grpc-js. Does anyone have any code snippet on how to use it?

[entropitor]

@faern It seems it uses the same API as the c-based one.

@jkryl It seems grpc-js does have server support now but when I tried switching https://github.com/grpc/grpc-node/blob/master/packages/grpc-js/src/server.ts#L203 to using const options = {path: port} if port is not a number, I'm getting an RST_STREAM with error code PROTOCOL_ERROR when I try to grpcurl to it so ... 😕

[jcantwell-JC]

Using the advice in this thread, I've been trying to get a grpc-js client to connect to a unix domain socket (running from a go server). I'm hoping someone might have some insight on a more correct way to potentially get this working.

When I try to get the helloworld example working with a UDS:

var grpc = require('@grpc/grpc-js');
...
var client = new hello_proto.Greeter('unix:///var/run/test.sock',
                                     grpc.credentials.createInsecure());

I get the following error (obtained by adding some additional logging):

connect error: Error: getaddrinfo ENOTFOUND var
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:60:26) {
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'var'
}

It seems that when supplying an address prefixed with unix:, some path parsing happens in the resolver, but when the http2 library tries to open a network connection, its not using the correct path. I seem to be able to work around this by overriding createConnection in the credentials options:

creds = grpc.credentials.createInsecure();
creds._getConnectionOptions = () => {
  return {
    createConnection: (address, options) => {
      return net.createConnection("/" + address.hostname + address.pathname);
    },
  };
};

var client = new hello_proto.Greeter('unix:///var/run/test.sock', creds, options);

Does anyone have any insight into why getaddrinfo is being called and what config option I may be missing to get this working? Or if the workaround above seems legitimate?

[murgatroid99]

I'm sorry that I haven't addressed this earlier. Unfortunately, the linked PR grpc/grpc#18435 has gotten stale again and has merge conflicts. If @jkryl updates it I will review it as soon as possible.

As for the @grpc/grpc-js issue, it looks like the unix: address handling was simply implemented incorrectly. I will look into fixing that.

Changing the credentials like that is not a great workaround because that is an implementation detail that just coincidentally does what you want.

[jcantwell-JC]

One other thing I'm looking into is supporting windows named pipes. Using the code above, and simply passing the named pipe name into the net.createConnection call appears to work. Obviously I understand this isn't a long term option, but it does indicate that the hurdles are similar, which is nice.

After I see what it takes to properly implement the unix: address, I can possibly look into supporting the windows named pipe as well.

[raksooo]

@jcantwell-JC I've also been experimenting with this and also got it to work by providingcreateConnection with a socket.

I've found that startConnectingInternal in grpc-js/src/subchannel.ts prefixes the address with either "http://" or "https://". In addition it uses http2.connect which in my understanding does not support uds (unless you provide it with a socket from net.createConnection, as in your example). Modifying that function to not add "http://"/"https://" and to open a connection using net.createConnection works as well.

When it comes to supporting names pipes on windows my guess is that the only additional thing that's missing is a resolver (like grpc-js/src/resolver-uds.ts) which resolves and formats "\\.\pipe\" and "//./pipe/" addresses.

@murgatroid99 Where is unix:address handling implemented incorrectly? Or in which commit? Let me know if my ideas sound good and I'll make a PR.

[jkryl]

@jcantwell-JC : thanks for putting together the example on how to use unix socket with grpc-js!

@murgatroid99 : It has been a long time since I submitted the original PR (over a year). It made sense at that time, but now it's maybe better to fix just grpc-js and move on. Especially if it is a simple fix. Is there a ticket to track the issue in grpc-js that prevents unix: addresses from working? I would like to test it once it gets fixed. thanks!

Also, I haven't been following grpc-js closely. When is it expected to become production-ready? I guess that's an important question for anyone who would like to use Unix sockets in his/her product.

[murgatroid99]

I have a speculative fix for this grpc-js problem in #1245.

I'm not exactly sure when we will consider the library production-ready, but the Google Cloud API client libraries have been using it officially for a while.

[murgatroid99]

I've published grpc-js version 0.6.16 with the change I linked to. Please try it out and say if it worked for you.

[entropitor]

@murgatroid99 do I understand correctly this is only support for the client side? Or should the server side also work with UDS now?

[murgatroid99]

What I have so far is only for the client side. The server code has a completely different code path, and it doesn't use any of the code in the linked change.

[jcantwell-JC]

@murgatroid99 I just quickly tested grpc-js version 0.6.16 and it works for my little test. Thanks so much for the quick response! This is awesome!

[raksooo]

@murgatroid99 I just tested both uds and windows named pipes and it works great! Thank you!

[travisghansen]

@jkryl I'm looking into creating multi-arch docker images with my project using grpc-uds package. Compiling it takes quite a while with qemu for aarch64 and armv7...any chance it's possible to create some pre-compiled binaries for a matrix of node/arch versions?

[jkryl]

@travisghansen : Yes, it takes a very long time. It compiles the whole c++ grpc library. The upstream grpc-node uses node-pre-gyp to cache the binaries. However, I did not feel comfortable with it as I don't know how exactly it works and AFAIK I would have to host the binaries somewhere for others to download them. For both reasons I decided to rip it off and use just node-gyp. I don't plan to make any changes to it as I'm looking forward to a long term solution using grpc-js now. Though if someone contributed a PR to https://github.com/jkryl/grpc-node it would be certainly welcomed.

[murgatroid99]

I can't find any code in the grpc-js server that looks at the authority header at all. But the Node http2 server itself may be requiring that clients send one of those headers.

[jkryl]

@entropitor : Problem with authority header is a story of its own. It is a source of various interoperability issues because the spec does not say what authority header should be if used over UDS. Related issue to grpc implementation in golang (contains a link to rust issue as well) here: grpc/grpc-go#2628.

[entropitor]

Okay, that's at least good to know. But I guess it would be nice if we could follow https://en.wikipedia.org/wiki/Robustness_principle in that over accepting an UDS connection, this library ignores the authority header / tries to set it to something so nodeJS accepts it

@murgatroid99 Do you think there is a way to force the authority header to localhost before it reaches nodeJS? Or is this something that would need to be fixed in NodeJS for sure?

[travisghansen]

Pretty sure there's no reasonable way to set it in the context/environment of csi which appears to be the primary use-case of several following this... could be wrong though.

[entropitor]

@travisghansen Yes, I also don't see a way. Especially since many clients are connecting so it would mean configuring / updating many components. The only way I can see it work is if we fix this library, so it behaves like grpc-uds behaves

[murgatroid99]

If I'm understanding what's happening here correctly, Node is rejecting the request before grpc-js ever sees it. grpc-js can't do anything about that.

[entropitor]

It seems like the go client is sending an invalid authority header and thus indeed it's gonna be hard for this library to fix that 😞

[entropitor]

Given NodeJS doesn't support the broken spec that Golang uses and gRPC is used quite a lot in kubernetes, where go is master, is there any chance of merging grpc-uds (#258 (comment)) into this repo?

[murgatroid99]

No, we have decided not to publish any more feature releases for the original grpc Node library. In the grpc-go issue linked above, it looks like they have agreed to set :authority to localhost for UDS connections, so the right course of action is to wait for that fix.

[CMCDragonkai]

The PR for getting UDS support got closed: grpc/grpc#18435

[murgatroid99]

The grpc package has been deprecated, so we will not add UDS support to it. The feature seems to be working as well as it can in @grpc/grpc-js, so we recommend using that instead.

[CMCDragonkai]

Where is the docs for how to connect to UDS?

[nicolasnoble]

Where is the docs for how to connect to UDS?

https://grpc.github.io/grpc/cpp/md_doc_naming.html

[andreasdj]

Does the @grpc/grpc-js library support GRPC over UDS for clients on Windows?
The referenced documentation above (https://grpc.github.io/grpc/cpp/md_doc_naming.html) says that the unix: URI scheme is only available on Unix systems.

I have created a GRPC service on windows and configured it to use UDS, and I can successfully communicate with it using a dotnet client, but when trying to use the @grpc/grpc-js library in node I always get the error 14 UNAVAILABLE: No connection established (or when enabling logs, something like: Pick result: TRANSIENT_FAILURE subchannel: undefined status: 14 No connection established).

I wonder if this is expected behavior or if I'm providing an incorrect URI? I have tried all sorts of unix: paths but it always gives the same error. I don't have any issues with the library when using GRPC over TCP and URI:s of the dns:localhost:[portnr] format.

[murgatroid99]

The unix: URI scheme is only officially supported on Unix systems because UDS only exists on Unix systems, but in @grpc/grpc-js, the network APIs used to create Unix Domain Sockets automatically create Windows Named Pipes instead on Windows. So, if you pass a valid Named Pipe name using the unix: URI scheme, it will probably just work to connect to a named pipe. I don't know what the dotnet library does in that case, so I don't know how to make them work together.

[lichti81]

@andreasdj: Where you in the meantime able to connect ASP.NET Core GRPC server with node GRPC client over UDS on Windows?
If yes: How you solved your problem?

[huangzhendong]

您好，您的邮件我已经收到！

[andreasdj]

@lichti81: No, unfortunately we weren't able to get it up and running and I haven't spent more time on it since. We are currently using GRPC over TCP. I might get back to investigating this later.

[markcanary]

because UDS only exists on Unix systems

This is not true. UDS is natively supported in Windows as of Windows 10 Version 1803. There is full support using C++ and .NET core 2.1 and above.

See: https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ and https://bsmadhu.wordpress.com/2018/08/22/unix-domain-socket-support-in-windows/

[huangzhendong]

您好，您的邮件我已经收到！

[andreasdj]

I forgot to give an update on this topic so doing it now.

Although UDS is supported on Windows nodejs doesn't support it (and probably won't in the foresee future), see:
https://nodejs.org/api/net.html#net_ipc_support
nodejs/node#35008

So on Windows, named pipes is the way to go.

[huangzhendong]

您好，您的邮件我已经收到！

[travisghansen]

For those looking to get around the authority header issue and/or work as a uds on windows I created this: https://github.com/democratic-csi/csi-grpc-proxy

It’s a simple proxy written in go which has work-arounds for both issues. I use it successfully to listen on a uds on windows and proxy to a nodejs named pipe etc. I also use it on linux to rewrite the host header before the request is sent to nodejs uds.

Despite the name, it will work in non-csi scenarios. It does not support ssl in any way so only h2c works. It does support tcp so you can mix/match any protocol of tcp/uds/npipe for up/down stream.

The releases have pre-compiled binaries for all supported go os/arch and docker images are produced for windows/linux.

[ar0n]

Had the same issue. Worked for me using grpc-caller using unix://

[huangzhendong]

您好，您的邮件我已经收到！