How to use the cloud-sql-auth proxy with Private Service Connect from developer machines

[espenmeidell]

Question

Hi, according to the docs [1] the cloud-sql-auth proxy requires the private DNS zone to contain a record mapping the instance DNS to the service attachment IP. This works great from within the VPC, but not from laptops connected to the VPC via a VPN.

The command I'm running:

cloud-sql-proxy --auto-iam-authn --psc PROJECT:REGION:REDACTED

Results in errors like this:

2024/03/25 11:06:09 [REDACTED] failed to connect to instance: Dial error: failed to dial (connection name = "PROJECT:REGION:REDACTED"): dial tcp: lookup XXXXXXXX.YYYYYYYYY.REGION.sql.goog.: no such host

To solve this I can add an entry in my local hosts file mapping XXXXXXXX.YYYYYYYYY.REGION.sql.goog to the IP address of the forwarding rule (private, but accessible via my VPN). This works, but the developer ergonomics isn't great.

Is there a plan to make this simpler, or is there something obvious I'm missing? I really want to be able to use the auth proxy so I don't have to manage certificates etc.

Let me know if you need any more information, thanks!

1. https://cloud.google.com/sql/docs/postgres/configure-private-service-connect#connect-cloud-sql-auth-proxy

Code

No response

Additional Details

No response

[enocom]

Have you confirmed that your Cloud DNS is reachable through the VPN?

See https://cloud.google.com/dns/docs/troubleshooting#private-zone-not-resolving-through-vpn-or-interconnect.

[espenmeidell]

Have you confirmed that your Cloud DNS is reachable through the VPN?

See https://cloud.google.com/dns/docs/troubleshooting#private-zone-not-resolving-through-vpn-or-interconnect.

In my case I use a wireguard based VPN, not Cloud VPN or Interconnect, but I'm going to have a look at my setup to see if I can solve the DNS resolution there.

In any way that could cause problems for situations where a database has several consumer networks connected via PSC, as they may have different IP addresses for XXXXXXXX.YYYYYYYYY.REGION.sql.goog depending on the VPC.

I had a brief look at the code, and it looks like the host resolution etc. is managed by the cloudsqlconn library, so I suspect this might not be that easy to solve in the cloud-sql-auth-proxy itself?

[enocom]

We also support cloudsqlconn (aka the Go Connector), so if there's a change to make, we could make it.

That said, this seems like a networking problem -- the Go Connector relies on a network path (and DNS being available) for PSC to function correctly.

Tell me more about your VPN setup. Does the VPN land in one VPC?

[espenmeidell]

The VPN routes based on CIDR block, and doesn't land in one specific VPC. The IP ranges of the VPCs don't overlap, so in theory I could connect to forwarding roles in two VPCs at once. One solution I am considering is adding a separate private DNS server that only is used for the VPN when looking up REGION.sql.goog domains.

[enocom]

I'm not familiar enough with the range of VPN setups, but one tricky issue of using a VPN that routes based on CIDR blocks across VPCs is that you're going to hit the problem you mentioned earlier: PSC endpoints are per-VPC. So in theory, you could have two PSC endpoints in two VPCs both with the same DNS name, but with non-determinate resolution (e.g., one resolves to one IP in one VPC, and another IP in another VPC).

Let me check with the Connectivity team on this to see if they have any suggestions.

[lrewega]

Relatedly it would be helpful if we could use the --psc flag (or similar) for an instance that is reachable via PSC, but for which the Cloud SQL "PSC" option is not enabled. In particular, the problem seems to be that such instances do not have any DNS assignment for cloud-sql-proxy/cloudsqlconn to consume. I would find it most useful if I could simply tell the proxy what address to connect to rather than relying on the discovery process. It seems silly that we can manipulate this freely via DNS, but only if the discovery process expects DNS to be used.

For example, if an instance has a Private IP, and that instance is ultimately accessible via a PSC Endpoint in a different project/VPC, there seems to be no way to convince cloud-sql-proxy to simply use that endpoint. From what I can tell it is functionally equivalent to what the --psc mode is intended for, given that this works fine without cloud-sql-proxy if such access is permitted.

Adding an option to provide an explicit address for an instance would make configuring internal/split-horizon DNS optional, making it easier to bridge access across VPCs.

[enocom]

@lrewega I think you're saying you want to use the Proxy with PSC endpoints that you've manually created. Is that correct?

[DMarby]

@lrewega I think you're saying you want to use the Proxy with PSC endpoints that you've manually created. Is that correct?

Yes, by effectively being able to tell the proxy that a database is accessible at a configurable IP/Port, but still have the proxy handle certs/auth/etc.

[enocom]

Got it. So that's an interesting idea. Part of the value of the proxy is that it handles the busy work of provisioning certificates and rotating them hourly. Assuming there's no security issue lurking (and there often is), we could disable the IP address resolution and let callers set the target IP.

Out of curiosity, what's stopping you from using Cloud SQL's built-in support for PSC? FWIW the team does have DNS automation on their roadmap too.

[DMarby]

Out of curiosity, what's stopping you from using Cloud SQL's built-in support for PSC? FWIW the team does have DNS automation on their roadmap too.

There's a few cases where one may not be able to use the built-in one (or the IP for it is not reachable directly), such as existing databases where it can't necessarily be enabled, certain types of cross-project access, or if you are exposing it through a VPN where you can't route the actual PSC IP over it due to CIDR conflicts, etc.

[ArthurCNR]

I'm getting the same problem when I use the --psc flag I get the no such host message. In my vpc network I have the dns registered. I believe that the active psc does not work on my local machine for development

[espenmeidell]

A small update on how I solved it: I created a small DNS server that contains records for the relevant databases and leveraged my VPNs support for split horizon DNS.

Apps running in the VPC use private Google managed DNS zone like normally.

This works, but it would be nice to just be able to specify how to reach the DB like @DMarby mentioned above.

[enocom]

In that case, let me change this to a feature request. The team is working on some major improvements to the server certificate hierarchy which might address this request. For now, I'd like to leave this open to gather more insight into what pains people are running into trying to use PSC.

@ArthurCNR One thing you can check is if your DNS record resolves, e.g.,

dig <PSC_DNS_HERE>

If that doesn't resolve, you have a DNS error (which it sounds like).