ca: Support using an external CA as the Trusted CA

[dnephin]

Related to #11448

Currently the Connect CA uses the following certificates with the Vault provider:

• a Root CA, stored in Vault using the root_pki_path, used to sign intermediate CA certificates
• an intermediate CA signed by that root, stored in Vault using the intermediate_pki_path, used to sign leaf certificates

Using a non-root (not self signed) CA certificate as the root_pki_path almost works. Consul CA can sign intermediate and leaf certificates, but when an application attempts to verify one of the leaf certs that verification may fail with "unknown authority". The problem is that with this setup the connect/ca API and xDS API have no way of returning the real external root, because it is never stored in Consul.

Some applications may be able to distribute that external root CA and add it as a trusted CA, but that may not always be possible. It does not appear to be possible with Envoy which Consul uses as the connect proxy.

Proposal

To support external root CAs as the trusted CA we need to store the public certificate of the real root CA in Consul, and return it as the RootCert in both the HTTP API and xDS API. The primary Connect CA will need to be added to the list of IntermediateCerts in that API response.

To make that happen, I believe we'll need to change the ca.Provider interface. The GenerateRoot, and ActiveRoot methods need to be changed to allow them to return the full chain (both the real root CA, and the intermediate used by Consul CA). That could be either as a single PEM encoded bundle, or it could be a slice of PEM encoded certificates.

To change that interface we need to audit all calls to those two methods. I believe that doing at least some of the work described in #11159 would benefit us here, as that would remove two of the calls to ActiveRoot, which removes the need to adjust them to this new interface. Any additional work required for #11159 can be left for a later time.

Testing

Since this setup does not fail until the leaf certificates are verified, the testing setup to verify this change is pretty involved. Most of the setup is already documented in #11448, but I'll repeat it here.

Environment:

• create a root CA in Vault
• create an intermediate A, signed by the root CA, and use that path as root_pki_path
• setup a third pki path as intermediate_pki_path

We need to test the following three flows in both a primary and a secondary DC (because the secondary DC has some different code paths from the primary):

1. Using the HTTP API sign a leaf certificate for a service. Download the roots using /connect/ca/roots, and show that the leaf can be verified using those roots.
2. Create two services, and run envoy proxies for each service. Show that service A can communicate with service B.
3. Enable auto_encrypt, and show that a Client Agent receives a leaf cert from the servers, and is able to perform RPC calls with TLS enabled (verify_incoming/verify_outgoing both set to true).

Other Considerations

1. Initially this will only work with the Vault provider, but by changing the interface we could add support for other providers in the future.
2. This change makes the root_pki_path (and possibly other config fields) somewhat misleading. That CA will no longer always be a root CA. In the future we may want to change the name here.

Related issues

• connect: certificate chain to contain root cert #6613 I believe is an ask for the same thing

[kphunter]

Would love to see this - it sounds like it might help solve hashicorp/consul-helm#784 (comment), where external standalone clients connect to a Kubernetes Consul cluster.

May also help avoid the requirement of a kubeconfig file on external clients (hashicorp/consul-helm#358). the hiccups observed with rotating Vault configured root CA's (hashicorp/consul-k8s#674, and #8681), and auto-encrypt CA certificate retrieval (hashicorp/consul-k8s#313).

[dnephin]

Done in #11910 (and related changes in a few PRs that you can see linked above).