OCSP mode assumes only one CA cert, for the current issuer

[philpennock]

Defect

Versions of nats-server and affected client libraries used:

nats-server 2.9.11, nats-cli current HEAD

OS/Container environment:

FreeBSD Jail, but reproduces anywhere

Steps or code to reproduce the issue:

Have a TLS block with a ca_file with more than one cert in it, and setup an OCSP block.

tls {
    ca_file:   /etc/ssl/cert.pem
    cert_file: /etc/nats/pkix/nats.home.pennock.cloud/tls.crt
    key_file:  /etc/nats/pkix/nats.home.pennock.cloud/tls.key
}
ocsp {
    mode: always
    url: "http://r3.o.lencr.org"
}

Expected result:

The server starts up, requests and obtains an OCSP staple, and I can query it.

Actual result:

If the ca_file bundle contains just one certificate, the intermediate used for issuing the current certificate, then things work. This is problematic, because at a bare minimum with a CA such as Let's Encrypt, you should be configuring both R3 and R4 (or E1 and E2) together, so that LE can fail-over to their stand-by. The ca_file should be a list of all certificate authorities trusted for verifying peer identities, not a statement of which specific intermediate issued the current server certificate.

If the ca_file contains a list of all the normal system CA certs, then with OCSP enabled the nats-server start-up errors out with:

% /usr/local/sbin/nats-server -DV -c /etc/nats/nats.conf -l /dev/stdout 
nats-server: CN=Thawte Universal CA Root,OU=Thawte Universal CA Root,O=Thawte invalid ca basic constraints: is not ca

If the ca_file contains /etc/ssl/bundle-lets-encrypt.pem which is a bundle of just the Let's Encrypt certs (E1, E2, R3, R4), then nats-server errors out with:

% /usr/local/sbin/nats-server -DV -c /etc/nats/nats.conf -l /dev/stdout  
nats-server: bad OCSP status update for certificate at '/etc/nats/pkix/nats.home.pennock.cloud/tls.crt': failed to get remote status: ocsp: error from server: unauthorized

which is OCSP-speak in this scenario for "sent the wrong cert details".

If I strip ca_file down to contain only the R3 certificate, then I get working OCSP; running the new natscli command:

nats account tls --ocsp --no-pem

correctly yields:

# OCSP: GOOD status=0 sn=295101430811608931834342633074769021965446 producedAt=(2023-01-09 22:46:00 +0000 UTC) thisUpdate=(2023-01-09 22:00:00 +0000 UTC) nextUpdate=(2023-01-16 21:59:58 +0000 UTC)

[tbeets]

Ran into this today doing an OCSP Peer + OCSP Staple interop unit test.

My OCSP Peer cert test tree has multiple intermediate CAs to rest revocation at both leaf and intermediate chainlinks. Basic assumption is that ca_file cannot be relied on to have intermediate CAs at all and that these will be included (along with the leaf cert) in the cert_file bundle pem.

[tbeets]

For OCSP Staple, challenge will be to correctly

A) Choose the right cert (the leaf) out of a cert_file bundle (and always a chance operator has misconfigured this bundle making it ambiguous)
B) Find the server's leaf direct issuer (which could reside in cert_file or ca_file or in neither)

If A & B are right it still may not account for the Intermediate CA being revoked (i.e. Leaf not revoked, but it's CA is revoked by its issuer).

[tbeets]

OCSP Peer has an easier time with it because OCSP verification happens at runtime only when the external peer has passed TLS verification (trust chain) and OCSP verification can happen over a nicely ordered set of full chain(s).

[tbeets]

Fixed in branch tgb/issue-3773-on-ocsp-peer. After OCSP Peer PR merges, I will rebase on Dev and PR.

Fixed local issuer determination is as follows:

1. Configured server certificate (typically cert_file but could be cert_store on Windows) will be checked for inclusion of valid issuer certificate in second position. A signature verification will be performed to insure that the candidate issuer certificate is in fact the issuer. The server's leaf cert is required to be in first position.

Otherwise,

2. Server's certificate will be locally verified -- meaning at least one full trust chain determinable -- from the server's CA trust store (typically ca_file but could be host system's store). If at least one verified chain determined, then issuer is determined from the second position of the first verified chain.

For [2] the ordering of certificates in the CA trust store does not matter as ordered chains are created.

[wallyqs]

Fixed via #4355