Support DoT (DNS over TLS) for Recursive Nameservers

[zamnuts]

Is your feature request related to a problem? Please describe.
When using split-horizon DNS or when a recursive resolver is not available, external recursive nameservers are recommended for use, see #2428 and #2143. Only UDP and TCP nameservers are supported, while neither DoH nor DoT are supported.

Describe the solution you'd like
Allow support for DNS over TLS. DoT is supported via miekg/dns#300. Specify using:

--dns01-recursive-nameservers=tls://9.9.9.9:853

# or possibly
--dns01-recursive-nameservers=tcp-tls://9.9.9.9:853

DNS over HTTPS is preferred, but DoH support was removed due to miekg/dns#800. Would specify using:

--dns01-recursive-nameservers=https://dns10.quad9.net/dns-query

Describe alternatives you've considered

• Be OK with plaintext DNS lookups over the Intranet and Internet (trying to avoid this!)
• Support a recursive resolver on-premise (not supported by forwarders, e.g. AdGuard)
• Don't use split-horizon (this is used to avoid network hairpinning via DNS)

Additional context
TCP and UDP are only supported per &dns.Client{Net: "udp", Timeout: DNSTimeout} and &dns.Client{Net: "tcp", Timeout: DNSTimeout}: https://github.com/jetstack/cert-manager/blob/59c2a2d9f410e49cd9c43890e22fcc922655144a/pkg/issuer/acme/dns/util/wait.go#L160-L190

I brute-forced many options thinking I missed something in the code, all failed:

--dns01-recursive-nameservers-only --dns01-recursive-nameservers=https://dns10.quad9.net/dns-query
E0628 18:43:46.294949       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="dial udp: address udp///dns10.quad9.net/dns-query: unknown port" "key"="foo/secret-rv76m-4132158546-3531276794" 

--dns01-recursive-nameservers-only --dns01-recursive-nameservers=https://dns10.quad9.net:443/dns-query
E0628 19:00:54.837306       1 main.go:38] cert-manager "msg"="error executing command" "error"="error validating options: invalid DNS server (address https://dns10.quad9.net:443/dns-query: too many colons in address): https://dns10.quad9.net:443/dns-query"  

--dns01-recursive-nameservers-only --dns01-recursive-nameservers=dns10.quad9.net/dns-query
E0628 18:50:52.175200       1 main.go:38] cert-manager "msg"="error executing command" "error"="error validating options: invalid DNS server (address dns10.quad9.net/dns-query: missing port in address): dns10.quad9.net/dns-query"  

--dns01-recursive-nameservers-only --dns01-recursive-nameservers=9.9.9.9:853
E0628 18:57:27.651395       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="read udp 10.42.1.134:39324->9.9.9.9:853: read: no route to host" "key"="foo/secret-rv76m-4132158546-3531276794" 

--dns01-recursive-nameservers-only --dns01-recursive-nameservers=tls://9.9.9.9:853
E0628 18:59:07.420920       1 main.go:38] cert-manager "msg"="error executing command" "error"="error validating options: invalid DNS server (address tls://9.9.9.9:853: too many colons in address): tls://9.9.9.9:853"  

--dns01-recursive-nameservers-only --dns01-recursive-nameservers=tcp-tls://9.9.9.9:853
E0628 18:59:58.567351       1 main.go:38] cert-manager "msg"="error executing command" "error"="error validating options: invalid DNS server (address tcp-tls://9.9.9.9:853: too many colons in address): tcp-tls://9.9.9.9:853"  

Environment details (remove if not applicable):

• Kubernetes version: v1.19.9
• Cloud-provider/provisioner: nocloud (self-hosted), letsencrypt.org, cloudflare
• cert-manager version: v1.0.4
• Install method: helm

helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.0.4 \
  --set installCRDs=true \
  --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=9.9.9.9:53}'

/kind feature

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[zamnuts]

/remove-lifecycle stale

[carpenike]

would love to see this as well. Ideally would like to redirect all :53 traffic to my router, but need external DNS for cert-manager.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[zamnuts]

/remove-lifecycle stale

[SgtCoDFish]

Something sort-of similar (DoH) is being worked on in this PR: #5003

If that landed, would it solve your issues? (I note you say you'd actually prefer DoH to DoT)

[zamnuts]

I think that would satisfy this request.
DoH or DoT, either would suffice.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[zamnuts]

waiting on #5003

/remove-lifecycle stale

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[zamnuts]

still waiting on #5003

/remove-lifecycle stale

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[zamnuts]

#5003 is still an open thing

/remove-lifecycle stale

[maelvls]

Note to anyone following this thread: @FlorianLiebhart and I are actively working on #5003. I feel I have a good case to show the other maintainers to convince them that DNS-over-HTTPS with the JSON protocol makes sense to be implemented right into cert-manager.

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[zamnuts]

can confirm DoH is now supported for --dns01-recursive-nameservers. resolved via #5003 and released as part of v1.13.0