Fix OCP HSTS errors #282

jacobdotcosta · 2023-03-29T15:59:20Z

Issue

When we access the ocp console, then we got such a message from google chrome

You cannot visit oauth-openshift.apps.ocp.snowdrop.dev right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

HSTS: https://developer.mozilla.org/en-US/docs/Glossary/HSTS

Temporary workaround is to type thisisunsafe within the google chrome window - https://stackoverflow.com/questions/33268264/chromethe-website-uses-hsts-network-errors-this-page-will-probably-work-late

The text was updated successfully, but these errors were encountered:

cmoulliard · 2023-04-11T08:15:40Z

Dont forget to add the link to the issue/PR to track that from your tasks ;-)

cmoulliard · 2023-05-25T10:44:11Z

Can you add the link to the ticket created please ? @jacobdotcosta

jacobdotcosta · 2023-05-29T09:45:48Z

https://redhat.service-now.com/surl.do?n=RITM1483142

@cmoulliard

cmoulliard · 2023-05-30T14:48:22Z

I think that I know how to fix it.

We dont have issues when we access https://console-openshift-console.snowdrop-eu-de-1-bx2-4x16-0c576f1a70d464f092d8591997631748-0000.eu-de.containers.appdomain.cloud/ as their HTTPS certificate has been signed for the domain *.cloud by Let's encrypt - https://letsencrypt.org/certificates/#intermediate-certificates

instead the certificate generated by the ocp ingress operator for us

So, if we ask to Let's encrypt and Godaddy to sign a Certificate request for *.apps.ocp.snowdrop.dev, that should work.

The next challenge will be to pass such a certificate within the Openstack ocp install config file. Maybe using: https://docs.openshift.com/container-platform/4.13/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-describing-cloud-parameters_installing-openstack-installer-custom

WDYT ? @jacobdotcosta

cmoulliard · 2023-05-30T16:28:38Z

I tested manually this procedure after updating the certificate and issuer role of k8s-infra, collected the updated secret and patched as described here: https://docs.openshift.com/container-platform/4.13/security/certificates/replacing-default-ingress-certificate.html#replacing-default-ingress_replacing-default-ingress

Now, the access to https://console-openshift-console.apps.ocp.snowdrop.dev/dashboards do not generate a HTST error ;-)

oc login is reporting nevertheless an error

oc login --token=sha256~xxxx --server=https://api.ocp.snowdrop.dev:6443
error: x509: “kube-apiserver-lb-signer” certificate is not trusted

FYI: @jacobdotcosta

cmoulliard · 2023-11-21T10:04:26Z

A part of the problem could be automated if we pass to the openshift installer config file the parameter additionalTrustBundle
See how that works here: https://github.com/openshift/installer/blob/master/docs/user/customization.md#proxy which corresponds to what it is described here: https://docs.openshift.com/container-platform/4.13/security/certificates/replacing-default-ingress-certificate.html#replacing-default-ingress_replacing-default-ingress (create a custom CA secret and patch proxy/cluster)

NOTE: We need of course to populate first and separately the CA crt and keys before to create a cluster !

@jacobdotcosta

cmoulliard · 2023-11-21T18:15:12Z

I still have an issue using such certificate/issuer

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod-qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email:
    privateKeySecretRef:
      name: letsencrypt-prod-snowdrop-dev
    solvers:
      - dns01:
          webhook:
            config:
              apiKeySecretRef:
                name: godaddy-api-key
                key: token
              production: true
              ttl: 600
            groupName: acme.mycompany.com
            solverName: godaddy
        selector:
          dnsNames:
            - 'api.qshift.snowdrop.dev'
            - 'oauth-openshift.apps.qshift.snowdrop.dev'
            - 'console-openshift-console.apps.qshift.snowdrop.dev'
            - 'grafana-openshift-monitoring.apps.qshift.snowdrop.dev'
            - 'prometheus-k8s-openshift-monitoring.apps.qshift.snowdrop.dev'
            - 'integrated-oauth-server-openshift-authentication.apps.qshift.snowdrop.dev'
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  renewBefore: 2136h
  duration: 2190h
  privateKey:
    size: 2048
    algorithm: RSA
  issuerRef:
    kind: Issuer
    name: letsencrypt-prod-qshift-snowdrop-dev
  secretName: qshift-snowdrop-dev-tls
  dnsNames:
    - 'api.qshift.snowdrop.dev'
    - 'oauth-openshift.apps.qshift.snowdrop.dev'
    - 'console-openshift-console.apps.qshift.snowdrop.dev'
    - 'grafana-openshift-monitoring.apps.qshift.snowdrop.dev'
    - 'prometheus-k8s-openshift-monitoring.apps.qshift.snowdrop.dev'
    - 'integrated-oauth-server-openshift-authentication.apps.qshift.snowdrop.dev'

cmoulliard · 2023-11-22T09:53:02Z

Problem fixed after issuing the following certificate

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod-qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email:
    privateKeySecretRef:
      name: letsencrypt-prod-snowdrop-dev
    solvers:
      - dns01:
          webhook:
            config:
              apiKeySecretRef:
                name: godaddy-api-key
                key: token
              production: true
              ttl: 600
            groupName: acme.mycompany.com
            solverName: godaddy
        selector:
          dnsNames:
            - '*.apps.qshift.snowdrop.dev'
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  renewBefore: 2136h
  duration: 2190h
  privateKey:
    size: 2048
    algorithm: RSA
  issuerRef:
    kind: Issuer
    name: letsencrypt-prod-qshift-snowdrop-dev
  secretName: qshift-snowdrop-dev-tls
  dnsNames:
    - '*.apps.qshift.snowdrop.dev'

cmoulliard · 2023-11-22T09:54:59Z

Question from Antonio: To install the new cert manager on the OCP cluster we would still need the one on the snowdrop k8s server right?

Charles's response:
No.

It is not needed to centralize everything on a cluster EXCEPT when rotation will take place and when secrets (= ca, tls) are still used.

So if we don't centralize, that means that we need to find a way to "copy/scp" the secret or rotated secret to the target cluster
If we install a cert manager / cluster, then it can also manage the rotation and save locally the secret. If the cluster is then removed, no need anymore to rotate the cert as needed by letsencrypt after 3months

@jacobdotcosta

jacobdotcosta self-assigned this Mar 29, 2023

jacobdotcosta transferred this issue from snowdrop/team Apr 10, 2023

jacobdotcosta mentioned this issue Apr 11, 2023

Fix OCP HSTS errors snowdrop/team#1026

Open

jacobdotcosta added the module::ocp label Apr 12, 2023

cmoulliard mentioned this issue Jun 6, 2023

Implement playbook/roles that manages GoDaddy DNS records #315

Open

4 tasks

jacobdotcosta mentioned this issue Aug 10, 2023

Changes for deploying k8s and certificate rotation #337

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix OCP HSTS errors #282

Fix OCP HSTS errors #282

jacobdotcosta commented Mar 29, 2023 •

edited by cmoulliard

cmoulliard commented Apr 11, 2023

cmoulliard commented May 25, 2023

jacobdotcosta commented May 29, 2023 •

edited

cmoulliard commented May 30, 2023

cmoulliard commented May 30, 2023 •

edited

cmoulliard commented Nov 21, 2023

cmoulliard commented Nov 21, 2023

cmoulliard commented Nov 22, 2023

cmoulliard commented Nov 22, 2023

Fix OCP HSTS errors #282

Fix OCP HSTS errors #282

Comments

jacobdotcosta commented Mar 29, 2023 • edited by cmoulliard

Issue

cmoulliard commented Apr 11, 2023

cmoulliard commented May 25, 2023

jacobdotcosta commented May 29, 2023 • edited

cmoulliard commented May 30, 2023

cmoulliard commented May 30, 2023 • edited

cmoulliard commented Nov 21, 2023

cmoulliard commented Nov 21, 2023

cmoulliard commented Nov 22, 2023

cmoulliard commented Nov 22, 2023

jacobdotcosta commented Mar 29, 2023 •

edited by cmoulliard

jacobdotcosta commented May 29, 2023 •

edited

cmoulliard commented May 30, 2023 •

edited