Ory OAuth2 unavailable on new project

[rverma-dev]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• This issue affects my Ory Network project.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Describe the bug

Hi all, getting below error while trying to retrieve openid configuration for a professional Project.
curl https://lucid-lumiere-j7x2l100b1.projects.oryapis.com/.well-known/openid-configuration or when trying to run any SSO operation.

Seems like Hydra setup on the ory network is messed up cause of some reason.

Reproducing the bug

Not aware

Relevant log output

Hi all, getting below error while trying to retrieve openid configuration for a professional Project.
curl https://lucid-lumiere-j7x2l100b1.projects.oryapis.com/.well-known/openid-configuration
{
  "error": "server_error",
  "error_description": "The authorization server encountered an unexpected condition that prevented it from fulfilling the request. Could not ensure that signing keys for 'hydra.openid.id-token' exists. If you are running against a persistent SQL database this is most likely because your 'secrets.system' ('SECRETS_SYSTEM' environment variable) is not set or changed. When running with an SQL database backend you need to make sure that the secret is set and stays the same, unless when doing key rotation. This may also happen when you forget to run 'hydra migrate sql.."
}
The above works fine for new free tier project out of box. Is there any migrations we need run?
Also similar error for
curl https://lucid-lumiere-j7x2l100b1.projects.oryapis.com/.well-known/jwks.json
{"error":"error","error_description":"The error is unrecognizable"}

Relevant configuration

{
  "id": "615c0d55-cb87-4ce3-80a9-a189e51ebc70",
  "name": "Temporal",
  "revision_id": "17faff64-100e-4aea-9ced-51b4b5e4c01e",
  "services": {
    "identity": {
      "config": {
        "cookies": {
          "domain": "lucid-lumiere-j7x2l100b1.projects.oryapis.com",
          "path": "/",
          "same_site": "Lax"
        },
        "courier": {
          "smtp": {
            "from_name": "Temporal via Ory"
          },
          "templates": {
            "recovery": {
              "invalid": {
                "email": {
                  "body": {}
                }
              },
              "valid": {
                "email": {
                  "body": {}
                }
              }
            },
            "recovery_code": {
              "invalid": {
                "email": {
                  "body": {}
                }
              },
              "valid": {
                "email": {
                  "body": {}
                }
              }
            },
            "verification": {
              "invalid": {
                "email": {
                  "body": {}
                }
              },
              "valid": {
                "email": {
                  "body": {}
                }
              }
            }
          }
        },
        "identity": {
          "default_schema_id": "6c53c2a82a5ed43fc50d7e9facd673195dedb80f604eff2343699a41a6e6bdd32830dbb4ea197375fb3d8a3191330a4eaeac59c2e5098b8f8ef48be183354bd5",
          "schemas": [
            {
              "id": "6c53c2a82a5ed43fc50d7e9facd673195dedb80f604eff2343699a41a6e6bdd32830dbb4ea197375fb3d8a3191330a4eaeac59c2e5098b8f8ef48be183354bd5",
              "url": "https://storage.googleapis.com/bac-gcs-production/6c53c2a82a5ed43fc50d7e9facd673195dedb80f604eff2343699a41a6e6bdd32830dbb4ea197375fb3d8a3191330a4eaeac59c2e5098b8f8ef48be183354bd5.json"
            },
            {
              "id": "preset://email",
              "url": "base64://ewogICIkaWQiOiAiaHR0cHM6Ly9zY2hlbWFzLm9yeS5zaC9wcmVzZXRzL2tyYXRvcy9pZGVudGl0eS5lbWFpbC5zY2hlbWEuanNvbiIsCiAgIiRzY2hlbWEiOiAiaHR0cDovL2pzb24tc2NoZW1hLm9yZy9kcmFmdC0wNy9zY2hlbWEjIiwKICAidGl0bGUiOiAiUGVyc29uIiwKICAidHlwZSI6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIjogewogICAgInRyYWl0cyI6IHsKICAgICAgInR5cGUiOiAib2JqZWN0IiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgImVtYWlsIjogewogICAgICAgICAgInR5cGUiOiAic3RyaW5nIiwKICAgICAgICAgICJmb3JtYXQiOiAiZW1haWwiLAogICAgICAgICAgInRpdGxlIjogIkUtTWFpbCIsCiAgICAgICAgICAib3J5LnNoL2tyYXRvcyI6IHsKICAgICAgICAgICAgImNyZWRlbnRpYWxzIjogewogICAgICAgICAgICAgICJwYXNzd29yZCI6IHsKICAgICAgICAgICAgICAgICJpZGVudGlmaWVyIjogdHJ1ZQogICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgIndlYmF1dGhuIjogewogICAgICAgICAgICAgICAgImlkZW50aWZpZXIiOiB0cnVlCiAgICAgICAgICAgICAgfSwKICAgICAgICAgICAgICAidG90cCI6IHsKICAgICAgICAgICAgICAgICJhY2NvdW50X25hbWUiOiB0cnVlCiAgICAgICAgICAgICAgfQogICAgICAgICAgICB9LAogICAgICAgICAgICAicmVjb3ZlcnkiOiB7CiAgICAgICAgICAgICAgInZpYSI6ICJlbWFpbCIKICAgICAgICAgICAgfSwKICAgICAgICAgICAgInZlcmlmaWNhdGlvbiI6IHsKICAgICAgICAgICAgICAidmlhIjogImVtYWlsIgogICAgICAgICAgICB9CiAgICAgICAgICB9LAogICAgICAgICAgIm1heExlbmd0aCI6IDMyMAogICAgICAgIH0KICAgICAgfSwKICAgICAgInJlcXVpcmVkIjogWwogICAgICAgICJlbWFpbCIKICAgICAgXSwKICAgICAgImFkZGl0aW9uYWxQcm9wZXJ0aWVzIjogZmFsc2UKICAgIH0KICB9Cn0K"
            },
            {
              "id": "009a8acd6c729da0127b7467c7afa75530e3a5494c5051e614a07bb72c2b038e5a20bd024cda71022adfd5e7592c530535c5c6905710712fb91edd906bfbcdf2",
              "url": "https://storage.googleapis.com/bac-gcs-production/009a8acd6c729da0127b7467c7afa75530e3a5494c5051e614a07bb72c2b038e5a20bd024cda71022adfd5e7592c530535c5c6905710712fb91edd906bfbcdf2.json"
            },
            {
              "id": "2dd825ee0617cbc9775b2422a6098d507128f055465f447c1eb71ae509a1a95f8665d83a0a5054e570098ed94893b1d7e4bc84a7705e39cff4f944b1e57f769b",
              "url": "https://storage.googleapis.com/bac-gcs-production/2dd825ee0617cbc9775b2422a6098d507128f055465f447c1eb71ae509a1a95f8665d83a0a5054e570098ed94893b1d7e4bc84a7705e39cff4f944b1e57f769b.json"
            },
            {
              "id": "ff0217b5640eed4a00b2e564b14fc11d73be09fb6889f451d04647d34b9cd0a86b1a70549dd49f14eb237c7a402da85e433d6e8d14049d120479c2538448726f",
              "url": "https://storage.googleapis.com/bac-gcs-production/ff0217b5640eed4a00b2e564b14fc11d73be09fb6889f451d04647d34b9cd0a86b1a70549dd49f14eb237c7a402da85e433d6e8d14049d120479c2538448726f.json"
            },
            {
              "id": "49053220b1ccdd54dcd85d426525b8441f9291948f1b9b356eef343bf8b942933402f720e1c7bb6635b2c1fcd26a2b8265194ed68ce672f8690fc86c7a9d45f8",
              "url": "https://storage.googleapis.com/bac-gcs-production/49053220b1ccdd54dcd85d426525b8441f9291948f1b9b356eef343bf8b942933402f720e1c7bb6635b2c1fcd26a2b8265194ed68ce672f8690fc86c7a9d45f8.json"
            }
          ]
        },
        "oauth2_provider": {},
        "selfservice": {
          "allowed_return_urls": [
            "https://spa.nslhub.click/"
          ],
          "default_browser_return_url": "/ui/welcome",
          "flows": {
            "error": {
              "ui_url": "/ui/error"
            },
            "login": {
              "after": {
                "hooks": [],
                "oidc": {
                  "hooks": []
                },
                "password": {
                  "hooks": []
                },
                "webauthn": {
                  "hooks": []
                }
              },
              "before": {
                "hooks": []
              },
              "lifespan": "30m0s",
              "ui_url": "/ui/login"
            },
            "logout": {
              "after": {}
            },
            "recovery": {
              "after": {
                "hooks": []
              },
              "before": {
                "hooks": []
              },
              "enabled": true,
              "lifespan": "30m0s",
              "ui_url": "/ui/recovery",
              "use": "link"
            },
            "registration": {
              "after": {
                "hooks": [],
                "oidc": {
                  "hooks": [
                    {
                      "hook": "session"
                    }
                  ]
                },
                "password": {
                  "hooks": [
                    {
                      "hook": "session"
                    }
                  ]
                },
                "webauthn": {
                  "hooks": []
                }
              },
              "before": {
                "hooks": []
              },
              "enabled": true,
              "lifespan": "30m0s",
              "ui_url": "/ui/registration"
            },
            "settings": {
              "after": {
                "hooks": [],
                "password": {
                  "hooks": []
                },
                "profile": {
                  "hooks": []
                }
              },
              "before": {
                "hooks": []
              },
              "lifespan": "30m0s",
              "privileged_session_max_age": "5m0s",
              "required_aal": "highest_available",
              "ui_url": "/ui/settings"
            },
            "verification": {
              "after": {
                "hooks": []
              },
              "before": {
                "hooks": []
              },
              "enabled": false,
              "lifespan": "30m0s",
              "ui_url": "/ui/verification"
            }
          },
          "methods": {
            "code": {
              "config": {
                "lifespan": "15m0s"
              },
              "enabled": true
            },
            "link": {
              "config": {
                "base_url": "https://auth.nslhub.click/",
                "lifespan": "15m0s"
              },
              "enabled": true
            },
            "lookup_secret": {
              "enabled": false
            },
            "oidc": {
              "config": {
                "base_redirect_uri": "https://auth.nslhub.click",
                "providers": []
              },
              "enabled": false
            },
            "password": {
              "config": {
                "haveibeenpwned_enabled": true,
                "identifier_similarity_check_enabled": true,
                "ignore_network_errors": true,
                "max_breaches": 1,
                "min_password_length": 8
              },
              "enabled": true
            },
            "profile": {
              "enabled": true
            },
            "totp": {
              "config": {
                "issuer": "Temporal"
              },
              "enabled": false
            },
            "webauthn": {
              "config": {
                "passwordless": false,
                "rp": {
                  "display_name": "Temporal",
                  "id": "lucid-lumiere-j7x2l100b1.projects.oryapis.com",
                  "origin": "https://lucid-lumiere-j7x2l100b1.projects.oryapis.com"
                }
              },
              "enabled": true
            }
          }
        },
        "serve": {
          "admin": {
            "base_url": "https://lucid-lumiere-j7x2l100b1.projects.oryapis.com/",
            "request_log": {
              "disable_for_health": true
            }
          },
          "public": {
            "base_url": "https://lucid-lumiere-j7x2l100b1.projects.oryapis.com/",
            "cors": {
              "enabled": false
            },
            "request_log": {
              "disable_for_health": true
            }
          }
        },
        "session": {
          "cookie": {
            "domain": "lucid-lumiere-j7x2l100b1.projects.oryapis.com",
            "name": "ory_session_lucidlumierej7x2l100b1",
            "path": "/",
            "persistent": false,
            "same_site": "Lax"
          },
          "lifespan": "1h0m0s",
          "whoami": {
            "required_aal": "highest_available"
          }
        }
      }
    },
    "oauth2": {
      "config": {
        "clients": {
          "http": {
            "disallow_private_ip_ranges": true
          }
        },
        "dev": true,
        "hsm": {
          "enabled": false
        },
        "oauth2": {
          "client_credentials": {
            "default_grant_allowed_scope": false
          },
          "expose_internal_errors": true,
          "grant": {
            "jwt": {
              "iat_optional": false,
              "jti_optional": false,
              "max_ttl": "720h0m0s"
            }
          },
          "hashers": {
            "algorithm": "pbkdf2",
            "pbkdf2": {
              "iterations": 10000
            }
          },
          "pkce": {
            "enforced": false,
            "enforced_for_public_clients": false
          },
          "session": {
            "encrypt_at_rest": true,
            "exclude_not_before_claim": false
          }
        },
        "oidc": {
          "dynamic_client_registration": {
            "enabled": false
          },
          "subject_identifiers": {}
        },
        "serve": {
          "admin": {
            "cors": {
              "allow_credentials": true,
              "allowed_headers": [
                "Accept",
                "Content-Type",
                "Content-Length",
                "Accept-Language",
                "Content-Language",
                "Authorization"
              ],
              "allowed_methods": [
                "POST",
                "GET",
                "PUT",
                "PATCH",
                "DELETE",
                "CONNECT",
                "HEAD",
                "OPTIONS",
                "TRACE"
              ],
              "debug": false,
              "enabled": false,
              "exposed_headers": [
                "Cache-Control",
                "Expires",
                "Last-Modified",
                "Pragma",
                "Content-Length",
                "Content-Language",
                "Content-Type"
              ],
              "max_age": 0
            },
            "tls": {
              "enabled": false
            }
          },
          "cookies": {
            "domain": "lucid-lumiere-j7x2l100b1.projects.oryapis.com",
            "names": {
              "consent_csrf": "ory_oauth2_consent_csrf_lucidlumierej7x2l100b1",
              "login_csrf": "ory_oauth2_login_csrf_lucidlumierej7x2l100b1",
              "session_csrf": "ory_oauth2_session_csrf_lucidlumierej7x2l100b1"
            },
            "same_site_legacy_workaround": false,
            "same_site_mode": "Lax",
            "secure": true
          },
          "public": {
            "cors": {
              "allow_credentials": true,
              "allowed_headers": [
                "Accept",
                "Content-Type",
                "Content-Length",
                "Accept-Language",
                "Content-Language",
                "Authorization"
              ],
              "allowed_methods": [
                "POST",
                "GET",
                "PUT",
                "PATCH",
                "DELETE",
                "CONNECT",
                "HEAD",
                "OPTIONS",
                "TRACE"
              ],
              "debug": false,
              "enabled": false,
              "exposed_headers": [
                "Cache-Control",
                "Expires",
                "Last-Modified",
                "Pragma",
                "Content-Length",
                "Content-Language",
                "Content-Type"
              ],
              "max_age": 0
            },
            "tls": {
              "enabled": false
            }
          },
          "tls": {
            "enabled": false
          }
        },
        "strategies": {
          "access_token": "opaque",
          "scope": "wildcard"
        },
        "ttl": {
          "access_token": "1h0m0s",
          "auth_code": "30m0s",
          "id_token": "1h0m0s",
          "login_consent_request": "30m0s",
          "refresh_token": "720h0m0s"
        },
        "urls": {
          "consent": "/ui/consent",
          "error": "/ui/error",
          "login": "/ui/login",
          "post_logout_redirect": "/oauth2/fallbacks/logout",
          "self": {
            "admin": "https://lucid-lumiere-j7x2l100b1.projects.oryapis.com/admin",
            "issuer": "https://lucid-lumiere-j7x2l100b1.projects.oryapis.com",
            "public": "https://lucid-lumiere-j7x2l100b1.projects.oryapis.com"
          }
        },
        "webfinger": {
          "jwks": {},
          "oidc_discovery": {}
        }
      }
    },
    "permission": {
      "config": {
        "limit": {},
        "namespaces": []
      }
    }
  },
  "slug": "lucid-lumiere-j7x2l100b1",
  "state": "running"
}

Version

Network

On which operating system are you observing this issue?

Ory Network

In which environment are you deploying?

Ory Network

Additional Context

No response

[rverma-dev]

Apparently the same error message appears in all the hydra operations, odic etc...

[aeneasr]

Thank you for the report! It seems like the instance has recovered. We will start an investigation to understand what happened!

[aeneasr]

We have solved this for new projects. If you have an old project that is affected by this, please reach out to support and we will fix your instance :)