How to discover hosts in ingresses with k8s targets

[0leksei]

I'm trying to find a way to dynamically discover k8s ingresses and make HTTP checks against "Hosts" which are defined in ingress spec. Ingresses seem to be discovered fine, but how to use "Hosts" spec value on which to run actual check?

The config I use:

probe {
      name: "uptime_checks"
      type: HTTP
      targets {
        k8s {
          ingresses: "app1-(.*)-api(.*)|app2-(.*)-api(.*)|app3-(.*)-api(.*)"
        }
      }
      http_probe {
        protocol: HTTPS
        port: 443
        relative_url: "/healthz"
      }
      validator {
        name: "Healthy"
        regex: "Healthy"
      }
    }

With the above configuration, all targets are reported as unhealthy.

If I explicitly specify targets with host_names, then targets are found healthy as they should:

probe {
      name: "uptime_checks"
      type: HTTP
      targets {
	host_names: "app1-trade-api-test.domain.com,app2-delivery-api-test.domain.com,app3-events-api.domain.com"
      }
      http_probe {
        protocol: HTTPS
        port: 443
        relative_url: "/healthz"
      }
      validator {
        name: "Healthy"
        regex: "Healthy"
      }
    }

This is how ingresses look on k8s:

> kubectl get ing

NAME                               CLASS    HOSTS                                   ADDRESS     PORTS   AGE
app1-trade-api-domain-com   	   <none>   app1-trade-api-test.domain.com          10.1.1.1   80      105d
app2-delivery-api-domain-com       <none>   app1-delivery-api-test.domain.com       10.1.1.1   80      77d
app3-events-api-domain-com         <none>   app1-events-api.domain.com              10.1.1.1   80      10d

I suspect that I'm missing some filters or labels in the config to use with k8s targets, appreciate advise here.

PS Great tool by the way! Really happy that I found it and eagerly looking to make it work in our environment to run uptime checks inside the network.

[manugarg]

@0leksei Cloudprober by default should use ingress.Spec.Rule.Host as host header, and either rule's path or probe's relative URL (with latter being preferred) as relative URL.

However, Cloudprober uses ingress's load balancer IP/hostname from status (status.LoadBalancer.Ingress.{IP,Hostname}) as the host to connect to. In case of AWS, it's usually in the form of <aws-resource-id>.<region>.elb.amazonaws.com.

Can you please share what you're getting in the logs for probe failures?

[0leksei]

I am running Cloudprober on GKE and using nginx-ingress as ingress controller.

Ingress status looks like this:

status:
  loadBalancer:
    ingress:
    - ip: 10.1.1.1

The targets names are populated in a form of <ingress_name>_<ingress_host>, with underscore between them - is this correct result? This is how it looks in /status (hiding real app and domain names):

Regarding the logs - which logs should I look into? By default I only get these logs in Google Logs from the Cloudprober deployment:

[manugarg]

Yeah, the target names look correct. Cloudprober does that to differentiate between hosts within an ingress. But it still uses the correct host and IP address, or at least it should.

On GKE logs go directly to Google Stackdriver. They call it Cloud Operations Suite now I think (logs in Stackdriver are index by probe names -- cloudprober.uptime_checks in your case). You can use the flag --disable_cloud_logging to disable stackdriver logging. Logs will have information on why probes are failing.

[0leksei]

@manugarg Thanks for the tips! Found the logs in the correct log name, and it shows this for all discovered ingresses:

Target:app1-trade-api-domain-com_app1-trade-api-test.domain.com, URL:https://10.1.1.1:443/healthz, http.doHTTPRequest: Get "https://10.1.1.1:443/healthz": tls: failed to verify certificate: x509: cannot validate certificate for 10.1.1.1 because it doesn't contain any IP SANs"

To me it looks that Cloudprober connects to ingress using IP address instead of a hostname, which is causing TLS certificate error. Is there anything I need to add to the config to make it connect using a hostname?

[0leksei]

Noticed that probes work if I add this to http_probe settings:

        tls_config {
          disable_cert_validation: true
        }

But would like to keep TLS cert validation enabled eventually.

[manugarg]

We should be using rule's host name for TLS ServerName. I have an idea of what might be going on here. To confirm, can you please try adding: resolve_first: true to your config?

If it works, I'll change it so that this option is not required anymore.

[0leksei]

It works after adding resolve_first: true to the http_probe config.

[manugarg]

Filed an issue to fix this: #417.