Support HTTP(s) healthcheck for K8s API server NLB

[jaininshah9]

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.
We have a use-case of supporting HTTP/HTTPS health checks (that gives different HTTP Codes) for the K8s api server NLB created via kOps when a cluster is created. Currently from code, I see we only support TCP healthchecks (since that's the default for NLB)

Can we have a optional setting in spec.api.loadbalancer where we can override the TargetGroup healthchecks options?
(ref: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html#health-check-settings)

2. Feel free to provide a design supporting your feature request.
This should be fairly straightforward to change for AWS (since I am familiar with AWS, I can speak on it). I can submit a PR if this is okay to do.

[hakman]

Hi @jaininshah9, could you explain more your use case?
@rifelpet, do you remember why we used TCP healthcheck for NLB instead of pointing it to wellknownports.KubeAPIServerHealthCheck?
I think this is what is used for GCE:

kops/pkg/model/gcemodel/api_loadbalancer.go

Lines 43 to 49 in fdd3f0e

	healthCheck := &gcetasks.HTTPHealthcheck{
	Name: s(b.NameForHealthcheck("api")),
	Port: i64(wellknownports.KubeAPIServerHealthCheck),
	RequestPath: s("/healthz"),
	Lifecycle: b.Lifecycle,
	}
	c.AddTask(healthCheck)

[rifelpet]

I'm guessing we used TCP healthchecks on the traffic port for NLB because it is the default for TCP target groups.

We could consider switching to HTTPS targeting the health check port and path to be consistent with GCE

[jaininshah9]

Yeah, for us we wanted better support for HTTP codes to make sure during the api-server rolling restart, we have more control over the health status of the server.