[kube-prometheus-stack] I am not able to create a prometheus ingress with aws alb controller.

[a-kumar5]

Describe the bug a clear and concise description of what the bug is.

I have enabled the ingress and provided all the necesaary information for annotations and hosts. The load balancer with target is getting created but the healthcheck is getting failed and we are not able to access the prometheus ui.

What's your helm version?

version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.20.11"

What's your kubectl version?

Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.10-eks-508b6b3

Which chart?

kube-prometheus-stack

What's the chart version?

57.1.1

What happened?

The alb helath check is getting failed.

What you expected to happen?

The alb healthcheck should pass and i should be able to access the prometheus ui.

How to reproduce it?

Create a ingress with stack using the aws alb controller and the healthcheck will fail.

Enter the changed values of values.yaml?

prometheus:
prometheusSpec:
podMonitorSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
externalUrl: "http://prometheus-dashboard.stage.nextgen.shs.com/prometheus"
routePrefix: /

ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/group.name: 'monitoring.prometheus-dashboard-internal-ingress'
alb.ingress.kubernetes.io/group.order: '1'
alb.ingress.kubernetes.io/scheme: 'internal'
alb.ingress.kubernetes.io/security-groups: sg-056d4775173488c2b
external-dns.alpha.kubernetes.io/hostname: prometheus-dashboard.stage.nextgen.oh.shs.com
external-dns.alpha.kubernetes.io/route53-zone-type: private
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:523140806637:certificate/f1202f91-8b24-4c19-b361-de22c3ad5739
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '10'
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '60'
alb.ingress.kubernetes.io/healthy-threshold-count: '2'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '10'
alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=60
alb.ingress.kubernetes.io/success-codes: '200-299'
labels:
k8s-app: prometheus-dashboard

hosts:
  - prometheus-dashboard.stage.nextgen.shs.com

## Paths to use for ingress rules - one path should match the prometheusSpec.routePrefix
##
paths:
  - /prometheus(/|$)(.*)

Enter the command that you execute and failing/misfunctioning.

helm upgrade prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --namespace monitoring --values values.yml

Anything else we need to know?

My eks is in private network.

[jimbeck]

I'm running into this issue too. I have grafana and the alertmanager ingress working fine using the alb ingress controller. For some reason with prometheus it doesn't create the target group in the load balancer. Will report back if I figure it out!