Loki integration Issue

[KaustubhKishore]

Fission/Kubernetes version

$ fission version
client:
  fission/core:
    BuildDate: "2024-01-14T15:43:35Z"
    GitCommit: 7e8d5dd7
    Version: v1.20.1
server:
  fission/core:
    BuildDate: "2024-01-14T15:43:35Z"
    GitCommit: 7e8d5dd7
    Version: v1.20.1


$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1

Kubernetes platform (e.g. Google Kubernetes Engine)
MacOS Local Kuberntes

Describe the bug
While following the documentation to add Loki to the monitoring namespace and then adding it as a data source, the data source does not get added.

To Reproduce
Follow the loki integration docs step by step

Expected result

The loki data source gets added

Actual result
Error while adding data source

Additional context

kubectl get svc -n monitoring
NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                     ClusterIP   None                     9093/TCP,9094/TCP,9094/UDP   45m
loki-grafana                              ClusterIP   10.110.129.254           3100/TCP                     30m
loki-grafana-headless                     ClusterIP   None                     3100/TCP                     30m
loki-grafana-memberlist                   ClusterIP   None                     7946/TCP                     30m
prometheus-grafana                        ClusterIP   10.110.233.154           80/TCP                       45m
prometheus-kube-prometheus-alertmanager   ClusterIP   10.97.81.247             9093/TCP,8080/TCP            45m
prometheus-kube-prometheus-operator       ClusterIP   10.96.243.72             443/TCP                      45m
prometheus-kube-prometheus-prometheus     ClusterIP   10.96.178.124            9090/TCP,8080/TCP            45m
prometheus-kube-state-metrics             ClusterIP   10.110.87.134            8080/TCP                     45m
prometheus-operated                       ClusterIP   None                     9090/TCP                     45m
prometheus-prometheus-node-exporter       ClusterIP   10.107.226.251           9100/TCP                     45m

kubectl get pods -n monitoring
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          43m
loki-grafana-0                                           1/1     Running   0          29m
prometheus-grafana-9c98f646b-h76rq                       3/3     Running   0          44m
prometheus-kube-prometheus-operator-7664bd5b4b-p5p95     1/1     Running   0          44m
prometheus-kube-state-metrics-6db866c85b-vz9wx           1/1     Running   0          44m
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          43m
prometheus-prometheus-node-exporter-p6c5p                1/1     Running   0          44m
promtail-qtbp9                                           0/1     Running   0          28m

kubectl describe service -n monitoring (truncated output)

Name:              loki-grafana
Namespace:         monitoring
Labels:            app=loki
                   app.kubernetes.io/managed-by=Helm
                   chart=loki-2.16.0
                   heritage=Helm
                   release=loki-grafana
Annotations:       meta.helm.sh/release-name: loki-grafana
                   meta.helm.sh/release-namespace: monitoring
Selector:          app=loki,release=loki-grafana
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.110.129.254
IPs:               10.110.129.254
Port:              http-metrics  3100/TCP
TargetPort:        http-metrics/TCP
Endpoints:         10.1.0.94:3100
Session Affinity:  None
Events:            


Name:              loki-grafana-headless
Namespace:         monitoring
Labels:            app=loki
                   app.kubernetes.io/managed-by=Helm
                   chart=loki-2.16.0
                   heritage=Helm
                   release=loki-grafana
                   variant=headless
Annotations:       meta.helm.sh/release-name: loki-grafana
                   meta.helm.sh/release-namespace: monitoring
Selector:          app=loki,release=loki-grafana
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              http-metrics  3100/TCP
TargetPort:        http-metrics/TCP
Endpoints:         10.1.0.94:3100
Session Affinity:  None
Events:            

The documentation expects to add the following url as datasource: http://loki.monitoring.svc.local.cluster:3100
Tried multiple iterations: http://loki-grafana.monitoring.svc.local.cluster:3100, http://loki-grafana-0.monitoring.svc.local.cluster:3100, http://10.1.0.94:3100, http://loki-grafana.monitoring.svc.cluster.local:3100, http://loki-grafana-0.monitoring.svc.cluster.local:3100

This is some networking issue that hasn't been described in the documentation and according to stackoverflow svc.local.cluster should be svc.cluster.local

[KaustubhKishore]

Additional Notes: I am able to open promtail ui

This does not look 100% correct though.

[dk03051996]

hi @KaustubhKishore , can you share what error you are facing while adding the data source in grafana , you can see this at bottom

[dk03051996]

hi @KaustubhKishore , please use this "http://loki.monitoring.svc.cluster.local:3100/" and let me know if it solves your issue. I have also raised a pull request to update the same.

[KaustubhKishore]

Hi @dk03051996 , The url you are suggesting has already been tried, we'll try it once more on a fresh setup and let you know.

[dk03051996]

One more thing that you can check is if you have installed loki using our documentation and your promtail and loki pods are up and running