[tempo] Add Network Policy capability

[Sheikh-Abubaker]

Fixes #2890

[zanhsieh]

I think feature should bump to minor version:

Suggested change

	version: 1.7.2
	version: 1.8.0

[zanhsieh]

Can you get rid of extra comment lines?

[zanhsieh]

Can you get rid of extra comment lines?

[Sheikh-Abubaker]

@zanhsieh please checkout I've done the required changes.

[Sheikh-Abubaker]

@zanhsieh I think I found some missing entries that should be in values.yaml but are not, please correct me if I am wrong, this code section below in charts/grafana/templates/networkpolicy.yaml is defined to include any labels and annotations defined in values.yaml right ? but when I searched through values.yaml I was unable to find any entries for labels though I can see entry for annotations but it is commented out, what do you think about this ?

    {{- with .Values.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
  {{- with .Values.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}

[zanhsieh]

@Sheikh-Abubaker
That's why the code go with {{ with }}.

[hkailantzis]

Hi, is there anything else pending for this ? Or just an additional review required ? Thanks a lot @Sheikh-Abubaker.

[Sheikh-Abubaker]

@hkailantzis I've added this resource from referring to : https://github.com/grafana/helm-charts/blob/main/charts/grafana/templates/networkpolicy.yaml

I have a doubt in this part of the code:

egress:
    {{- if not .Values.networkPolicy.egress.blockDNSResolution }}
    - ports:
        - port: 53
          protocol: UDP
    {{- end }}

As for the grafana networkpolicy the port is defined as 53, which port should be assigned for tempo networkpolicy ? Please correct me if I am wrong but as per my understanding since it a general K8s resource then we should utilize the same port i.e 53 ?

[hkailantzis]

@Sheikh-Abubaker , I'm really not sure about this tbh. I suspect is something general on networking, like when blocking DNS resolution which runs on port 53. is related with Firewalls etc. ie: https://library.netapp.com/ecmdocs/ECMP1155586/html/GUID-D052D155-EF55-4D19-A70F-B9A8FA86A6D3.html#:~:text=The%20Domain%20Name%20System%20(DNS,name%20and%20IP%20address%20lookups.

So, is basically if someone wishes to Block DNS resolution, then the template just says: if blockDNSResolution=true, then do NOT allow outgoing traffic via UDP port 53. So is an generic additional feature, I guess you could include it also here.

[Sheikh-Abubaker]

So, is basically if someone wishes to Block DNS resolution, then the template just says: if blockDNSResolution=true, then do NOT allow outgoing traffic via UDP port 53. So is an generic additional feature, I guess you could include it also here.

@hkailantzis thanks for the additional info, I hope this gets merged after the awaited additional review.

[Sheikh-Abubaker]

@hkailantzis can you please help me in testing the network policy, I'm facing some issues:

Here are the insights from my system:

$ helm install my-tempo ./tempo-1.8.0.tgz
NAME: my-tempo
LAST DEPLOYED: Wed Mar 27 03:58:51 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

$ kubectl get all
NAME             READY   STATUS    RESTARTS   AGE
pod/my-tempo-0   1/1     Running   0          18s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
            AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP
            36s
service/my-tempo     ClusterIP   10.102.51.181   <none>        3100/TCP,6831/UDP,6832/UDP,14268/TCP,14250/TCP,9411/TCP,55680/TCP,55681/TCP,4317/TCP,4318/TCP,55678/TCP   18s

NAME                        READY   AGE
statefulset.apps/my-tempo   1/1     18s

$ kubectl get networkpolicy
NAME       POD-SELECTOR                                                       AGE
my-tempo   app.kubernetes.io/instance=my-tempo,app.kubernetes.io/name=tempo   30s

$ kubectl describe networkpolicy my-tempo
Name:         my-tempo
Namespace:    default
Created on:   2024-03-27 03:58:51 +0530 IST
Labels:       app.kubernetes.io/instance=my-tempo
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=tempo
              app.kubernetes.io/version=2.3.1
              helm.sh/chart=tempo-1.8.0
Annotations:  meta.helm.sh/release-name: my-tempo
              meta.helm.sh/release-namespace: default
Spec:
  PodSelector:     app.kubernetes.io/instance=my-tempo,app.kubernetes.io/name=tempo
  Allowing ingress traffic:
    To Port: <nil>/TCP
    From: <any> (traffic not restricted by source)
  Not affecting egress traffic
  Policy Types: Ingress

$ kubectl describe service my-tempo
Name:              my-tempo
Namespace:         default
Labels:            app.kubernetes.io/instance=my-tempo
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=tempo
                   app.kubernetes.io/version=2.3.1
                   helm.sh/chart=tempo-1.8.0
Annotations:       meta.helm.sh/release-name: my-tempo
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/instance=my-tempo,app.kubernetes.io/name=tempo
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.102.51.181
IPs:               10.102.51.181
Port:              tempo-prom-metrics  3100/TCP
TargetPort:        3100/TCP
Endpoints:         10.244.0.3:3100
Port:              tempo-jaeger-thrift-compact  6831/UDP
TargetPort:        6831/UDP
Endpoints:         10.244.0.3:6831
Port:              tempo-jaeger-thrift-binary  6832/UDP
TargetPort:        6832/UDP
Endpoints:         10.244.0.3:6832
Port:              tempo-jaeger-thrift-http  14268/TCP
TargetPort:        14268/TCP
Endpoints:         10.244.0.3:14268
Port:              grpc-tempo-jaeger  14250/TCP
TargetPort:        14250/TCP
Endpoints:         10.244.0.3:14250
Port:              tempo-zipkin  9411/TCP
TargetPort:        9411/TCP
Endpoints:         10.244.0.3:9411
Port:              tempo-otlp-legacy  55680/TCP
TargetPort:        55680/TCP
Endpoints:         10.244.0.3:55680
Port:              tempo-otlp-http-legacy  55681/TCP
TargetPort:        4318/TCP
Endpoints:         10.244.0.3:4318
Port:              grpc-tempo-otlp  4317/TCP
TargetPort:        4317/TCP
Endpoints:         10.244.0.3:4317
Port:              tempo-otlp-http  4318/TCP
TargetPort:        4318/TCP
Endpoints:         10.244.0.3:4318
Port:              tempo-opencensus  55678/TCP
TargetPort:        55678/TCP
Endpoints:         10.244.0.3:55678
Session Affinity:  None
Events:            <none>

$ kubectl run busybox --rm -ti --namespace=default --image=busybox:1.28 --labels="app.kubernetes.io/instance=my-tempo,app.kubernetes.io/name=tempo" -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # wget -O - http://my-tempo:3100
Connecting to my-tempo:3100 (10.102.51.181:3100)
wget: can't connect to remote host (10.102.51.181): Connection refused

I've been trying to figure out this Connection refused error though I'm trying to connect using valid labels and namespace can you please look into this any suggestions would help!

[Sheikh-Abubaker]

Tried again with different method this time

$ helm install my-tempo ./tempo-1.8.0.tgz
NAME: my-tempo
LAST DEPLOYED: Fri Apr  5 01:02:10 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

$ kubectl get all

NAME                         READY   STATUS    RESTARTS   AGE
pod/my-tempo-0               1/1     Running   0          6s
pod/nginx-7854ff8877-4jdnr   1/1     Running   0          32m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
             AGE
service/kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP
             24h
service/my-tempo     ClusterIP   10.110.185.197   <none>        3100/TCP,6831/UDP,6832/UDP,14268/TCP,14250/TCP,9411/TCP,55680/TCP,55681/TCP,4317/TCP,4318/TCP,55678/TCP   6s
service/nginx        ClusterIP   10.111.118.2     <none>        80/TCP
             32m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           32m

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7854ff8877   1         1         1       32m

NAME                        READY   AGE
statefulset.apps/my-tempo   1/1     6s

Trying connecting to my-tempo service from nginx pod without appropriate labes

$ kubectl get pods -n default --show-labels
NAME                     READY   STATUS    RESTARTS   AGE    LABELS
my-tempo-0               1/1     Running   0          7m7s   app.kubernetes.io/instance=my-tempo,app.kubernetes.io/name=tempo,apps.kubernetes.io/pod-index=0,controller-revision-hash=my-tempo-958864bc7,statefulset.kubernetes.io/pod-name=my-tempo-0
nginx-7854ff8877-4jdnr   1/1     Running   0          39m    app=nginx,pod-template-hash=7854ff8877


$ kubectl exec nginx-7854ff8877-4jdnr -n default -- nc -vz my-tempo 3100
nc: connect to my-tempo (10.110.185.197) port 3100 (tcp) failed: Connection timed out
command terminated with exit code 1

Connecting to my-tempo service again from nginx pod but this time with appropriate labels

$ kubectl describe pod nginx-7854ff8877-4jdnr
Name:             nginx-7854ff8877-4jdnr
Namespace:        default
Priority:         0
Service Account:  default
Node:             multinode-demo-m02/192.168.58.3
Start Time:       Fri, 05 Apr 2024 00:29:41 +0530
Labels:           app=nginx
                  app.kubernetes.io/instance=my-tempo
                  app.kubernetes.io/name=tempo
                  app.kubernetes.io/version=2.3.1
                  helm.sh/chart=tempo-1.8.0
                  my-tempo-client=true
                  pod-template-hash=7854ff8877
                  role=read
Annotations:      cni.projectcalico.org/containerID: 8c48ed2c4c4656d46334c7938f43c0b45520ed06f2d209b832d1dcace821ae4f
                  cni.projectcalico.org/podIP: 10.244.239.11/32
                  cni.projectcalico.org/podIPs: 10.244.239.11/32
Status:           Running
IP:               10.244.239.11
IPs:
  IP:           10.244.239.11
Controlled By:  ReplicaSet/nginx-7854ff8877
Containers:
  nginx:
    Container ID:   docker://4461cd0b346eec18d0b05e3bea8e932147c1850eb45674d1def0282802246d41
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:6db391d1c0cfb30588ba0bf72ea999404f2764febf0f1f196acd5867ac7efa7e
    
$ kubectl exec nginx-7854ff8877-4jdnr -n default -- nc -vz my-tempo 3100
Connection to my-tempo (10.110.185.197) 3100 port [tcp/*] succeeded!

This PR is all set to be merged now.

[Sheikh-Abubaker]

@hkailantzis I wanted to ask you one thing in values.yaml networkPolicy.allowExternal is set to true, in this case it allows pods to connect to service even if the pod doesn't have required labels requiring only correct destination port, but if it is set to false in that case it only allows pods with appropriate labels to connect to service, what do you suggest should networkPolicy.allowExternal default to true or false ??

[hkailantzis]

hi @Sheikh-Abubaker , I would suggest to set it true, as the main grafana chart is setting it. Thanks! Do you need still help with testing this PR ?

[Sheikh-Abubaker]

hi @Sheikh-Abubaker , I would suggest to set it true, as the main grafana chart is setting it. |

alright 👍.

Thanks! Do you need still help with testing this PR ?

Thanks for asking but I've already tested it with another method and it works fine as expected, you can checkout the above updated comments where I tested it using nginx pod.

As of now this PR is all set to be merged.