Problem with dns resolution on a fluentd configCheck pod

[Earsil]

Hi,
I am running kube-logging on RKE(Rancher cluster) and I have a following issue after applying a new cluster output targeting external fluentd service. The config-check pod fails with a following error

2023-10-20 14:44:07 +0000 [info]: [flow:test:test-log:clusteroutput:cattle-logging-system:logging-system] adding forwarding server 'myendpoint.com:12002' host="myendpoint.com" port=12002 weight=60 plugin_id="flow:test:test-log:clusteroutput:cattle-logging-system:logging-system"
/usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_forward.rb:745:in `getaddrinfo': getaddrinfo: Name does not resolve (SocketError) 

My clusteroutput looks like this:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  creationTimestamp: "2023-10-20T14:42:57Z"
  generation: 2
  name: logging-system
  namespace: cattle-logging-system
  resourceVersion: "517633005"
  uid: bc6abcd2-baa4-49bd-b3eb-fd4b16466b0e
spec:
  forward:
    servers:
    - host: myendpoint.com
      port: 12002
    transport: tls
status:
  active: true

This domain exists (myendpoint.com is just an example), and I know what is the reason for this failure. Fluentd image is based on AlpineOS, which has its own implementation of DNS client (non glibc), which does not work very well on Kubernetes. (It returns nx domain, because it somehow tries to check a lot of search domains - or sth similar to this - did not really investigate this) There are two workarounds for this issue.

• I can add . at the end of a target url e.g. myendpoint.com. However, it is problematic for me when I target this external fluentd. Hostname in a request has to exact like myendpoint.com, so there cannot be a dot at the end. (Probably LB requires this)
• Second workaround is to add following config to a pod:

    dnsConfig:
     options:
     - name: ndots
       value: "1"

I added this to logging API object for fluentd:

kind: Logging
metadata:
  annotations:
  creationTimestamp: "2023-10-17T12:31:16Z"
  generation: 3
  name: logging-config
spec:
  controlNamespace: cattle-logging-system
  fluentbit:
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "1"
        memory: 1Gi
  fluentd:
    dnsConfig:
      options:
      - name: ndots
        value: "1"
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "1"
        memory: 1Gi

It is applied to fluentd stateful set. However, it is not applied to config-check pods. I checked documentation and I only see that I can add configCheckResources and configCheckAnnotations. But there is nothing about dnsConfig for config check pod.

Did I miss something? Or maybe there is a fluentd image not based on Alpine somewhere to use ?

FYI, I ran kube-logging 4.2.3 helm chart version of kube-logging and RKE v1.24.10

[Earsil]

Ok, I fixed this by patching logging-operator file ./pkg/resources/fluentd/appconfigmap.go and rebuilding it

I added this
DNSConfig: r.Logging.Spec.FluentdSpec.DNSConfig,
to following func

func (r *Reconciler) newCheckPod(hashKey string) *corev1.Pod {

	volumes := r.volumesCheckPod(hashKey)
	container := r.containerCheckPod(hashKey)
	initContainer := r.initContainerCheckPod()

	pod := &corev1.Pod{
		ObjectMeta: r.FluentdObjectMeta(fmt.Sprintf("fluentd-configcheck-%s", hashKey), ComponentConfigCheck),
		Spec: corev1.PodSpec{
			RestartPolicy:      corev1.RestartPolicyNever,
			ServiceAccountName: r.getServiceAccount(),
			NodeSelector:       r.Logging.Spec.FluentdSpec.NodeSelector,
			Tolerations:        r.Logging.Spec.FluentdSpec.Tolerations,
			Affinity:           r.Logging.Spec.FluentdSpec.Affinity,
			PriorityClassName:  r.Logging.Spec.FluentdSpec.PodPriorityClassName,
			SecurityContext: &corev1.PodSecurityContext{
				RunAsNonRoot: r.Logging.Spec.FluentdSpec.Security.PodSecurityContext.RunAsNonRoot,
				FSGroup:      r.Logging.Spec.FluentdSpec.Security.PodSecurityContext.FSGroup,
				RunAsUser:    r.Logging.Spec.FluentdSpec.Security.PodSecurityContext.RunAsUser,
				RunAsGroup:   r.Logging.Spec.FluentdSpec.Security.PodSecurityContext.RunAsGroup,
			},
			Volumes:          volumes,
			ImagePullSecrets: r.Logging.Spec.FluentdSpec.Image.ImagePullSecrets,
			InitContainers:   initContainer,
			Containers:       container,
			DNSConfig:        r.Logging.Spec.FluentdSpec.DNSConfig,
		},
	}

It looks that it works, but I am not a Go expert at all ;) Maybe it would be nice to add it to master branch and a newest version.

[pepov]

Glad you could solve it in the end! Feel free to open a PR if you like, the solution is nothing more complex than what you did there. If not, I'm happy to convert this to an issue and fix it in the next upcoming release.

[Earsil]

If you could do it, that would be great