too many open files

[gricuk]

We're gettingtoo many open fileserror in few clusters with image ghcr.io/banzaicloud/fluentd:v1.14.5-alpine-1 and higher.

It happens randomly and I can't provide any conditions, but could you please check the following PR fluent/fluentd#3844. It was fixed in 1.15.2 release.

Now we have downgraded fluentd to ghcr.io/banzaicloud/fluentd:v1.14.4-alpine-2 and all is okay with this version (but I'm not sure because as I mentioned above it happens randomly and not all clusters have this error)

Environment details:

• Kubernetes version v1.21.12+rke2r2:
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): RKE2
• logging-operator version (e.g. 2.1.1): logging-operator-3.17.9
• Install method (e.g. helm or static manifests): helm

/kind bug

[1it]

It could be caused by chunk_limit_size: 8MB, which is set here. This actually a nonsense, it should be at least 256MB (Fluentd default). 8MB is a default value for in memory buffer, not for file based.
I've had about 32k buffer files in one instance because of that and constant too many open files errors.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

Thanks for reporting this and sorry for the late reply. If I remember correctly previously fluentd used 256M for memory buffers by default (or at least it was the case in some old versions), which caused OOM issues, that was the reason why we did this:
#751

I beleive we can remove this explicit default and rely on the fluentd settings if we can verify our supported fluentd images now does it correctly.

Scheduling this to the next minor release.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

According to the docs, the chunk_limit_size is now smarter as it uses 8MB for memory and 256MB for file buffers by default:
https://docs.fluentd.org/configuration/buffer-section#buffering-parameters

We can remove our default setting to rely on fluentd's settings.

[giakinh0823]

This is my issue. I have 2 node in k8s.

• Node1: master + worker: This is node error too many open files
• Node2: worker: This is node running success with deamonset
  Please help me. Thank you.

Here are my settings:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: default-logging-simple
  namespace: logging
spec:
  fluentd:
    image:
      pullPolicy: IfNotPresent
      repository: rancher/banzaicloud-fluentd
      tag: v1.11.5-alpine-9
    configReloaderImage:
      repository: jimmidyson/configmap-reload 
      tag: latest  # Lỗi thì thay đổi version
      pullPolicy: IfNotPresent
    security:
      roleBasedAccessControlCreate: true
    resources:
      requests:
        cpu: 10m
        memory: 100M
      limits:
        cpu: 1
        memory: 3Gi
    bufferStorageVolume:
      pvc:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 40Gi
          storageClassName: longhorn
          volumeMode: Filesystem
    scaling:
      drain:
        enabled: true
      replicas: 1

  fluentbit:
    image:
      pullPolicy: Always
      repository: fluent/fluent-bit
      tag: 2.1.8
    security:
      roleBasedAccessControlCreate: true
    resources:
      requests:
        cpu: 10m
        memory: 100M
      limits:
        cpu: 1
        memory: 2Gi
    positiondb:
      hostPath:
        path: ""
    bufferStorageVolume:
      hostPath:
        path: ""
  controlNamespace: logging
  watchNamespaces: ["production"]

[giakinh0823]

Thanks but I fixed it.

I created another daemonset and set up the information below

sysctl -w fs.inotify.max_queued_events=524288
sysctl -w fs.inotify.max_user_instances=16383
sysctl -w fs.inotify.max_user_watches=524288

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    name: es-ds
  name: es-ds
  namespace: logging
spec:
  selector:
    matchLabels:
      name: es-ds
  template:
    metadata:
      labels:
        name: es-ds
    spec:
      containers:
      - name: es-ds
        image: gcr.io/google-containers/startup-script:v1
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        env:
        - name: STARTUP_SCRIPT
          value: |
            #! /bin/bash
            sysctl -w vm.max_map_count=262144
            sysctl -w fs.inotify.max_queued_events=524288
            sysctl -w fs.inotify.max_user_instances=16383
            sysctl -w  fs.inotify.max_user_watches=524288
            echo "done"

[pepov]

@giakinh0823 These last two comments are unrelated to the original issue topic (fluentd vs fluentbit), could you please create a separate issue for that?