Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k8sattributes processor doesnt work with loki.source.kubernetes #780

Open
lieberlois opened this issue May 7, 2024 · 0 comments
Open

k8sattributes processor doesnt work with loki.source.kubernetes #780

lieberlois opened this issue May 7, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@lieberlois
Copy link

lieberlois commented May 7, 2024 •
edited

What's wrong?

Im trying to setup the otelcol.processor.k8sattributes with Grafana Loki. Unfortunately, the k8sattributes processor seems to not be capable of extracting the k8s metadata from the loki.source.kubernetes module. The module outputs no new labels. My assumption is that it doesnt work because loki.source.kubernetes does not expose k8s.pod.ip, which the k8sattributes processor uses for pod matching

This is my config:

        logging {
          level  = "info"
          format = "logfmt"
        }

        discovery.kubernetes "pods" {
          role = "pod"

          namespaces {
            names = ["default"]
          }
        }

        loki.source.kubernetes "poc" {
          targets    = discovery.kubernetes.pods.targets
          forward_to = [otelcol.receiver.loki.default.receiver]
        }

        otelcol.receiver.loki "default" {
          output {
            logs = [otelcol.processor.k8sattributes.default.input]
          }
        }

        otelcol.receiver.otlp "default" {
          http {}
          grpc {}

          output {
            logs    = [otelcol.processor.k8sattributes.default.input]
          }
        }

        otelcol.processor.k8sattributes "default" {
          extract {
            metadata = [
              "k8s.namespace.name",
              "k8s.deployment.name",
              "k8s.statefulset.name",
              "k8s.daemonset.name",
              "k8s.cronjob.name",
              "k8s.job.name",
              "k8s.node.name",
              "k8s.pod.name",
              "k8s.pod.uid",
              "k8s.pod.start_time",
            ]
          }

          output {
            logs    = [otelcol.processor.batch.default.input]
          }
        }

        otelcol.processor.batch "default" {
          output {
            logs    = [otelcol.exporter.loki.default.input]
          }
        }

        otelcol.exporter.loki "default" {
          forward_to = [loki.echo.debug.receiver]
        }

        loki.echo "debug" {}

The pipeline thus looks like this:

image

The debug component however only writes this to the console (there is a ubuntu container in the default namespace that writes hello world to stdout):

ts=2024-05-07T13:30:29.836589274Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"hello world\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/ubuntu:ubuntu\", job=\"loki.source.kubernetes.poc\"}"

Steps to reproduce

  • Debug Alloy with config above
  • Run ubuntu container like this: kubectl run -n default ubuntu --image=ubuntu:latest -- sh -c 'while true; do echo "hello world"; sleep 1; done'
  • Attach to logs using kubectl logs alloy-infra-0 -f

System information

Linux, Azure AKS

Software version

Grafana Alloy v1.0.0

Configuration

logging {
  level  = "info"
  format = "logfmt"
}

discovery.kubernetes "pods" {
  role = "pod"

  namespaces {
    names = ["default"]
  }
}

loki.source.kubernetes "poc" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    logs    = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    metadata = [
      "k8s.namespace.name",
      "k8s.deployment.name",
      "k8s.statefulset.name",
      "k8s.daemonset.name",
      "k8s.cronjob.name",
      "k8s.job.name",
      "k8s.node.name",
      "k8s.pod.name",
      "k8s.pod.uid",
      "k8s.pod.start_time",
    ]
  }

  output {
    logs    = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    logs    = [otelcol.exporter.loki.default.input]
  }
}

otelcol.exporter.loki "default" {
  forward_to = [loki.echo.debug.receiver]
}

loki.echo "debug" {}

Logs

ts=2024-05-07T13:36:17.89708731Z level=info "boringcrypto enabled"=false
ts=2024-05-07T13:36:17.89712761Z level=info msg="running usage stats reporter"
ts=2024-05-07T13:36:17.89713361Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b
ts=2024-05-07T13:36:17.89714471Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=loki.echo.debug duration=34.7µs
ts=2024-05-07T13:36:17.89715211Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.exporter.loki.default duration=374.898µs
ts=2024-05-07T13:36:17.89715801Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.processor.batch.default duration=335.499µs
ts=2024-05-07T13:36:17.89716371Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.processor.k8sattributes.default duration=175.199µs
ts=2024-05-07T13:36:17.89716911Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.receiver.loki.default duration=28.3µs
ts=2024-05-07T13:36:17.89717491Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otelcol.receiver.otlp.default duration=161.699µs
ts=2024-05-07T13:36:17.89718051Z level=info msg="Using pod service account via in-cluster config" component_path=/ component_id=discovery.kubernetes.pods
ts=2024-05-07T13:36:17.89718521Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=discovery.kubernetes.pods duration=785.896µs
ts=2024-05-07T13:36:17.897191009Z level=info msg="Using pod service account via in-cluster config" component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:17.897196309Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=loki.source.kubernetes.poc duration=575.498µs
ts=2024-05-07T13:36:17.897206509Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=remotecfg duration=97.499µs
ts=2024-05-07T13:36:17.897216709Z level=info msg="applying non-TLS config to HTTP server" service=http
ts=2024-05-07T13:36:17.897223609Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=http duration=17.7µs
ts=2024-05-07T13:36:17.897234109Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=ui duration=10.4µs
ts=2024-05-07T13:36:17.897248009Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=cluster duration=9.5µs
ts=2024-05-07T13:36:17.897258909Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=otel duration=15.3µs
ts=2024-05-07T13:36:17.897267609Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=tracing duration=15.6µs
ts=2024-05-07T13:36:17.897279509Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=logging duration=227.899µs
ts=2024-05-07T13:36:17.897329409Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b node_id=labelstore duration=27.8µs
ts=2024-05-07T13:36:17.897359609Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=e32481e9e3bb8187e9ea80bfdc70f68b duration=3.333686ms
ts=2024-05-07T13:36:17.897866607Z level=info msg="scheduling loaded components and services"
ts=2024-05-07T13:36:17.898087006Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.kubernetes.poc duration=172.1µs
ts=2024-05-07T13:36:17.898423404Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning." component_path=/ component_id=otelcol.receiver.otlp.default documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks "feature gate ID"=component.UseLocalHostAsDefaultHost
ts=2024-05-07T13:36:17.898784703Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345
ts=2024-05-07T13:36:17.898901302Z level=info msg="k8s filtering" component_path=/ component_id=otelcol.processor.k8sattributes.default labelSelector="" fieldSelector=""
ts=2024-05-07T13:36:17.899223101Z level=info msg="Starting GRPC server" component_path=/ component_id=otelcol.receiver.otlp.default endpoint=0.0.0.0:4317
ts=2024-05-07T13:36:17.899540999Z level=warn msg="Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks. Enable the feature gate to change the default and remove this warning." component_path=/ component_id=otelcol.receiver.otlp.default documentation=https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks "feature gate ID"=component.UseLocalHostAsDefaultHost
ts=2024-05-07T13:36:17.899753198Z level=info msg="Starting HTTP server" component_path=/ component_id=otelcol.receiver.otlp.default endpoint=0.0.0.0:4318
ts=2024-05-07T13:36:17.899984897Z level=info msg="starting cluster node" peers="" advertise_addr=127.0.0.1:12345
ts=2024-05-07T13:36:17.900304496Z level=info msg="peers changed" new_peers=alloy-infra-0
ts=2024-05-07T13:36:17.900501595Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.processor.k8sattributes.default duration=2.37169ms
ts=2024-05-07T13:36:17.900473095Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.processor.batch.default duration=2.532889ms
ts=2024-05-07T13:36:17.900646094Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.receiver.loki.default duration=98.499µs
ts=2024-05-07T13:36:17.900743894Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.receiver.otlp.default duration=2.829988ms
ts=2024-05-07T13:36:17.900768394Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.exporter.loki.default duration=2.857088ms
ts=2024-05-07T13:36:22.89945045Z level=info msg="Using pod service account via in-cluster config" component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900047648Z level=info msg="tailer running" target=default/nginx-proxy-f95fdf4b-68mpx:nginx component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900148447Z level=info msg="tailer running" target=default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900283747Z level=info msg="tailer running" target=default/ubuntu:ubuntu component_path=/ component_id=loki.source.kubernetes.poc
ts=2024-05-07T13:36:22.900232747Z level=info msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.kubernetes.poc duration=1.189896ms
ts=2024-05-07T13:36:22.91724249Z level=info msg="opened log stream" target=default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter component_path=/ component_id=loki.source.kubernetes.poc "start time"=1970-01-01T00:00:00.000Z
ts=2024-05-07T13:36:22.917795288Z level=info msg="opened log stream" target=default/ubuntu:ubuntu component_path=/ component_id=loki.source.kubernetes.poc "start time"=1970-01-01T00:00:00.000Z
ts=2024-05-07T13:36:22.919356983Z level=info msg="opened log stream" target=default/nginx-proxy-f95fdf4b-68mpx:nginx component_path=/ component_id=loki.source.kubernetes.poc "start time"=1970-01-01T00:00:00.000Z
ts=2024-05-07T13:36:23.118116118Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.758Z caller=main.go:87 level=info msg=\\\"Starting blackbox_exporter\\\" version=\\\"(version=0.25.0, branch=HEAD, revision=ef3ff4fef195333fb8ee0039fb487b2f5007908f)\\\"\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118209518Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.758Z caller=main.go:88 level=info build_context=\\\"(go=go1.22.2, platform=linux/amd64, user=root@47d5b0d99f18, date=20240409-12:58:39, tags=unknown)\\\"\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118242918Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.759Z caller=main.go:100 level=info msg=\\\"Loaded config file\\\"\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118258318Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.760Z caller=tls_config.go:313 level=info msg=\\\"Listening on\\\" address=[::]:9115\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118277118Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"ts=2024-05-03T12:33:09.760Z caller=tls_config.go:316 level=info msg=\\\"TLS is disabled.\\\" http2=false address=[::]:9115\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/blackbox-exporter-prometheus-blackbox-exporter-6b8d77db6b-gjk9k:blackbox-exporter\", job=\"loki.source.kubernetes.poc\"}"
ts=2024-05-07T13:36:23.118311418Z level=info component_path=/ component_id=loki.echo.debug receiver=loki.echo.debug entry="{\"body\":\"hello world\\n\"}" labels="{exporter=\"OTLP\", instance=\"default/ubuntu:ubuntu\", job=\"loki.source.kubernetes.poc\"}"
@lieberlois lieberlois added the bug Something isn't working label May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lieberlois