Can't get opentelemetry logs in skywalking in k8s #12050

tvildo · 2024-03-25T18:44:30Z

tvildo
Mar 25, 2024

Hi, I deployed skywalking 9.7.0 in kubernetes, I use elasticsearch as database, based on skywalking logs it seems it is up and running. But In elastic tables are empty. No logs are ingested by skywalking but they are sent from opentelemetry pod and I see no error as well.

Am I missing something, What can I do to make it work ?

Skywalking config:

{
	"SW_CLUSTER": "kubernetes",
	"SW_CLUSTER_K8S_LABEL": "app=skywalking,component=oap",
	"SW_CLUSTER_K8S_NAMESPACE": "dev-david-monitoring",
	"SW_ES_PASSWORD": "Q9XKEp14wwwwwDeAvvOQJTEST",
	"SW_ES_USER": "admin",
	"SW_OTEL_RECEIVER": "default",
	"SW_OTEL_RECEIVER_ENABLED_HANDLERS": "otlp-metrics,otlp-logs,otlp-traces",
	"SW_STORAGE": "elasticsearch",
	"SW_STORAGE_ES_CLUSTER_NODES": "elasticsearch-es-http.dev-david-monitoring.svc:9200",
	"SW_STORAGE_ES_HTTP_PROTOCOL": "https",
	"SW_STORAGE_ES_INDEX_REPLICAS_NUMBER": "0",
	"SW_STORAGE_ES_INDEX_SHARDS_NUMBER": "1",
	"SW_STORAGE_ES_SSL_JKS_PASS": "jM22ORGwojwjjBllYS2lTEST",
	"SW_STORAGE_ES_SSL_JKS_PATH": "/jks/certs.jks"
}

Following is log from skywalking:
skywalking_log.txt

This is what is in elastic:

This is my opentelemetry helm chart config related to skywalking:

config:
  exporters:
    otlp:
      endpoint: skywalking.dev-david-monitoring.svc.cluster.local:11800
      tls:
        insecure: true
    debug: {}
    logging: {}
  extensions:
    health_check:
      endpoint: ${env:MY_POD_IP}:13133
    memory_ballast: {}
  processors:
    batch: {}
    memory_limiter: null
  receivers:
    jaeger:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:14250
        thrift_http:
          endpoint: ${env:MY_POD_IP}:14268
        thrift_compact:
          endpoint: ${env:MY_POD_IP}:6831
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318
    prometheus:
      config:
        scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
              - targets:
                  - ${env:MY_POD_IP}:8888
    zipkin:
      endpoint: ${env:MY_POD_IP}:9411
  service:
    telemetry:
      metrics:
        address: ${env:MY_POD_IP}:8888
    extensions:
      - health_check
      - memory_ballast
    pipelines:
      logs:
        exporters:
          - debug
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
      metrics:
        exporters:
          - debug
          - otlp
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
          - prometheus
      traces:
        exporters:
          - debug
          - otlp
        processors:
          - memory_limiter
          - batch
        receivers:
          - otlp
          - jaeger
          - zipkin