Monitoring tools for Apache Kafka ecosystem (PoC)

The goal of this project is to demonstrate different approaches how to monitor Apache Kafka ecosystem.

Caution

The examples in this repository are only for testing purposes only.

Getting Started

This repository contains multiple docker-compose stacks which includes all necessary monitoring infrastructure setup.

Each sample demonstrates how to collect and monitor useful metrics and visualize records using various types of tools available online.

Docker compose stacks can be found under docker directory.

Build and Install

Please refer to the individual readme files for further instructions how to build/install selected monitoring demo.

Available demos:

• Confluent Platform Community Edition

  Kafka Overview	Kafka Topics	Kafka Connect Overview

  Schema Registry Overview	ksqldb Overview	Zookeeper Overview

  Kafka Lag Exporter

• Burrow

  Burrow Dashboard - Home Page	BurrowUI - Home Page	BurrowUI - Total Lag Graph

• Kafka Lag Exporter

  Grafana Kafka Lag Exporter

• KMinion

  Cluster Dashboard	Consumer Group Dashboard	Topic Dashboard

• Kowl - Apache Kafka Web UI

  Kowl - Topics Page	Kowl - Consumer Groups

• AKHQ (KafkaHQ) - Apache Kafka Web UI

  KafkaHQ

References

• Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana

• Prometheus

• ksqlDB

• Schema Registry

• Kafka REST Proxy

• Kafka Connect

• Linkedin/Burrow

• GeneralMills/BurrowUI

• Joway/BurrowDashboard

• Lightbend Kafka Lag Exporter

• Kafka Lag Exporter Grafana Dashboard

• tchiotludo/akhq

• KMinion - Prometheus Exporter for Apache Kafka

• KMinion Cluster Dashboard

• KMinion Consumer Group Dashboard

• KMinion Topic Dashboard

• cloudhut/kowl

Additional Links

• What is the State of my Kafka Streams Application? Unleashing Metrics
• Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients

License

Distributed under the MIT License. See LICENSE for more information.