9. Monitoring Apache Kafka

Consumer lag is the most important Kafka metric to watch. It measures the distance between the latest available offset and the consumer’s committed offset, so rising lag means consumers are falling behind. Monitoring lag at the partition level is essential because one hot partition can hide behind a healthy aggregate number.

Broker health should be tracked through under-replicated partitions and offline partitions. Under-replicated partitions mean fault tolerance is reduced, and offline partitions mean data is unavailable. Producer latency, consumer fetch latency, network traffic per broker, and disk utilization are also important signals.

Kafka exposes metrics through JMX, which can be collected by tools like the JMX Exporter. Prometheus and Grafana are the most common open-source stack for storing and visualizing Kafka metrics. Community Grafana dashboards are often the fastest way to get useful monitoring in place.

Confluent Control Center is a commercial monitoring option with Kafka-specific views for lag, health, connectors, and schema registry activity. CMAK, formerly Kafka Manager, is a useful open-source admin console for topic inspection and partition management. Each tool solves a different part of the operational problem.

Growing consumer lag usually points to slow processing, hot partitions, or frequent rebalances. The first step is to inspect lag per partition, not just in total. If one partition is consistently behind, the issue is usually imbalance rather than total consumer capacity.

High producer latency often indicates broker disk pressure, replication lag, or leadership imbalance. Rebalance storms usually happen when consumers take too long between polls and exceed session limits. Schema deserialization failures usually mean a schema compatibility mistake, which is why registry compatibility rules matter so much.