Infrastructure Monitoring

Lenses works with any version of Apache Kafka 1.0 or later. All services: Kafka Brokers, Zookeepers, Schema Registries, Kafka Connect Clusters are automatically monitored.

Kafka services

  1. From the Header Bar Menu, go to the Dashboard panel.
  2. On the side navigation, select Services under the Monitor section.
Kafka Monitoring

Enabling JMX is recommended for advanced monitoring.

Kafka monitoring

Lenses monitors (by default every 10 seconds) your entire streaming data platform infrastructure and has the following alert rules built-in:

RuleThis rule fires when
Kafka broker is downA Kafka broker from the cluster is not healthy
Zookeeper node is downA Zookeeper node is not healthy
Connect Worker is downA Kafka Connect worker node is not healthy
Schema Registry is downA Schema Registry instance is not healthy
Under replicated partitionsThe Kafka cluster has 1 or more under-replicated partitions
Partitions offlineThe Kafka cluster has 1 or more partitions offline (partitions without an active leader)
Active ControllerThe Kafka cluster has 0 or more than 1 active controllers
Multiple Broker versionsThe Kafka cluster is under a version upgrade, and not all brokers have been upgraded
File-open descriptors on BrokersA Kafka broker has an alarming number of file-open descriptors. When the operating system is exceeding 90% of the available open file descriptors
Average % the request handler is idleThe average fraction of time the request handler threads are idle is dangerously low. The alert is HIGH when the value is smaller than 10%, and CRITICAL when it is smaller than 2%.
Fetch requests failureFetch requests are failing. If the rate of failures per second is > 10% the alert level is set to CRITICAL, otherwise it is set to HIGH.
Produce requests failureProducer requests are failing. When the value is > 10% the alert level is set to CRITICAL, otherwise it is set to HIGH.
Broker disk usageA Kafka broker’s disk usage is greater than the cluster average. The build-in threshold is 1 GByte.
Leader imbalanceA Kafka broker has more leader replicas than the average broker in the cluster.

Broker decommission

If you change your Kafka cluster size, or replace an existing Kafka broker with another, Lenses will raise an active alert as it will detect that a broker of your Kafka cluster is no longer available. If the Kafka broker has been intentionally removed, then decommission it:

  1. Navigate to Services.
  2. Select the broker and click on the actions in the options menu and click on the Decommission option.

Kafka Consumer monitoring

Downstream failures of streaming data pipelines can be prevented by monitoring Kafka Consumers

Kafka Producer monitoring

Upstream failures of streaming data pipelines can be prevented by monitoring Kafka Producers

Other services

Lenses monitors the current metrics for each service providing like this a current health check state. If you wish to have historical data about the services, see the Grafana and Prometheus for Kafka section.