Monitor Kafka Consumers

Lenses monitors all your Kafka Consumers. It utilizes native Kafka clients to connect to Kafka and calculates the required metrics such as the Lag per partition.

Consumer Groups in Kafka are applications that consume data from Kafka topics. Depending on the policy you choose to consume data, your consumers commit their offsets to a system topic called __consumer_offsets so they know until which point they have consumed.
  1. From the Header Bar Menu, go to the Dashboard panel.
  2. On the side navigation, select Consumers under the Monitor section.

Consumer Groups

When the red bar appears under the consumer lag column, it indicates the lag range (maximum and minimum).

Consumer details per instance

You can drill down to a specific consumer group, and see the details for each of its instances by clicking the consumer group name.

From the detail page, you can see how the topic partitions have been distributed to each consumer instance as well as the lag for each topic-partition.

Consumer Groups

  • The coordinator ID refers to the broker who is the coordinator for this group
  • Log End Offset is the offset where the topic is currently at
  • Current Offset is the offset until which the consumer has read the message.
Consumer Groups may consume from multiple topics at the same time. They distribute the load to each instance of the group. Sink Connectors also create consumers, SQL Processors, or any other application consuming data from Kafka topics.

Add alerts to consumers

Alert Rules can be set for consumers to monitor consumer lags:

  1. Click on the View Alerts button to see existing rules set for this consumer.
  2. You will be navigated to the Admin > Alert Rules > Custom Rules page.
  3. Click on the “New Rule” button and fill in the details.
Consumer Groups

The alert will be triggered when the consumer lag exceeds the desired value.

Integrate consumer lag

Consumer Lag is calculated and available in a Lenses API:

<your-host>/metrics

You can then hook this to your Grafana:

Consumer Lag

Permissions on consumers

Application permissions govern consumers. This means that we can access the nodes related to the namespaces you have access too. See more on Application Permissions

Specifically, to Add Custom Rules to Consumer Groups, you need to have ViewKafkaConsumers Permission on top of the ManageAlertRules that are required for the rest of the Alerts.

If you try to Add a Custom Rule with ViewKafkaConsumers, you will be treated by a PopUp Message on the top right-hand side of the screen, informing you that the list of Kafka Consumers cannot be fetched.

No View Kafka Consumers Permission