Monitor Kafka Consumers
Lenses monitors all your Kafka Consumers. It utilizes native Kafka clients to connect to Kafka and calculates the required metrics such as the Lag per partition.
__consumer_offsetsso they know until which point they have consumed.
Navigate to consumers
- From the Header Bar Menu, go to the Dashboard panel.
- On the side navigation, select Consumers under the Monitor section.
When the red bar appears under the consumer lag column, it indicates the lag range (maximum and minimum).
Consumer details per instance
You can drill down to a specific consumer group, and see the details for each of its instances by clicking the consumer group name.
From the detail page, you can see how the topic partitions have been distributed to each consumer instance as well as the lag for each topic-partition.
- The coordinator ID refers to the broker who is the coordinator for this group
- Log End Offset is the offset where the topic is currently at
- Current Offset is the offset until which the consumer has read the message.
Add alerts to consumers
Alert Rules can be set for consumers to monitor consumer lags:
- Click on the View Alerts button to see existing rules set for this consumer.
- You will be navigated to the Admin > Alert Rules > Custom Rules page.
- Click on the “New Rule” button and fill in the details.
The alert will be triggered when the consumer lag exceeds the desired value.
Integrate consumer lag
Consumer Lag is calculated and available in a Lenses API:
You can then hook this to your Grafana:
Permissions on consumers
Application permissions govern consumers. This means that we can access the nodes related to the namespaces you have access too. See more on Application Permissions
Specifically, to Add Custom Rules to Consumer Groups, you need to have
on top of the
ManageAlertRules that are required for the rest of the Alerts.
If you try to Add a Custom Rule with
ViewKafkaConsumers, you will be treated by a PopUp Message on the top right-hand side of the screen, informing you that the list of Kafka Consumers cannot be fetched.