Kafka Topics

Lenses enables users to create, edit and delete Kafka topics to be self-serviced. A data centric security model (based on Namespaces) can be used to fine tune the levels of access per user group, and apply RBAC (Role Based Access Control). All CRUD actions (Create, Update, Delete) are audited for compliance. Every time an action is taken, the event is audited for compliance.

Kafka topic concepts

Topics in Apache Kafka consist of three basic fields and the topic configurations:

• Name, the name of the topic - needs to be unique.
• Partitions, the number of partitions the topic will consist of.
• Replication, the replication factor for each partition across the cluster.

All topic configurations have default values. Important configs include: Cleanup policy, Retention size and Retention time

Kafka topic create

To create a new topic, access the Data Explore screen and click on the New Topic button:

When creating a topic, the topic settings apply.

Kafka topic list

Lenses features a real-time Data Catalog that continuously learns about the data in the event streaming data platform. You can search, based on topic names, topic fields, and topic descriptions to find the relevant Kafka topic.

The Kafka topic list displays only the topics that a particular user is allowed to see, and a data centric security model, means that you can have multiple tenants on the same Kafka cluster.

Kafka Topic indicators

The indicators next to the topic name display additional information:

• C, highlights when a topic is compacted.
• S, appears when a topic is a system topic.
• d, appears when a topic is marked for deletion.

Systems datasets are used to indicate which topics have been automatically created by the system. By default they are excluded from the view, but you can choose to view them by ticking the box on the filter list.

Kafka topic partitions

Lenses adds observability and health monitoring of Kafka partitions. The visual map displays the number of messages that are currently stored per topic-partition. Using this map, a user can quickly identify if a partition has been over-utilized and requires to rearrange the partitioning strategy.

Navigate to the Topic and select the Partitions tab.

At the bottom of the screen one can see the technical information of the partitions of the topic, including the replication status. Partition information includes:

• Leader broker per partition.
• Records per partition.
• Data size per partition.
• Max and min offset per partition.
• Whether replicas are in-sync.

Increasing Kafka partitions

Lenses provides the ability to increase the number of partitions per topic. To increase partitions of a topic:

1. From the Header Bar Menu, go to the Dashboard panel
2. On the side navigation, select Explore
3. Select the topic you wish to edit
4. Click on the Actions menu on the top right of your screen
5. Select the Add Partitions option

Fill in the new amount of partitions on the modal. Make sure you understand the side effects and confirm by checking the related checkbox underneath.

Kafka topic consumers

To view the Kafka consumers for a specific topic, navigate to the Topic and select the Consumers tab.

For each Kafka consumer group, the following details are available:

• The number of partitions they are consuming from.
• The lag offset (the messages that have not been consumed yet).
• The health state of the Consumer group.

You can drill further into the consumer by clicking on the consumer name.

To set up notifications and alerts on kafka consumers see the relevant monitoring section.

Kafka topic metrics

The metrics tab provides detailed information about producers of the Kafka topic.

To set up notifications and alerts on kafka producers see the relevant monitoring section.

Kafka topic config

A Kafka Topic can be configured via a Key-Value pair.

To Edit and View the Topic Configuration:

1. From the Header Bar Menu, go to the Dashboard panel.
2. On the side navigation, select Topics under the data section.
3. Click on the Topic Name from the list and navigate to the Config tab.

Kafka topic delete

To Delete a Topic

1. From the Header Bar Menu, go to the Dashboard panel.
2. On the side navigation, select Explore under the data section.
3. Click the trash icon on the topic you want to be deleted.

Kafka topic schemas

Lenses keeps a schema and data types for each topic. When a topic is detected for the first time, Lenses tries to identify the type for the Key and the Value and work out the schema. If the auto-detection is incorrect, you can override the format.

To change the Schema Type:

1. From the Header Bar Menu, go to the Dashboard panel
2. On the side navigation, select Explore
3. Select the topic you wish to edit
4. Click on the Actions menu on the top right of your screen
5. Select the Change Type (Key, Value) option

6. Change the Key or Value types.

To change the schema:

If schema registry for Avro types exists, Lenses communicates directly with the schema registry to get the Avro schemas.

If the type is not Avro, Lenses tries to detect the schema for the Key and Value. Lenses stores this schema in its storage.

To edit the Lenses Schema:

1. From the Header Bar Menu, go to the Dashboard panel
2. On the side navigation, select Explore
3. Select the topic you wish to edit
4. Click on the Actions menu on the top right of your screen
5. Select the Edit Schema option

6. Edit the schema as you see fit

Kafka topic SQL

Lenses SQL supports managing Topics via SQL statements, CREATE TABLE, DESCRIBE TABLE, DROP TABLE etc. Here is an example:

CREATE TABLE customer (id string, address.line string, address.city string, address.postcode int, email string)
FORMAT (string, json)
PROPERTIES (partitions=1, compacted=true)

From the topic detail page, you can explore data in table browsing mode, or sample a data via live stream.

See more on SQL docs

Kafka topic GitOps

Lenses CLI also supports all managing Topics actions. Here is an example:

$ cat topicName.yml

name: topicName
replication: 2
partitions: 4
configs:
  max.message.bytes: "1000010"

$ lenses-cli topic create ./topicName.yml

Lenses CLI can also be used to promote Topics from one cluster to another, using YAML config files and GitOps:

$ lenses-cli topic update ./topic.yaml

For further automations refer to the API and CLI docs.

Listing empty topics

Empty topics are defined as topics with no records reported by the brokers.

To list empty topics you can use either one of these methods:

• the Lenses API - using the listDatasets endpoint with the recordCount filter set to empty. For more details see the Datasets API Documentation .
• the Lenses CLI - using the dataset list --records empty command. For more details see the Datasets CLI Documentation .