Kafka topic settings

WHAT YOU'LL LEARN

What are Topic settings and how to use them.
F.A.Q and troubleshooting guides, in case you get stuck somewhere.

Introduction

Topic settings gives the option to define constraints around topic partitions, replication, retention, and naming. To be able to change the rules the user needs to belong to at least one group which grants manage permission for Kafka Settings in the Admin section. They apply when creating new topics or configuring the existing ones through Lenses UI, CLI, and REST API.

Please note that such constrains are specific to Lenses and will be ignored when creating/configuring topics through other tooling.

IMPORTANT INFO

Under certain circumstances, the topic settings configured through Lenses might no longer be applicable. For instance, if you set a min replication factor of 3, but the number of active brokers in your Kafka cluster suddenly drops to 2, your settings will enter into an unapplicable state. When this happens, Lenses will lock all API calls around topic creation and configuration until this inconsistency is addressed. This can be done either by bringing the third broker back up, or by decreasing the minimum replication factor to 2.

CLI

Lenses CLI, can be used to view and update the Topic Settings, as well as import and export to promote settings between different environments.

View

The following command can be used to retrieve the Topics Settings details. Output can be either JSON or YAML.

lenses-cli topic-settings --output="JSON"

Update

Use the following command to update the topic settings policies:

lenses-cli topic-settings update \
    --partitions-min 1 --partitions-max 3 \
    --replication-min 1 --replication-max 2 \
    --retention-size-default -1 --retention-size-max -1 \
    --retention-time-default -1 --retention-time-max -1 \
    --naming-pattern="[a-zA-Z]*" --naming-description="Only letters ([a-zA-Z]*)"

Export/Import

Importing/exporting the topic settings policies can be done via this command:

lenses-cli export topic-settings --dir="DIRECTORY-PATH"
lenses-cli import topic-settings --dir="DIRECTORY-PATH"

UI

The Topic Settings policies can be found under the Admin section of Lenses. The user requires View permission for the Kafka Settings option.

Form Parameter	Description	Required	Notes
Min Partitions	Min number of acceptable partition	YES	Must be more or equal to 1
Max Partitions	Max number of acceptable partition	NO	Must be more or equal to 1
Min Replication	Min number of acceptable replication factor	YES	Must be more or equal to 1
Max Replication	Max number of acceptable replication factor	NO	Must be more or equal to 1
Default RetentionTime	Default time a Topic can retain data	NO	Must be positive or -1(Infinite)
Max RetentionTime	Max alowed time a Topic can retain data	YES	Must be positive or -1(Infinite)
Default RetentionSize	Default size a Topic can retain data	NO	Must be positive or -1(Infinite)
Max RetentionSize	Max alowed size a Topic can retain data	YES	Must be positive or -1(Infinite)
Naming Pattern	Regex to determine if a Topic name is allowed	NO
Naming Description	Desription for the regex	NO

USEFUL TIP

Topic names can be restricted via a given pattern. Lenses provides out of the box a few options: "Only letters", "Letters and numbers" and "Letters, numbers and dots"

Troubleshooting & F.A.Qs

If you have any question, please refer to the following list. If you still have more, we are more than welcome to answer any question at our community channel.

Can I specify a custom naming strategy?

Under Topic Name, one can set the value of “Custom” for the characters allowed and provide their own regular expression string and description. The description will be used as a naming hint when creating new topics .

What happens if I have set defaults on the Broker?

Topic settings policies take precedence over Kafka Broker defaults. By default the policies are empty.