Kafka topic settings

WHAT YOU'LL LEARN
  1. What are Topic settings and how to user it.
  2. F.A.Q and troubleshooting guides, in case you get stuck somewhere.

Introduction

Topic settings gives the option to define constraints around topic partitions, replication, retention, and naming. To be able to change the rules the user needs to belong to at least one group which grants manage permission for Kafka Settings in the Admin section. They apply when creating new topics or configuring the existing ones through Lenses UI, CLI, and REST API.

Please note that such constrains are specific to Lenses and will be ignored when creating/configuring topics through other tooling.

IMPORTANT INFO
Under certain circumstances, the topic settings configured through Lenses might no longer be applicable. For instance, if you set a min replication factor of 3, but the number of active brokers in your Kafka cluster suddenly drops to 2, your settings will enter into an unapplicable state. When this happens, Lenses will lock all API calls around topic creation and configuration until this inconsistency is addressed. This can be done either by bringing the third broker back up, or by decreasing the minimum replication factor to 2.

CLI

Lenses CLI, can be used to view and update the Topic Settings, as well as import and export to promote settings between different environments.

View

The following command can be used to retrieve the Topics Settings details. Output can be either JSON or YAML.

lenses-cli topic-settings --output="JSON"

Update

Use the following command to update the topic settings policies:

lenses-cli topic-settings update \
    --partitions-min 1 --partitions-max 3 \
    --replication-min 1 --replication-max 2 \
    --retention-size-default -1 --retention-size-max -1 \
    --retention-time-default -1 --retention-time-max -1 \
    --naming-pattern="[a-zA-Z]*" --naming-description="Only letters ([a-zA-Z]*)"

Export/Import

Importing/exporting the topic settings policies can be done via this command:

lenses-cli export topic-settings --dir="DIRECTORY-PATH"
lenses-cli import topic-settings --dir="DIRECTORY-PATH"

UI

The Topic Settings policies can be found under the Admin section of Lenses. The user requires View permission for the Kafka Settings option.

Form ParameterDescriptionRequiredNotes
Min PartitionsMin number of acceptable partitionYESMust be more or equal to 1
Max PartitionsMax number of acceptable partitionNOMust be more or equal to 1
Min ReplicationMin number of acceptable replication factorYESMust be more or equal to 1
Max ReplicationMax number of acceptable replication factorNOMust be more or equal to 1
Default RetentionTimeDefault time a Topic can retain dataNOMust be positive or -1(Infinite)
Max RetentionTimeMax alowed time a Topic can retain dataYESMust be positive or -1(Infinite)
Default RetentionSizeDefault size a Topic can retain dataNOMust be positive or -1(Infinite)
Max RetentionSizeMax alowed size a Topic can retain dataYESMust be positive or -1(Infinite)
Naming PatternRegex to determine if a Topic name is allowedNO
Naming DescriptionDesription for the regexNO
Topic settings
USEFUL TIP
Topic names can be restricted via a given pattern. Lenses provides out of the box a few options: "Only letters", "Letters and numbers" and "Letters, numbers and dots"

Troubleshooting & F.A.Qs

If you have any question, please refer to the following list. If you still have more, we are more than welcome to answer any question at our community channel.

Can I specify a custom naming strategy?

Under Topic Name, one can set the value of “Custom” for the characters allowed and provide their own regular expression string and description. The description will be used as a naming hint when creating new topics .

Topic name

What happens if I have set defaults on the Broker?

Topic settings policies take precedence over Kafka Broker defaults. By default the policies are empty.