Kafka Quotas

Kafka Quotas can be used to

  • deliver better SLAs
  • reduce latency
  • offer an improved multi-tenant experience and
  • provide a charge back model for cost-center allocation of the operation cost

of you data platform, by applying quotas on consume/produce resources.

Manage Kafka quotas

  1. From the Header bar menu, go to Admin panel.
  2. On the side navigation, select Kafka Quotas under the Kafka Settings section.
Kafka Quotas

Prerequisites

To manage quotas on Kafka, you need to i) have the Kafka Admin permission on your account and ii) have Zookeeper access. See how to configure ZK .

Cloud and managed Kafka services do not currently offer access to Zookeeper and quota management will become available via KIP-546.

Create Kafka quotas

You can create new Quotas, with the New Quota button and selecting the client or user (or combination) to apply this quota on. You can also set up default quotas.

For example, to apply a Quota on a particular application that can create an aggressive spike on your data platform (i.e. an Apache Spark application), select CLIENT:

Add a new Kafka Quota

And for example, enforce that specific application to not produce and consume more than 5MBytes/sec:

Create new Kafka Quota

Explore quotas

With quota management enabled, you can create, search, and manage your Quotas:

List Kafka Quotas

Quota precedence

You can enforce quotas on produce and fetch requests for:

  • A user ( requires an authenticated Kafka )
  • An application ( using the client-id )
  • The application of a specific user ( requires an authenticated Kafka )

The following order of precedence is followed when multiple quotas are in place.

  1. (user) / (client-id)
  2. (user) / (default)
  3. (user)
  4. default / (client-id)
  5. default / clients / (default)
  6. users / default
  7. clients / (client-id)
  8. clients / default

Learn more about Kafka Quotas .

API & GitOps

For automation use the API and CLI

Kafka Quotas permissions

You can use RBAC (Role Based Access) to control users that have visibility and management roles for Kafka Quotas.

Specifically, the permissions ViewKafkaSettings and ManageKafkaSettings are needed to view and manage respectively.