Data Policies

Data policies enable compliance with regulations such as GDPR, CCPA or HIPAA. Policies operate in a secure layer of the architecture, yet they do not affect your raw Kafka data and applications.

Policies concepts

Lenses follows the National Institute of Standards and Technology (NIST) standards. Policies gives you the ability to apply masking to Kafka specific fields to all Lenses channels (UI/CLI/API/SQL).

  • Redaction Policy: Whether to protect messages at a field level.
  • Category: Category of sensitivity in the data.
  • Impact: The business impact levels concerning the data.
  • Fields: Definition of fields that the data policy will apply to.
The Policies option is available if the permission Data Policies is set

Create a policy

To create a policy:

1. Navigate to Policies
2. Click the New Policy button

No Data Policies for sensitive data

No data policies are enabled by default. Import automatically recommended data policies, or create a new one.

New Data Policy

In the above example, we are:

  1. Creating a new Credit Card Numbers data policy
  2. Selecting LAST-4 as the redaction policy
  3. Setting Financial Data as the data category
  4. Setting HIGH as the impact to the business
  5. Applying this policy to all topics containing the field credit_card

Data policies in action

Once the above has been created, Lenses will automatically identify ALL Kafka topics and Elasticsearch indexes that contain credit card info.

New Data Policy

Any data on Kafka, whether serialized as Avro, JSON, XML or even ProtoBuf that contains credit_card information will automatically be detected.

Apart from identifying all the sensitive data at a field level, Lenses will also protect the data for you.

Kafka PII Data Policy

That means that anyone accessing data via Lenses (UI/CLI/Python) can access production data while respecting the the sensitivity of the underlying data.

Mask Kafka data

Lenses applies the masking to data any time you request to access it. The available policies are:

  • None Track sensitive data, but do not protect them.
  • Last-4 Display the last 4 characters of the value.
  • First-4 Display the first 4 characters of the value.
  • Initials Display the first letter of each word.
  • Email Mask email address, showing the domain name.
  • Number Mask numbers by replacing them with null / zero / or -1.

Manage policies

Once you setup a data policy, you can view all policies and how sensitive data exist in your data platform.

List Lenses Data Policies

Provided your account has the relevant access level, you can click on data policy and edit or remove it.

Edit or remove Data Policies

If you select to edit a data policy you can change its configuration.

Edit Lenses Data Policy

Policy associated resources


Lenses identifies all data resources automatically with messages that contain any sensitive payload field. This happens across all data format (JSON, AVRO, XML etc.) and whether the field lives at the record level in the key or value or even in a nested structure.

Flows / Connectors

Lenses identifies all flows and connectors that are consuming or producing such sensitive data so that you can track their usage across multiple data systems.

Flows / SQL Processors

Lenses identifies all streaming SQL processors that produce or consume sensitive data.

Flows / Custom Applications

Lenses identifies all your micro-services and application that are producing or consuming sensitive data.