Data policies enable compliance with regulations such as
Policies operate in a secure layer of the architecture, yet they do not affect
your raw Kafka data and applications.
Lenses follows the National Institute of Standards and Technology (NIST) standards. Policies gives you the ability to apply masking to Kafka specific fields to all Lenses channels (UI/CLI/API/SQL).
- Redaction Policy: Whether to protect messages at a field level.
- Category: Category of sensitivity in the data.
- Impact: The business impact levels concerning the data.
- Fields: Definition of fields that the data policy will apply to.
Create a policy
To create a policy:
1. Navigate to Policies
2. Click the New Policy button
No data policies are enabled by default. Import automatically recommended data policies, or create a new one.
In the above example, we are:
- Creating a new
Credit Card Numbersdata policy
LAST-4as the redaction policy
Financial Dataas the data category
HIGHas the impact to the business
- Applying this policy to all topics containing the field
Data policies in action
Once the above has been created, Lenses will automatically identify ALL Kafka topics and Elasticsearch indexes that contain credit card info.
Any data on Kafka, whether serialized as Avro, JSON, XML or even ProtoBuf that
credit_card information will automatically be detected.
Apart from identifying all the sensitive data at a field level, Lenses will also protect the data for you.
That means that anyone accessing data via Lenses (UI/CLI/Python) can access production data while respecting the the sensitivity of the underlying data.
Mask Kafka data
Lenses applies the masking to data any time you request to access it. The available policies are:
- None Track sensitive data, but do not protect them.
- Last-4 Display the last 4 characters of the value.
- First-4 Display the first 4 characters of the value.
- Initials Display the first letter of each word.
- Email Mask email address, showing the domain name.
- Number Mask numbers by replacing them with null / zero / or -1.
Once you setup a data policy, you can view all policies and how sensitive data exist in your data platform.
Provided your account has the relevant access level, you can click on data policy and edit or remove it.
If you select to edit a data policy you can change its configuration.
Policy associated resources
Lenses identifies all data resources automatically with messages that contain any sensitive payload field. This happens across all data format (JSON, AVRO, XML etc.) and whether the field lives at the record level in the key or value or even in a nested structure.
Flows / Connectors
Lenses identifies all flows and connectors that are consuming or producing such sensitive data so that you can track their usage across multiple data systems.
Flows / SQL Processors
Lenses identifies all streaming SQL processors that produce or consume sensitive data.
Flows / Custom Applications
Lenses identifies all your micro-services and application that are producing or consuming sensitive data.