Latest version: 4.1.x
- How Lenses Data Catalog works.
- How Permissions and Compliance works.
- How Lenses accelerates Metadata Exploration.
- F.A.Qs and troubleshooting guides, in case you get stuck somewhere.
Lenses Data Catalog, can preview data present for various sources. Currently, we support
PostgreSQL. Using the Data Catalog, you can assign Metadata(Description, Tags), giving your users the ability to surface relevant information faster, all while keeping your data secure and compliant.
By default, Lenses is connected to an instance of Kafka. But, we can use
Connections to connect Lenses with more sources. You can find out how to connect to
PostgreSQL in our dedicated articles.
Once successful connections have been established, we can view them in the Explore screen, under the available Data Sources. If you are having trouble connecting Lenses to PostgreSQL, please refer to our Troubleshooting Guide and/or F.A.Qs for each source.
Security & Compliance
As with every source(Kafka, Elasticsearch until now), PostgreSQL, is subject to Lenses RBAC permissions, and policies for mathcing fields. Keep in mind that Lenses RBAC, are completely independant from your DBs permissions.
Lenses is using a RBAC(Role Based Access Control) permissioning system, to allow for granular control across all your sources(Kafka, Elasticseach and PostgreSQL)  by creating Groups with the appropriate permissions.
Data policies enable compliance with regulations such as GDPR, CCPA, or HIPAA. We use Data Policies to obfuscate data retrieved from Lenses via the UI, CLI, or API without affecting how the underlying data is stored . When we apply the policy, Lenses will automatically obfuscate all fields that match.
Search for Metadata
Lenses Data Catalog(D.C from now on), can help you identify
Datasets based on some metadata criteria. You can either
filter based on
Data Sources and/or
Tags, or search for
fields and their
descriptions. Now, let’s focus a bit more, on each individual functionality.
We can filter, either based on
Data Sources or
Tags. Please keep in mind, that while the
Data Sources filter is accumulative(Logical OR), the
Tags are aggregating(Logical AND).
|Data Source||Each individual connection to Lenses.|
|Tags||Metadata assigned to each Dataset.|
We can search for specific keywords. They will be matched to one of the following properties of a Dataset. Please keep in mind, that the keyword matching is not case sensitive.
|Datset name||The name of the individual Dataset.|
|Field name||The name for each field of that Dataset.|
|Field description||The part of the description for each field.|
Lenses UI, can preview Datasets in the D.C. We can also move to the Details Page, for each Daset and dive deeper for more information. Let’s take a tour of its capabilities.
Data Catalog, Basics
By navigating to the Explore screen, we can see that we have a few ways to search for Metadata, and identify the information we want. Contexual information about the applied filters is also present  along with the results of your search.
We can fine tune our search by filtering, searching, including/excluding information for internal Datasets and/or Schema information . We can also customise the Columns that are present in the Table .
Data Catalog, Filter
We can filter, by Data Source. Below, we can see, that we have 3 Connections available to us. One for each of the Data Sources, Lenses is able to connect to
PostgreSQL along with its health-check status.
We can also filter based on Tags. We can see, in the image below, we have search for the character
K, amongst the
Tags and we can see our selection as well as mathcing results.
Data Catalog, Search
Just like we mentioned, we are able to search for specific keywords. Below, we are able to see, that the keyword, match part of the name for each field. The correspoding match is colored, for easier navigation.
F.A.Qs & Troubleshooting
Why isn’t the D.C update instantly?
We have a cache mechanism, that is update periodically. By default, it is 30 secs, but you can override it, in the configuration. Learn more
Is the keyword matcher case-sensitive?
No, when we match a keyword, we match without checking for case sensitivity. Meaning, that if you search for
info, it will match both
INFO for all the corresponding properties.
Can I connect other Data Sources to Lenses?
Currently, we support aside from
Kafka just Elasticsearh and PostgreSQL. We are planning to support more in the near future, so please feel free to submit your idea in our