Data Catalog

Introduction

Lenses Data Catalog, can preview data present for various sources. Currently, we support Kafka, Elasticsearch and PostgreSQL. Using the Data Catalog, you can assign Metadata(Description, Tags), giving your users the ability to surface relevant information faster, all while keeping your data secure and compliant.

Connecting Sources

By default, Lenses is connected to an instance of Kafka. But, we can use Connections to connect Lenses with more sources. You can find out how to connect to Elasticsearch and PostgreSQL in our dedicated articles.

Once successful connections have been established, we can view them in the Explore screen, under the available Data Sources. If you are having trouble connecting Lenses to PostgreSQL, please refer to our Troubleshooting Guide and/or F.A.Qs for each source.

Security & Compliance

As with every source(Kafka, Elasticsearch until now), PostgreSQL, is subject to Lenses RBAC permissions, and policies for mathcing fields. Keep in mind that Lenses RBAC, are completely independant from your DBs permissions.

Lenses RBAC

Lenses is using a RBAC(Role Based Access Control) permissioning system, to allow for granular control across all your sources(Kafka, Elasticseach and PostgreSQL) [1] by creating Groups with the appropriate permissions.

Data Governance

Data policies enable compliance with regulations such as GDPR, CCPA, or HIPAA. We use Data Policies to obfuscate data retrieved from Lenses via the UI, CLI, or API without affecting how the underlying data is stored [1]. When we apply the policy, Lenses will automatically obfuscate all fields that match.

Search for Metadata

Lenses Data Catalog(D.C from now on), can help you identify Datasets based on some metadata criteria. You can either filter based on Data Sources and/or Tags, or search for names, fields and their descriptions. Now, let’s focus a bit more, on each individual functionality.

Filter

We can filter, either based on Data Sources or Tags. Please keep in mind, that while the Data Sources filter is accumulative(Logical OR), the Tags are aggregating(Logical AND).

Search

We can search for specific keywords. They will be matched to one of the following properties of a Dataset. Please keep in mind, that the keyword matching is not case sensitive.

UI

Lenses UI, can preview Datasets in the D.C. We can also move to the Details Page, for each Dataset and dive deeper for more information. Let’s take a tour of its capabilities.

Data Catalog, Basics

By navigating to the Explore screen, we can see that we have a few ways to search for Metadata, and identify the information we want. Contexual information about the applied filters is also present [1] along with the results of your search.

We can fine tune our search by filtering, searching, including/excluding information for internal Datasets and/or Schema information [2]. We can also customise the Columns that are present in the Table [3].

Data Catalog, Filter

We can filter, by Data Source. Below, we can see, that we have 3 Connections available to us. One for each of the Data Sources, Lenses is able to connect to Kafka, Elasticsearch and PostgreSQL along with its health-check status.

We can also filter based on Tags. We can see, in the image below, we have search for the character K, amongst the Tags and we can see our selection as well as mathcing results.

Data Catalog, Search

Just like we mentioned, we are able to search for specific keywords. Below, we are able to see, that the keyword, match part of the name for each field. The correspoding match is colored, for easier navigation.

F.A.Qs & Troubleshooting

Why isn’t the D.C update instantly?

We have a cache mechanism, that is update periodically. By default, it is 30 secs, but you can override it, in the configuration. Learn more

Is the keyword matcher case-sensitive?

No, when we match a keyword, we match without checking for case sensitivity. Meaning, that if you search for info, it will match both info and INFO for all the corresponding properties.

Can I connect other Data Sources to Lenses?

Currently, we support aside from Kafka just Elasticsearh and PostgreSQL. We are planning to support more in the near future, so please feel free to submit your idea in our productboard .