Dataset Metadata

WHAT YOU'LL LEARN

How metadata work in the Lenses Data Catalog
What kind of Permissions, you need to View/Manage them.
How to view, search and Manage Metadata from the UI/CLI.
F.A.Q and troubleshooting guides, in case you get stuck somewhere.

Introduction

Lenses Data Catalog, supports the usage of Metadata properties. You can assign metadata(Description, Tags), giving your users the ability to surface relevant information faster. Metadata, are applicable for each source, that Lenses supports.

Metadata augments the business context to technical entities while helping with the information graph of a DataOps workspace. It is an essential part of a cataloguing experience because it helps users bring a different dimension on their data

RBAC Security

Lenses is using a RBAC(Role Based Access Control) permissioning system, to allow for granular control across all your sources [1] by creating Groups with the appropriate permissions.

Both “Description” and “Tags” are applicable to the same Namespace Permission. By default all users can view the metadata for a dataset. In order to manage your metadata though, a user needs the updateMetadata permission.

USEFUL TIP

This permission is categorised as a "Namespace" Permissions, thus you will need to apply it for every different connection Lenses is connected to. Meaning that if you have (10 PostgreSQL Connections) you will need to apply the permission for each group.

API Clients

You can view and manage, your dataset’s metadata, either through the Lenses UI or the CLI. All permissions, are applied on the API Level, so every client consuming the API is protected.

CLI

Using the CLI, you can view the Metadata for each dataset, as well as manage them. If you are interested in the CLI, you can find more info in our Docs

UI

The Data Catalog, allows you to filter by Tags [1], to identify the relevant resources.

USEFUL TIP

Please keep in mind, that while the data sources filter is accumulative(Logical OR), the tags are aggregating(Logical AND).

Then you can dive deeper into each dataset. In the Details page, you can see also the description for each Dataset. Given the necessary permissions, you will also be able to update the metadata [2],[3].

Troubleshooting & F.A.Qs

Are there any limitations for the Description?

Description accepts for 0 to 512 characters long and we do not count whitespace characters. Bear in mind, that while Markdown is a valid input for description, we do not render any markdown elements. We treat it as plain string.

Are there any limitations for the Tags?

Tags are alphanumeric strings, that can be from 1 to 255 characters long. Special characters are allowed. The only limitation is that Tags, cannot have spaces. You can create as many Tags as you want.

Why isn’t the Data Catalog update instantly?

We have a cache mechanism, that is update periodically. By default, it is 30 secs, but you can override it, in the configuration. Learn more .