How to Work with your data

Data Types & Schemas

For every Topic, Lenses tracks and maintains a schema. The Kafka Topic records consist of a Key and a Value and in some cases a Header. For Keys and Values, Lenses assigns a type format.

Topic Schemas

Supported types

Lenses auto-detects and keeps a schema and data formats for each Topic. When a topic is detected for the first time, Lenses tries to identify the type for the Key and the Value as well as work out the schema.

The supported formats for Key and Value are:

  • JSON, AVRO, XML, CSV, PROTOBUF, STRING, INT, LONG, BYTES (default), Custom pluggable formats.

Update types

Topic Schema

To update the Schema Type and Lenses Schema the user will require the appropriate Namespace permission.

To update the Schema Types:

  • 1. Navigate to the Topic page, and select the Schema tab.
  • 2. Select the Change Type option.
  • 3. Change the Key or Valye types.

Update schema

To update the Schema:

If schema registry for Avro types exists, Lenses communicates directly with the schema registry to get the Avro schemas.

If the type is not Avro, Lenses is trying to detect the schema for the Key and Value. Lenses stores this schema in its tablestore.

To edit the Lenses Schema:

  • 1. Navigate to the Topic page, and select the Schema tab.
  • 2. Select the Edit Schema option.

Topic Schema Edit

Custom types

Lenses supports custom serializers in order to support custom data formats.

See Docs for custom formats