Data Types & Schemas

For every Kafka Topic, Lenses tracks and maintains a schema. The Kafka Topic records consist of a Key and a Value and, in some cases, a Header. For Keys and Values, Lenses assigns a type format.

Supported types

At regular time intervals, Lenses auto-detects and keeps track of topic formats and schemas.

The supported Key/Value formats are:

• JSON, AVRO, PROTOBUF, XML, CSV, STRING, INT, LONG, BYTES (default), as well as custom pluggable formats.

For the AVRO and the PROTOBUF formats, Lenses relies on the configured schema registry to detect the topic format and schema. For other formats, detection happens by inspecting the raw bytes of the first message in the first topic partition.

For semi-structured formats such as JSON and XML, Lenses tries to infer a schema if the message value is an object with one or more fields. Please note that such schema is inferred and set only once: it’s down to you managing its evolution safely (e.g. re-setting the schema when a new field is introduced, avoiding backward incompatible changes, etc).

Update types

To update topic formats and the Lenses detected schemas (in JSON/XML formats), the user will require the appropriate Namespace permission.

To update the Schema Types:

1. From the Header Bar Menu, go to the Dashboard panel
2. On the side navigation, select Explore
3. Select the topic you wish to edit
4. Click on the Actions menu on the top right of your screen
5. Select the Change Type (Key, Value) option

6. Change the Key and / or Value types.

Update schema

If the topic payload format is AVRO or PROTOBUF, Lenses delegates schema management to the configured Schema Registry and continuously updates its format/schema information by polling the registry at regular intervals. For other structured formats that are currently not managed by via the schema segistry (i.e. JSON/XML), Lenses
persist and manages the inferred schema in its own internal database.

To edit the Lenses schema for XML and JSON formats:

1. From the Header Bar Menu, go to the Dashboard panel
2. On the side navigation, select Topics under Schema tab
3. Select the topic you wish to edit
4. Click on the Actions menu on the top right of your screen
5. Select the Edit Schema option

6. Edit the schema as you see fit

Custom types

Lenses supports custom serializers to support custom data formats.

See Docs for custom formats