Lenses SQL Processors

Lenses SQL processors allow you to create continuous data queries to aggregate, join and/or transform data streams. Through the Lenses, the topologies can be visualized, monitored, or even scaled out for more throughput. Furthermore, the execution plan viewer allows you to optimize for performance.

Learn about Lenses SQL engine for Apache Kafka

  1. From the Header Bar Menu, go to the Dashboard panel.
  2. On the side navigation, select SQL Processors under Flows section.

processors

Create SQL Processors

SQL Processors require a friendly Name but are also be assigned a unique App ID automatically.

The processor will follow the Lenses SQL streaming semantics. Make sure you are familiar with the syntax.

  1. Click on the New SQL Processor button
  2. Fill in the form details as appear below.
Create Processors

Running on Connect mode

If the deployment mode is CONNECT, the Cluster and must also be set.

Running on Kubernetes mode

If the deployment mode is KUBERNETES, the Namespace must be set in addition to the cluster.

Monitor Processors

After the processor is created, you can monitor the data, aggregated and runner level metrics, as well as add alerts to its generated consumer groups.

In Kubernetes mode, you can also inspect the logs for each runner.

Topology

Once the SQL processor is created, Lenses generates the internal Topology on how the data is processed based the SQL query provided.

Click on each node to see the details for every step of the query.

processors

When the node is a Topic, sample data of the live stream appear the help debug and compare the before and after payloads.

Metrics

Metrics are provided for the SQL Processors. You can see Records IN and OUT, Size of data moved and Byte rates to plan your scaling strategy.

processors

Runners

Each Runner can be monitored individually.

processors

Manage Processors

Processors can be Stopped, Resumed or Deleted and also Scaled up or down. The Scale option is available for Kubernetes and Connect modes only.

Processors
Processors

When scaling, partitions will be re-distributed as per the Kafka semantics, and scale the processor horizontally.

Learn more about scaling SQL Processors

Use the CLI

See CLI Docs

Use the API

See API Docs

Permissions for processors

Application permissions govern processors. This means that we can access the nodes related to the namespaces you have access too. See more on Application Permissions