Latest version: 4.3.x
Lenses SQL Processors
Lenses SQL processors allow you to create continuous data queries to aggregate, join and/or transform data streams. Through the Lenses, the topologies can be visualized, monitored, or even scaled out for more throughput. Furthermore, the execution plan viewer allows you to optimize for performance.
SQL engine for Apache Kafka documentation
Navigate to Processors
- From the Header Bar Menu, go to the Dashboard panel.
- On the side navigation, select SQL Processors under Flows section.
Create SQL Processors
SQL Processors require a friendly Name but are also assigned a unique App ID automatically.
The processor will follow the Lenses SQL streaming semantics. Make sure you are familiar with the syntax.
- Click on the New SQL Processor button
- Fill in the form details as appear below.
Running on Connect mode
If the deployment mode is CONNECT, the Cluster must also be set.
Running on Kubernetes mode
If the deployment mode is KUBERNETES, the Namespace must be set in addition to the cluster.
Monitor Processors
After the processor is created, you can monitor the data, aggregated and runner level metrics, as well as add alerts to its generated consumer groups.
In Kubernetes mode, you can also inspect the logs for each runner.
Topology
Once the SQL processor is created, Lenses generates the internal Topology on how the data is processed based the SQL query provided.
Click on each node to see the details for every step of the query.
When the node is a Topic, sample data of the live stream appear the help debug and compare the before and after payloads.
Metrics
Metrics are provided for the SQL Processors. You can see Records IN and OUT, Size of data moved and Byte rates to plan your scaling strategy.
Runners
Each Runner can be monitored individually.
Manage Processors
Processors can be Stopped, Resumed or Deleted and also Scaled up or down. The Scale option is available for Kubernetes and Connect modes only.
When scaling, partitions will be re-distributed as per the Kafka semantics, and scale the processor horizontally.
Learn more about scaling SQL Processors
Use the CLI
Use the API
Permissions for processors
Application permissions govern processors. This means that we can access the nodes related to the namespaces you have access to. See more on Application Permissions