Scale SQL Processors

The SQL streaming engine supports three execution modes: IN_PROC, CONNECT, and KUBERNETES. The last two are made available to Enterprise clients and offer fault-tolerant streaming applications built via Lenses SQL.

In Process mode

IN_PROC is the default execution mode with processors executed locally inside Lenses. This mode is recommended only for testing because when Lenses is down, so are the processors.

Kubernetes mode

KUBERNETES is an execution mode that provides scalability by deploying Lenses SQL runners into Kubernetes clusters. Lenses can deploy and monitor SQL runner deployments created through Lenses or external tools. We recommend only using Lenses to manage Lenses SQL processor deployments.

Connect mode

CONNECT is an execution mode that provides scalability by deploying Lenses SQL runners in Kafka Connect if you do not have access to a Kubernetes cluster. Kafka Connect provides a distributed, fault-tolerant, and scalable framework as part of the core Apache Kafka distribution. We advise allocating a Kafka Connect cluster to run the stream processing with SQL.