Re-Create SQL Processors to a different deployment target

Moving Lenses configuration to deploy SQLProcessors from one deployment mode (aka deployment target) to a different one cannot be done automatically. That's because some configuration specific to the deployment mode need to be provided by the user.

As an example, let's say we want to change the deployment mode from InProc to Kubernetes. The problem comes that, for deploying in Kubernetes, we need some information from the user, such as the cluster name, or the namespace.

Bearing the above in mind, we can use Lenses CLI to perform these manual steps to ensure that existing processors are recreated using a different deployment mode:

For the below examples, we assume lenses-cli is in the classpath

List existing processors

lenses-cli processors list

Stop each running SQLProcessor

lenses-cli processor stop --id <application_id>

Export processors

lenses-cli export processors

You can also export only specific entities:

lenses-cli export processors --id <application_id>

In any case, an apps/sql folder will be created, containing a processor descriptor in yaml format

Update the processor yaml file

Set or update the following keys, depending on the final deployment target:

Key	Instructions
cluster	Set to the cluster name where you want to deploy your SQLProcessor. In case of moving to IN_PROC, this field will be ignored.
namespace	Set it only if you final deployment target is Kubernetes. It will be the namespace where the SQLProcessor will be deployed to once you decide to start it.

Note: Bear in mind that the appId represents the ConsumerGroupId of the underlying Kafka client. This means that if the value we have for our SQLProcessors is not available in the Kafka/Zookeeper cluster, the processor will start processing from the beginning once we start it again, instead of resuming its work from the offset it was when we stopped it.

Create the processors

Having Lenses running with the new configuration - lenses.sql.execution.mode=[IN_PROC|CONNECT|KUBERNETES] - it’s time to create the processors

lenses-cli processor create ./apps/sql/*

List all processors again

If we now list all processors again, we’ll see our processors, but now registered with the new deployment target

lenses-cli processors list

Start a processor

lenses-cli processor start --id <application_id>

Example

The exported yaml file looks like:

name: processor-1
sql: |-
  -- Create target topic(s) if they do not exist
  SET defaults.topic.autocreate=true;
  -- When automatically creating topic(s) set non-default values
  -- SET defaults.topic.partitions=1;
  -- SET defaults.topic.replication=1;
  -- SET defaults.topic.cleanup.policy=delete;
  -- Control error policy [continue,fail(default),dlq]:
  -- SET error.policy='fail';

  INSERT INTO sink-topic SELECT STREAM * FROM source-topic;
runnerCount: 1
cluster: IN-PROC
namespace: Lenses
pipeline: processor-1
appId: lsql-processor-1-1551637290

To move the processor to Kubernetes, using the cluster name my-custom-cluster and the namespace sql-lenses, the updated yaml file will look like this:

name: processor-1
sql: |-
  -- Create target topic(s) if they do not exist
  SET defaults.topic.autocreate=true;
  -- When automatically creating topic(s) set non-default values
  -- SET defaults.topic.partitions=1;
  -- SET defaults.topic.replication=1;
  -- SET defaults.topic.cleanup.policy=delete;
  -- Control error policy [continue,fail(default),dlq]:
  -- SET error.policy='fail';

  INSERT INTO sink-topic SELECT STREAM * FROM source-topic;
runnerCount: 1
cluster: my-custom-cluster
namespace: sql-lenses
pipeline: processor-1
appId: lsql-processor-1-1551637290