AWS MSK

Get started with Lenses. Set it up with Docker and connect to an AWS managed Kafka (MSK) with the standard configuration.

1. Get your Lenses access key

Sign up with Lenses and download your access key.

2. Sign up for AWS and create an MSK Kafka

MSK for Apache Kafka is a fully managed streaming platform by AWS, deployable in the AWS cloud.

Start by signing up for an AWS account . Follow the in-product instructions to launch Kafka clusters within the AWS console interface.

AWS MSK Kafka

Then, enable connectivity and open monitoring to get the most of it.

Enable connectivity

Allow your Lenses instance to connect to the MSK cluster. Edit the MSK security group and add the IP address of your Lenses Docker host.

Enable connectivity

Enable open monitoring

Enable it on an existing cluster with this AWS guide .

Enable open monitoring

Gather 2 connection points

Go to View Client Information

  1. Kafka endpoints (TLS)
  2. Zookeeper endpoints (TLS)
Kafka and Zookeeper endpoints

3. Get Docker

You will use docker-compose to spin up Lenses.

Start by signing up with Docker . Then install docker-compose .

Install Docker

4. Setup Lenses docker-compose

Use this docker-compose.yml file to spin up Lenses and connect to your AWS MSK Kafka.

Just 2 steps:

  1. Fill in the connection details:
    1. Kafka endpoint
    2. Zookeeper endpoint
    3. Kafka JMX metrics
  2. Share the authentication files with your docker. Copy these files in the same directory as your docker-compose.yml file and reference them:
    1. Your Lenses license key license.json

The file:

version: '3'
services:
  lenses:
    image: lensesio/lenses
    environment:
      # Admin user credentials
      LENSES_SECURITY_USER: admin
      LENSES_SECURITY_PASSWORD: admin
      LENSES_PORT: 8080
      
      # === Set your values in this section ===
      
      # ===Step 1=== Put your MSK Kafka TLS endpoints here as SSL here
      # e.g. 
      # LENSES_KAFKA_BROKERS: "SSL://b-1.lenses-msk.4o9la5.c4.kafka.eu-west-2.amazonaws.com:9094,SSL://b-2.lenses-msk.4o9la5.c4.kafka.eu-west-2.amazonaws.com:9094"
      # Kafka Endpoint
      LENSES_KAFKA_BROKERS: "SSL://[BROKER_HOST_1]:[TLS_PORT_1], SSL://[BROKER_HOST_2]:[TLS_PORT_2],…"
      # MSK SSL configuration
      LENSES_KAFKA_SETTINGS_CLIENT_SECURITY_PROTOCOL: SSL
      
      # ===Step 2=== Put your MSK Zookeeper TLS endpoints here
      LENSES_ZOOKEEPER_HOSTS: |
        [
          { url: "[ZK_HOST_1]:[TLS_PORT_1]" },
          { url: "[ZK_HOST_2]:[TLS_PORT_2]" },
          { url: "[ZK_HOST_3]:[TLS_PORT_3]" }
        ]

      # ===Step 3=== Put your MSK open monitoring config here
      # Kafka JMX metrics with MSK open monitoring
      # e.g.
      #  {
      #    type: "AWS",
      #    port: [
      #      {id: 1,  url:"http://b-1.lenses-msk.4o9la5.c4.kafka.eu-west-2.amazonaws.com:11001/metrics"},
      #      {id: 2,  url:"http://b-2.lenses-msk.4o9la5.c4.kafka.eu-west-2.amazonaws.com:11001/metrics"}
      #    ]
      #  }
      LENSES_KAFKA_METRICS: |
        {
          type: "AWS",
          port: [
            {id: 1,  url:"http://[BROKER_HOST_1]:11001/metrics"},
            {id: 2,  url:"http://[BROKER_HOST_2]:11001/metrics"}
            …
          ]
        }
      
    ports:
      - 8080:8080
    volumes:
      # ===Step 4=== Share your authentication files here
      # Map the Lenses license key file here to activate Lenses
      - ./license.json:/license.json

4. Start Lenses up

docker-compose up

5. Login

Open your browser and go to:

http://localhost:8080

Login as admin:

User:     admin
Password: admin

6. Eureka!

Lenses Dashboard

7. Next steps

Learn the basics.

Limitations

Currently, AWS doesn’t provide managed versions of these services:

  • Schema Registry
  • Kafka Connect

Without these services these Lenses features are not available: