Kafdrop

The situation

; TLDR: need an easy way to visualize messages payload in kafka, in read only

There are a number of tools that allow users to visualize the messages and configurations of Kafka topics. Confluent has their Control Center which is very comprehensive and has a lot of features. But, it has a major flow for me: it requires pretty much admin level of access and unless you have it deployed yourself, implementing RBAC for it is pityful, and very limited in actual roles assignments. And, no read-only mode…

But forget about Confluent for a minute, generally speaking, providing access to users to productions clusters can be a risk:

  • The information in the topics might contain PPI, very sensitive

  • You want people to be able to visualize the information but not alter it, if there is no need for it.

So I looked at alternatives and directed to using Kafdrop

The documentation is not the most comprehensive but it does the job and although I would have preferred a solution which could take all the settings from environment variables and not in b64, kafdrop does a really good job at doing what I needed it to do.

Deployment

I want to use an ALB and cognito + Azure SAML provider. So I set everything up to be ready for it, which I won’t cover it here, and in the example files, I commented out the Cognito configuration for you to be able to deploy it without worrying about it.

However, I will be having NGINX in front of it to have TLS termination between the ALB and kafdrop. This also allows me to add HSTS to it, for some extra security.

NGINX will be here as a simple reverse proxy, but one could add settings or features to it to and really anything you’d want that NGINX can do for you.

I could have re-use the original NGINX image provided on Dockerhub or AWS ECR Public, but, for this one I want to use Amazon Linux as the base OS, so I am building that image myself.

For kafdrop itself, there also is a docker image published on dockerhub, but I preferred to re-compile it and host the jar on S3 and that helps me test out further Files Composer .

Hint

For NGINX and kadrop, feel free to re-use the images from dockerhub, this would work in the same way, totally up to your preference.

Files and config in docker volumes

We are going to need two volumes: one for NGINX configuration, and one for kafdrop JAR and configuration.

Attention

Although I am doing this here on purpose for testing reasons, I do not recommend to have your application files, here the kafdrop JAR, outside of the docker image, to respect the principle of immutability of the application.

We then indicate how we are going to mount them in our services. First container starting, files-composer, will retrieve the various files, generate the self-signed certificates for NGINX to use, and on success, exit successfully (return 0).

Tip

Only if the retrieval of the files and generating configurations, certificates etc, will then the other containers start. That is achieved with the deploy.labels ecs.depends.condition: SUCCESS property.

So let’s have a look at our volumes and services using them.

volumes:
  nginx:
  kafdrop:

services:
  files-composer:
    volumes:
    - kafdrop:/app
    - nginx:/etc/nginx

  nginx:
    volumes:
    - nginx:/etc/nginx/ssl:ro

  kafdrop:
    volumes:
    - kafdrop:/app:ro

Note

We only mount them in the files-composer container with RW access, but the other two containers shan’t modify files in these mount paths, so we mount them in read only

So all we have to make sure of is that the configuration files for NGINX and the config etc. for startup script of kafdrop are in the right path.

NGINX Configuration and files

For NGINX we generated our certificates via

files:
  /etc/nginx/dhparam.pem:
    source:
      S3:
        BucketName: files.compose-x.io
        Key: labs/files-composer/dhkeys/dhparam_9.pem
    mode: 600

certificates:
  x509:
    /etc/nginx:
      keyFileName: nginx.key
      certFileName: nginx.crt

Hint

I pre-generated the DH key because that operation can take a long time and that would further delay the start of the container. I highly recommend to generate your own and re-use them across your applications.

We generated all these files in the /etc/nginx path on the files composer, but, we want to mount them in /etc/nginx/ssl on the NGINX container, as the /etc/nginx path contains already our nginx.conf and other default NGINX config that we do not want to alter.

We reflect that path in the NGINX configuration

server {
  listen 443 ssl;
  server_name _;
  ssl_certificate /etc/nginx/ssl/nginx.crt;
  ssl_certificate_key /etc/nginx/ssl/nginx.key;
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  ssl_dhparam /etc/nginx/ssl/dhparam.pem;
  ssl_ecdh_curve secp384r1;
  location / {
    proxy_pass http://kafdrop;
    add_header Strict-Transport-Security "max-age=16156800; includeSubDomains" always;
  }
}

Kafdrop configuration

For Kafdrop, we are doing the unusual thing that is to pull the JAR for it and use that. That is pretty much as if we used COPY or ADD and built a dedicate image for it. So if you prefer to, simply use the image kafdrop maintainers published, the only thing for you to do is replace the image URL.

Now for the connection to our kafka cluster, we created a new kafka user, which principal we will dub to kafdrop .

Tip

For ACLs required for kafdrop to work, have a look at acls.yaml . Note that topic are set to ANY (*) which is only here for convenience, please do set ACLs properly for your needs.

Here, I am going to assume that your kafka cluster is using SASL to connect, therefore re-use this kafka credentials template .

We expose that secret to both files-composer and the kafdrop container.

secrets:
  KAFKA_CREDS:
    x-secrets:
      Name: kafka/eu-west-1/lkc-z6v51/kafdrop.prod
      JsonKeys:
        - SecretKey: BOOTSTRAP_SERVERS
        - SecretKey: SASL_USERNAME
        - SecretKey: SASL_PASSWORD
        - SecretKey: SCHEMA_REGISTRY_URL
        - SecretKey: SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO

For files-composer, that will allow to generate a small bash script with all the configuration needed for kafdrop to work and the kafka.properties file

services:
  files-composer:
    secrets:
      - KAFKA_CREDS
    environment:
      ECS_CONFIG_CONTENT: |

        files:
          /app/kafka.properties:
            content: |
              # Properties
              ssl.endpoint.identification.algorithm=https
              sasl.mechanism=PLAIN
              sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"{{ default | env_override('SASL_USERNAME') }}\" password=\"{{ default | env_override('SASL_PASSWORD') }}\";
              security.protocol=SASL_SSL
              # EOF

            mode: 644
            context: jinja2

          /app/start.sh:
            content: |

              echo ${!PWD}
              echo {{ default | env_override('BOOTSTRAP_SERVERS') }}
              echo {{ default | env_override('SCHEMA_REGISTRY_URL') }}
              ls -l /app
              cd /app
              echo ${!PWD}

              java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED                                           \
                -jar /app/kafdrop.jar                                                                     \
                --kafka.brokerConnect={{ default | env_override('BOOTSTRAP_SERVERS') }}                   \
                --schemaregistry.connect={{ default | env_override('SCHEMA_REGISTRY_URL') }}              \
                --schemaregistry.auth={{ default | env_override('SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO') }}\
                --topic.deleteEnabled=false                                                               \
                --topic.createEnabled=false
            mode: 755
            context: jinja2

          /app/kafdrop.jar:
            source:
              S3:
                BucketName: files.compose-x.io
                Key: app-demos/kafdrop/kafdrop-3.28.0-SNAPSHOT.jar
            mode: 644

The above configuration will instruct files-composer to create 3 files:

  • /app/kafka.properties the kafka properties which include our SASL username and password

  • /app/start.sh which is the script we will use as entrypoint to get kafdrop started

  • /app/kafdrop.jar the application jar

Tip

As mentioned above, you could use the kafdrop image from dockerhub, just make sure to have the kafka.properties file mounted in the right location for kafdrop to find it, and pass on the BOOTSTRAP/Schema registry parameters.

And that’s it for the services configuration part. This might feel a little overwhelming, but all together this is very standard to what you would need to do for most services.

The AWS configurations

Let’s get the network part out of the way. In my case, I already have a VPC and network infrastructure, so I simply perform a lookup to identify these using AWS Tags

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: vpc--composex-prod
    InternalSubnets :
      Tags:
        - vpc::usage: application
        - vpc::internal: "true"
        - vpc::primary: "false"
    AppSubnets:
      Tags:
        - vpc::usage: application
        - vpc::internal: "false"
        - vpc::primary: "true"
    PublicSubnets:
      Tags:
        - vpc::usage: public
    StorageSubnets:
      Tags:
        - vpc::usage: storage

Then our SSL certificates for our ALB, plus DNS, which will create a DNS record pointing at our ALB

x-dns:
  PublicZone:
    Name: prod.compose-x.io
    Lookup:
      RoleArn: ${PROD_RO_ROLE_ARN}
  PrivateNamespace:
    Name: prod.compose-x.internal
    Lookup:
      RoleArn: ${PROD_RO_ROLE_ARN}
  Records:
    - Properties:
        Name: kafdrop.prod.compose-x.io
        Type: A
      Target: x-elbv2::kafdrop-cc-scAlb

x-acm:
  kafdrop-certs:
    MacroParameters:
      DomainNames:
    - kafdrop.prod.compose-x.io

And finally, our ALB

x-elbv2:
  kafdrop-cc-scAlb:
    Settings:
      Subnets: PublicSubnets
    Properties:
      Scheme: internet-facing
      Type: application
    MacroParameters:
      Ingress:
        ExtSources:
          - IPv4: 0.0.0.0/0
            Name: ANY
            Description: ANY
    Listeners:
      - Port: 80
        Protocol: HTTP
        DefaultActions:
          - Redirect: HTTP_TO_HTTPS
      - Port: 443
        Protocol: HTTPS
        SslPolicy: ELBSecurityPolicy-FS-1-2-Res-2020-10
        Certificates:
          - x-acm: kafdrop-certs
        Targets:
          - name: kafdrop:nginx
            access: kafdrop.prod.compose-x.io/

    Services:
      - name: kafdrop:nginx
        port: 443
        protocol: HTTPS
        healthcheck: 443:HTTPS:4:2:10:5:/actuator:200

As you can see, we are pointing the load balancer to send the traffic to our NGINX container, not the kafdrop one, by defining that our service is kafdrop:nginx

See also

The full configuration with support for cognito is available here

Deploy to AWS

So now, we have a docker-compose.yaml file and configuration that represents our production environment, aws.yaml and we are going to deploy this to AWS.

Attention

We assume that you already have some form of access to AWS credentials to interact with your AWS account.

# If you have not yet install ecs-compose-x
python -m pip install pip -U
python -m pip install ecs-composex>=0.15.7

# Initialize some settings
AWS_PROFILE=${AWS_PROFILE:-default} ecs-compose-x init
if [ -z ${AWS_ACCOUNT_ID+x} ]; then AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r .Account); fi
REGISTRY_URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION:-$AWS_DEFAULT_REGION}.amazonaws.com/
# If the repository does not exist in ECR, create
aws ecr describe-repositories --repository-name kafdrop-nginx 2>/dev/null
if [ "$?" -ne 0 ]; then REPO_NAME=kafdrop-nginx make ecr ; fi
docker-compose build
docker-compose push
AWS_PROFILE=${AWS_PROFILE:-default} ecs-compose-x plan -n kafdrop-prod -f docker-compose.yaml -f aws.yaml

Conclusion

Using docker volumes and Files Composer we generated all the files necessary by our applications, NGINX and kafdrop, to start with the wanted configuration and settings, with 0 code and only some configuration to deloy it to AWS ECS.