1. 2018-05-05 - Run Kafka Manager behind a proxy; Tags: Run Kafka Manager behind a proxy

    Run Kafka Manager behind a proxy

    I have two ramp up for two data-centers each a Apache Kafka cluster. Maintaining the cluster with the Kafka Manager GUI by Yahoo is pretty comfortable. Since both data-centers are protected by firewall, a web-proxy must forward it for my remote access.

    The basic problem is, that the Kafka Manager GUI has the context path / and forwarding it to the url /kafka-manager/ will cause a malfunction.

    In the application.properties you can change it and I wrote my nginx rule.

    One instance of Kafka Manager is enough to maintaining both Kafka clusters. For a high availability scenario, both data centers have an instance running. If one data center is down, e.g. power outage, the other instance is still reachable for administering the cluster. The risk though is pretty low, since you can use the builtin Kafka cli tools to administrate the cluster and topics.


    Leave a comment

  2. 2018-03-13 - Apache Kafka Management and Monitoring; Tags: Apache Kafka Management and Monitoring

    Apache Kafka Management and Monitoring

    Monitoring for Apache Kafka is crucial to know the moment when to act or scale out your Kafka clusters. Besides the CLI commands, metrics are also accessible over JMX and jconsole. A more convenient way is to have a GUI that displays it. This post focus on Kafka Manager, a administration GUI for Kafka by Yahoo.

    Other solutions like elasticsearch and metricbeat kafka module by elastic also exists, but the module itself is beta and subject to change. Therefore I discard the idea using Kibana and Elasticsearch.

    Enable JMX

    JMX must be enabled in order to get them displayed in kafka-manager.

    In my ansible playbook.yml for the Apache Kafka Docker container, add these environment variables.

    KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote \
      -Dcom.sun.management.jmxremote.authenticate=false \
      -Dcom.sun.management.jmxremote.ssl=false \
      -Djava.rmi.server.hostname={{ansible_hostname}} \
    JMX_PORT: 9099

    Pay attention to the exposed JMX port. You can choose any port. I think port 9099 is fitting, as the default exposed port is 9092.

    Kafka Manager

    If Apache ZooKeeper and Apache Kafka is operational I start a separate docker container for kafka-manager. The most recent docker image for kafka-manager is by sheepkiller.

    Run interactive, adjust the ZK_HOSTS to your needs.

    docker run -it --rm -p 9000:9000 \
    -e ZK_HOSTS="alpha:2181,beta:2181,gamma:2181" \
    sheepkiller/kafka-manager -Djava.net.preferIPv4Stack=true

    Or run detached for production mode

    docker run -d --name kafka-manager -p 9000:9000 \
    -e ZK_HOSTS="alpha:2181,beta:2181,gamma:2181" \
    --restart always \
    --log-driver json-file --log-opt max-size=10m \
    sheepkiller/kafka-manager -Djava.net.preferIPv4Stack=true

    You can access the GUI on the docker host over port 9000. See below exposed metrics.

    Kafka Manager Admin Interface


    Leave a comment