1. 2018-03-13 - Apache Kafka Management and Monitoring; Tags: Apache Kafka Management and Monitoring

    Apache Kafka Management and Monitoring

    Monitoring for Apache Kafka is crucial to know the moment when to act or scale out your Kafka clusters. Besides the CLI commands, metrics are also accessible over JMX and jconsole. A more convenient way is to have a GUI that displays it. This post focus on Kafka Manager, a administration GUI for Kafka by Yahoo.

    Other solutions like elasticsearch and metricbeat kafka module by elastic also exists, but the module itself is beta and subject to change. Therefore I discard the idea using Kibana and Elasticsearch.

    Enable JMX

    JMX must be enabled in order to get them displayed in kafka-manager.

    In my ansible playbook.yml for the Apache Kafka Docker container, add these environment variables.

    KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote \
      -Dcom.sun.management.jmxremote.authenticate=false \
      -Dcom.sun.management.jmxremote.ssl=false \
      -Djava.rmi.server.hostname={{ansible_hostname}} \
    JMX_PORT: 9099

    Pay attention to the exposed JMX port. You can choose any port. I think port 9099 is fitting, as the default exposed port is 9092.

    Kafka Manager

    If Apache ZooKeeper and Apache Kafka is operational I start a separate docker container for kafka-manager. The most recent docker image for kafka-manager is by sheepkiller.

    Run interactive, adjust the ZK_HOSTS to your needs.

    docker run -it --rm -p 9000:9000 \
    -e ZK_HOSTS="alpha:2181,beta:2181,gamma:2181" \
    sheepkiller/kafka-manager -Djava.net.preferIPv4Stack=true

    Or run detached for production mode

    docker run -d --name kafka-manager -p 9000:9000 \
    -e ZK_HOSTS="alpha:2181,beta:2181,gamma:2181" \
    --restart always \
    --log-driver json-file --log-opt max-size=10m \
    sheepkiller/kafka-manager -Djava.net.preferIPv4Stack=true

    You can access the GUI on the docker host over port 9000. See below exposed metrics.

    Kafka Manager Admin Interface


    Leave a comment

  2. 2015-06-18 - Jolokia - JMX with JSON over HTTP; Tags: Jolokia - JMX with JSON over HTTP

    Jolokia - JMX with JSON over HTTP

    Jolokia is remote JMX with JSON over HTTP. It provides several agents to expose JMX MBeans. OSGi is one of them :smile:. Using DropWizard Metrics with the JmxReporter, you can easily access the metrics. If you use the Java EE agent (deploy the jolokia.war to your Java EE application server), it enables you to access the metric under the given MBeans objectName, e.g. The context-root can be simplified to jolokia via the web.xml.

    curl http://localhost:8080/jolokia-war-1.3.1/read/metrics:name=trx.process.approved
    • read is the operation
    • metrics:name=trx.process.approved the JMX objectName of the MBean

    Example output of processed transaction that haven been approved.

    class="language-json">{ "request": { "mbean": "metrics:name=trx.process.approved", "type": "read" }, "value": { "Count": 2 }, "timestamp": 1434618683, "status": 200 }