I have two ramp up for two data-centers each a Apache Kafka cluster. Maintaining the cluster with the Kafka Manager GUI by Yahoo is pretty comfortable. Since both data-centers are protected by firewall, a web-proxy must forward it for my remote access.
The basic problem is, that the Kafka Manager GUI has the context path / and forwarding it to the url /kafka-manager/ will cause a malfunction.
In the application.properties you can change it and I wrote my nginx rule.
One instance of Kafka Manager is enough to maintaining both Kafka clusters. For a high availability scenario, both data centers have an instance running. If one data center is down, e.g. power outage, the other instance is still reachable for administering the cluster. The risk though is pretty low, since you can use the builtin Kafka cli tools to administrate the cluster and topics.
Monitoring for Apache Kafka is crucial to know the moment when to act or scale out your Kafka clusters. Besides the CLI commands, metrics are also accessible over JMX and jconsole. A more convenient way is to have a GUI that displays it. This post focus on Kafka Manager, a administration GUI for Kafka by Yahoo.
Other solutions like elasticsearch and metricbeatkafka module by elastic also exists, but the module itself is beta and subject to change. Therefore I discard the idea using Kibana and Elasticsearch.
JMX must be enabled in order to get them displayed in kafka-manager.
In my ansible playbook.yml for the Apache Kafka Docker container, add these environment variables.