This is part of the Charmed Kafka K8s Tutorial. Please refer to this page for more information and the overview of the content.
Deploy Charmed Kafka K8s (and Charmed ZooKeeper K8s)
To deploy Charmed Kafka K8s, all you need to do is run the following commands, which will automatically fetch Kafka and ZooKeeper charms from Charmhub and deploy them to your model. For example, to deploy a five Zookeeper unit and three Kafka unit cluster, you can simply run:
$ juju deploy zookeeper-k8s -n 3
$ juju deploy kafka-k8s -n 3 --trust
After this, it is necessary to connect them:
$ juju relate kafka-k8s zookeeper-k8s
Juju will now fetch Charmed Kafka K8s and Charmed ZooKeeper K8s and begin deploying it to the local MicroK8s. This process can take several minutes depending on how provisioned (RAM, CPU, etc) your machine is. You can track the progress by running:
juju status --watch 1s
This command is useful for checking the status of Charmed ZooKeeper K8s and Charmed Kafka K8s. Some of the helpful information it displays include IP pods addresses, ports, state, etc. The command updates the status of the cluster every second and as the application starts you can watch the status and messages of Charmed Kafka K8s and Charmed ZooKeeper K8s change.
Wait until the application is ready - when it is ready, juju status --watch 1s will show:
...
Model Controller Cloud/Region Version SLA Timestamp
tutorial microk8s microk8s/localhost 3.1.5 unsupported 17:22:21+02:00
App Version Status Scale Charm Channel Rev Address Exposed Message
kafka-k8s active 3 kafka-k8s 3/beta 46 10.152.183.237 no
zookeeper-k8s active 3 zookeeper-k8s 3/beta 37 10.152.183.134 no
Unit Workload Agent Address Ports Message
kafka-k8s/0 active idle 10.1.36.78
kafka-k8s/1 active idle 10.1.36.80
kafka-k8s/2* active idle 10.1.36.79
zookeeper-k8s/0 active idle 10.1.36.84
zookeeper-k8s/1* active idle 10.1.36.86
zookeeper-k8s/2 active idle 10.1.36.85
To exit the screen with juju status --watch 1s, enter Ctrl+c.
Access Kafka cluster
To watch the process, juju status can be used. Once all the units show as active|idle the credentials to access a broker can be queried with:
juju run kafka-k8s/leader get-admin-credentials
The output of the previous command is something like this:
Running operation 1 with 1 task
- task 2 on unit-kafka-k8s-2
Waiting for task 2...
client-properties: |-
security.protocol=SASL_PLAINTEXT
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="0FIQ5QxSaNfl1bXHtV5dyttb21Nbzmpp";
sasl.mechanism=SCRAM-SHA-512
bootstrap.servers=kafka-k8s-1.kafka-k8s-endpoints:9092,kafka-k8s-2.kafka-k8s-endpoints:9092,kafka-k8s-0.kafka-k8s-endpoints:9092
password: 0FIQ5QxSaNfl1bXHtV5dyttb21Nbzmpp
username: admin
Providing you the username and password of the Kafka cluster admin user.
IMPORTANT Note that when no other application is related to Kafka, the cluster is secured-by-default and external listeners (bound to port
9092) are disabled, thus preventing any external incoming connection.
Nevertheless, it is still possible to run a command from within the Kafka cluster. To do so, log in into one of the Kafka container in one of the units
juju ssh --container kafka kafka-k8s/leader /bin/bash
The Charmed Kafka K8s image ships with the Apache Kafka bin/*.sh commands, that can be found under /opt/kafka/bin/.
They can be used to do various administrative tasks, e.g., bin/kafka-config.sh to update cluster configuration, bin/kafka-topics.sh for topic management, and many more!
Within the image you can also find a client.properties file that already provides the relevant settings to connect to the cluster using the CLI:
export CLIENT_PROPERTIES=/etc/kafka/client.properties
Since we don’t have any client applications related yet and therefore external listeners are initially closed, if you wish to run a command from the cluster you ought to use the internal listeners exposed at ports 19092.
export INTERNAL_LISTENERS=kafka-k8s-1.kafka-k8s-endpoints:19092,kafka-k8s-2.kafka-k8s-endpoints:19092,kafka-k8s-0.kafka-k8s-endpoints:19092
We are now ready to perform some administrative tasks. For example, to create a topic, you can run:
/opt/kafka/bin/kafka-topics.sh \
--create --topic test_topic \
--bootstrap-server $INTERNAL_LISTENERS \
--command-config $CLIENT_PROPERTIES
You can similarly then list the topic:
/opt/kafka/bin//kafka-topics.sh \
--list \
--bootstrap-server $INTERNAL_LISTENERS \
--command-config $CLIENT_PROPERTIES
making sure the topic was successfully created.
You can finally delete the topic:
/opt/kafka/bin/kafka-topics.sh \
--delete --topic test_topic \
--bootstrap-server $INTERNAL_LISTENERS \
--command-config $CLIENT_PROPERTIES
What’s next?
However, although the commands above can run within the cluster, it is generally recommended during operations to enable external listeners and use these for running the admin commands from outside the cluster. To do so, as we will see in the next section, we will deploy a data-integrator charm and relate it to Kafka.