Running Things
These lessons require several services running on your laptop (Kafka, Schema Registry, etc). There are many ways to run them; we’ll run them in Minikube. To set up Minikube, please see this tutorial.
The Kubernetes manifests that run these services in Minikube are currently part of the Banno Conversations repository. Please clone that repo now.
You should also download Confluent Open Source and extract it somewhere on your laptop. It has useful utilities that we’ll use later.
If you need help with anything in this lesson, please ask in #sig-kafka.
Setting Up
The banno-conversations/minikube/00-kafka directory contains all of the Kubernetes manifests that we need to run everything. You can examine them if you want, but since these lessons are about using Kafka, instead of operating Kafka, we won’t discuss them further.
To run everything:
minikube start
sudo route -n add 10.0.0.0/8 `minikube ip`
sudo route -n add 172.17.0.0/24 `minikube ip`
cd banno-conversations/minikube
kubectl apply -R -f 00-kafka
That’s it!
That kubectl apply told Kubernetes to run many things; it may take a few minutes for Docker images to pull, and dependent things to run, etc. Kubernetes will automatically restart containers until it’s all working.
A convenient way to see when things are ready is watch kubectl get pods -n kafka. You should eventually see something like:
NAME READY STATUS RESTARTS AGE
cp-kafka-rest-64dd7f575f-qmp2j 1/1 Running 2 4m
cp-schema-registry-756694dfbc-b4qxx 1/1 Running 2 4m
kafka-0 1/1 Running 3 4m
kafka-1 1/1 Running 0 2m
kafka-2 1/1 Running 0 2m
kafka-topics-ui-6fd8849577-jmkw5 1/1 Running 0 4m
pzoo-0 1/1 Running 0 4m
pzoo-1 1/1 Running 0 3m
pzoo-2 1/1 Running 0 3m
schema-registry-ui-6d67fc56bc-jk6k9 1/1 Running 0 4m
zoo-0 1/1 Running 0 4m
zoo-1 1/1 Running 0 3m
You can also run minikube dashboard, select the kafka namespace, and keep refreshing the page until everything has green checkmarks.
Tearing Down
kubectl delete --now -R -f 00-kafka will stop everything. It may take awhile; you can watch kubectl get namespace until kafka disappears.
You can run kubectl apply -R -f 00-kafka to run everything again.
Testing it Out
Open http://kafka-topics-ui.kafka.svc.cluster.local:8001, do you see the Kafka Topics UI?
Open http://schema-registry-ui.kafka.svc.cluster.local:8000, do you see the Schema Registry UI?
In your confluent-4.0.0 dir, run the following:
bin/kafka-topics --zookeeper zookeeper.kafka.svc.cluster.local --list
You should see (at least) the following topic:
_schemas
Now let’s quickly write some records to a topic:
bin/kafka-console-producer --broker-list kafka-0.broker.kafka.svc.cluster.local:9092 --topic lesson1-topic1
At the > prompt, type something and hit Enter. Do that a few times. Each line becomes a record sent to the lesson1-topic1 Kafka topic. Then Ctrl+C to exit the console producer. It’s OK if you see a warn log about “Error while fetching metadata”, just ignore it.
Now run the consumer, as follows. Do you see the messages you typed previously? It may take a few seconds.
bin/kafka-console-consumer --bootstrap-server kafka-0.broker.kafka.svc.cluster.local:9092 --topic lesson1-topic1 --from-beginning
You can also view this lesson1-topic1 topic in the Kafka Topics UI, and if you select it you should see the messages you typed.
Summary
In this lesson, we ran Kafka (and several other services) in Minikube, and made sure everything worked properly. If anything didn’t work as expected, please ask in #sig-kafka before proceeding to other lessons.
Next, we’ll start writing code that uses Kafka.