Apache Kafka on Kubernetes with Strimzi — Part 1: Creating and Deploying a Strimzi Kafka Cluster

Introduction

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.

People who run workloads on Kubernetes often like to use automation to take care of repeatable tasks. The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides.

Kubernetes, Operator Pattern

What is Strimzi?

Strimzi Operators

Operators within the Strimzi architecture (source)

Deploying Kafka using Strimzi

Deploying the Cluster Operator

$ kubectl create ns kafka
$ sed -i 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml
$ sed -i '' 's/namespace: .*/namespace: kafka/' install/cluster-operator/*RoleBinding*.yaml

Single Namespace

Multiple Namespaces

# ...
env:
- name: STRIMZI_NAMESPACE
value: kafka-cluster-1,kafka-cluster-2,kafka-cluster-3
# ...
$ kubectl apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n watched-namespace$ kubectl apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n watched-namespace$ kubectl apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n watched-namespace

All Namespaces

# ...
env:
- name: STRIMZI_NAMESPACE
value: "*"
# ...
$ kubectl create clusterrolebinding strimzi-cluster-operator-namespaced --clusterrole=strimzi-cluster-operator-namespaced --serviceaccount my-cluster-operator-ns:strimzi-cluster-operator

$ kubectl create clusterrolebinding strimzi-cluster-operator-entity-operator-delegation --clusterrole=strimzi-entity-operator --serviceaccount my-cluster-operator-ns:strimzi-cluster-operator

$ kubectl create clusterrolebinding strimzi-cluster-operator-topic-operator-delegation --clusterrole=strimzi-topic-operator --serviceaccount my-cluster-operator-ns:strimzi-cluster-operator
$ kubectl apply -f install/cluster-operator -n kafka
$ kubectl get deployments -n kafkaNAME                         READY   UP-TO-DATE   AVAILABLE   AGE
strimzi-cluster-operator 1/1 1 1 40m

Deploying the Kafka Cluster

Simple Kafka deployment YAML for Strimzi

This is intentional. We do not want third party applications use the Zookeeper because it could have negative impact on Kafka cluster availability and because Zookeeper is quite hard to secure. If you really need a workaround, you can use this deployment which can proxy Zookeeper (it expects your Kafka cluster to be named my-cluster — if you use different name you should change it in the fields where my-cluster is used). Afterwards you should be just able to connect to zoo-entrance:2181.

scholzj, Strimzi Github

$ kubectl apply -f kafka-deployment.yaml -n kafka
$ kubectl get deployments -n kafka

Managing Topics

Creating a Topic in Strimzi Kafka Cluster

Testing our Kafka Cluster

$ kubectl get svc -n kafka
Kafka Cluster service name (address)
$ kubectl run kafka-producer -ti --image=strimzi/kafka:0.20.0-rc1-kafka-2.6.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --broker-list my-cluster-kafka-bootstrap.kafka:9092 --topic my-topic
$ kubectl run kafka-consumer -ti --image=strimzi/kafka:0.20.0-rc1-kafka-2.6.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap.kafka:9092 --topic my-topic --from-beginning
$ kafka-console-producer.sh --broker-list 192.168.99.105:30825 --topic my-topic$ kafka-console-consumer.sh --bootstrap-server 192.168.99.105:30825 --topic my-topic --from-beginning

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store