Apache Kafka on Kubernetes with Strimzi — Part 3: Monitoring our Strimzi Kafka Cluster with Prometheus and Grafana

Prometheus

Observability is an important aspect in software engineering. Wikipedia explains it very well:

Grafana

Grafana is an open source project created by Torkel Ödegaard in 2014. Grafana connects directly to various data sources (time-series databases) and it allows you to query, visualize and explore metrics and also set alerts for different conditions based on the logs.

Our Requirements

We need to understand how our producer, consumer and the Kafka cluster behaves in different scenarios. We need to know the number of online brokers and replicas, CPU, memory and disk usage, garbage collection metrics, incoming and outcoming rate of data, etc. And also, details about topics, unconsumed messages and Consumer Lag. It’s highly important to monitor Consumer Lag in Kafka.

Adding Metrics to Kafka

All the YAMLs needed to add metrics to our Kafka cluster are available inside the examples folder inside the Strimzi package.

$ kubectl apply -f kafka-deployment.yaml -n kafka

Deploying Prometheus

Same as Kafka Cluster, We are using Operators to deploy Prometheus on Kubernetes.

Introducing Prometheus Operator

With Prometheus Operator, you can deploy and use Prometheus on Kubernetes as easy as possible. Same as the Strimzi Operator that watches for specific resources (e.g. Kafka, KafkaTopic, etc) and reacts on them, the Prometheus Operator does the same. The Operator acts on these resources:

  • Prometheus. Defines a desired Prometheus deployment.
  • ServiceMonitor. Defines how a dynamic set of services should be monitored
  • PodMonitor. Defines how a dynamic set of pods should be monitored
  • AlertManager. Defines a desired Alertmanager deployment.
  • PrometheusRule. Defines a desired Prometheus rule to be consumed by one or more Prometheus instances.
Operator workflow and relationships (source)

Deploying Prometheus Operator

Download the Prometheus Operator deployment bundle file from here. This yaml file contains all the resources needed to deploy the Prometheus Operator. Before deploying, you need to replace the default namespace used inside the yaml file with your own namespace (we are using monitoring namespace). If you are using Linux or MacOS, you can use curl and sed commands to easily download the bundle.yaml file and replace the namespace (commands are taken from Strimzi document with a slight change):

$ curl -s https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml | sed -e 's/namespace: .*/namespace: monitoring/' > bundle.yaml
$ curl -s https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml | sed -e '' 's/namespace: .*/namespace: monitoring/' > bundle.yaml
kubectl apply -f bundle.yaml -n monitoring
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS
prometheus-operator-7568575f58-2phj6 1/1 Running 0

$ kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
prometheus-operator ClusterIP None <none> 8080/TCP

Deploying Prometheus

Strimzi provides all the example resources needed for deploying Prometheus inside the examples folder. Head to the examples/metrics folder. Before deploying the main Prometheus resource, we need to add some other resources. We are going to deploy all the files inside monitoring namespace.

$ kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n monitoring
  namespaceSelector:
matchNames:
- kafka
$ kubectl apply -f strimzi-pod-monitor.yaml -n monitoring
$ kubectl apply -f prometheus-rules.yaml -n monitoring
$ kubectl apply -f prometheus.yaml -n monitoring
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS
grafana-7b7bfd6ffc-lmg6z 1/1 Running 0
prometheus-operator-7568575f58-2phj6 1/1 Running 1
prometheus-prometheus-0 3/3 Running 1

Deploying Grafana

Let’s deploy Grafana and connect it to our Prometheus instance. The Strimzi also provides a configuration file for deploying Grafana. Head to the examples/metrics/grafana-install folder and deploy the grafana.yaml file.

$ kubectl apply -f grafana.yaml -n monitoring
$ kubectl port-forward svc/grafana 3000:3000 -n monitoring
$ kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
grafana ClusterIP 10.103.151.53 <none> 3000/TCP
prometheus-operated ClusterIP None <none> 9090/TCP
prometheus-operator ClusterIP None <none> 8080/TCP
strimzi-kafka.json
strimzi-kafka-exporter.json
strimzi-operators.json
strimzi-zookeeper.json
Grafana Dashboard for our Kafka Cluster
Grafana Dashboard for our Kafka Cluster (Kafka Exporter)

Conclusion

In this blog post series, we have learnt how to Deploy a Kafka cluster using Strimzi, writing Producers and Consumers with Go and Scala, deploying them in Kubernetes and connecting to our Kafka cluster and in this post, we have learnt how to add metrics to our Kafka and ZooKeeper, deploy Prometheus and Grafana and configure it to scrape metrics from our Pods.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sina Nourian

Sina Nourian

CTO at Daneshgar Technology Co. Ltd | Interested in Distributed Systems, Cloud Computing, Serverless computing, Data Stream Processing and Microservices