Apache Kafka on Kubernetes with Strimzi — Part 3: Monitoring our Strimzi Kafka Cluster with Prometheus and Grafana

Prometheus

Grafana

Our Requirements

Adding Metrics to Kafka

$ kubectl apply -f kafka-deployment.yaml -n kafka

Deploying Prometheus

Introducing Prometheus Operator

  • Prometheus. Defines a desired Prometheus deployment.
  • ServiceMonitor. Defines how a dynamic set of services should be monitored
  • PodMonitor. Defines how a dynamic set of pods should be monitored
  • AlertManager. Defines a desired Alertmanager deployment.
  • PrometheusRule. Defines a desired Prometheus rule to be consumed by one or more Prometheus instances.
Operator workflow and relationships (source)

Deploying Prometheus Operator

$ curl -s https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml | sed -e 's/namespace: .*/namespace: monitoring/' > bundle.yaml
$ curl -s https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml | sed -e '' 's/namespace: .*/namespace: monitoring/' > bundle.yaml
kubectl apply -f bundle.yaml -n monitoring
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS
prometheus-operator-7568575f58-2phj6 1/1 Running 0

$ kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
prometheus-operator ClusterIP None <none> 8080/TCP

Deploying Prometheus

$ kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n monitoring
  namespaceSelector:
matchNames:
- kafka
$ kubectl apply -f strimzi-pod-monitor.yaml -n monitoring
$ kubectl apply -f prometheus-rules.yaml -n monitoring
$ kubectl apply -f prometheus.yaml -n monitoring
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS
grafana-7b7bfd6ffc-lmg6z 1/1 Running 0
prometheus-operator-7568575f58-2phj6 1/1 Running 1
prometheus-prometheus-0 3/3 Running 1

Deploying Grafana

$ kubectl apply -f grafana.yaml -n monitoring
$ kubectl port-forward svc/grafana 3000:3000 -n monitoring
$ kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
grafana ClusterIP 10.103.151.53 <none> 3000/TCP
prometheus-operated ClusterIP None <none> 9090/TCP
prometheus-operator ClusterIP None <none> 8080/TCP
strimzi-kafka.json
strimzi-kafka-exporter.json
strimzi-operators.json
strimzi-zookeeper.json
Grafana Dashboard for our Kafka Cluster
Grafana Dashboard for our Kafka Cluster (Kafka Exporter)

Conclusion

--

--

--

CTO at Daneshgar Technology Co. Ltd | Interested in Distributed Systems, Cloud Computing, Serverless computing, Data Stream Processing and Microservices

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to Use the Magical @staticmethod, @classmethod, and @property Decorators in Python

T-SQL: How To Write A RIGHT JOIN and Why You Should Be Aware of Them

Elfinder Vulnerability Fix for PHP

Kotlin Features You Should Use Now

Processing Data from MQ with Spark Streaming: Part 4 - Implementation details for Spark MQ…

Introducing BF: The Great, the Glorious, and the Completely Silly

Техническое введение в сеть Axelar Network Что представляет собой Axelar Network?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sina Nourian

Sina Nourian

CTO at Daneshgar Technology Co. Ltd | Interested in Distributed Systems, Cloud Computing, Serverless computing, Data Stream Processing and Microservices

More from Medium

Open Application Model (OAM) — Bringing some sanity to application deployment on kubernetes (k8s).

Kubernetes, rke2, containerd, Elasticsearch, limits.conf , Increasing the # of open files.

Easily Manage your Application Shipment With Differentiated Configuration in Multi-Cluster

A Kubernetes cluster from Virtualbox