Apache Kafka on Kubernetes with Strimzi — Part 2: Creating Producer and Consumer apps using Go and Scala and deploying on Kubernetes

The Producer

github.com/segmentio/kafka-go // Kafkago.uber.org/zap // Logginggoogle.golang.org/protobuf // Message Serializationgithub.com/urfave/cli/v2 // CLI
  • Kafka Bootstrap servers
  • Topic name
  • Buffer size (for batching messages)
  • Sleep time between producing messages
$ docker build -t nrsina/strimzi-producer:v1 .
Deploying Strimzi Producer app in Kubernetes

The Consumer

Delivery Semantics

  • At most once. This is the most relaxed semantic. With this semantic, message might get processed one time or not at all (message can get lost and it cannot be redelivered). This semantic happens when the commitment of the offsets has nothing to to with the processing of the message.
    By setting the enable.auto.commit property of the consumer to true, the consumer offsets will be periodically committed in the background. So we have no control on the offsets and if the process of a message fails, the consumer can’t request the message from the topic again since the offset has been committed before and moved passed the failed message.
  • At least once. This semantic ensures that a message may never get lost but may get redelivered. So the message might get processed more than one time (but never lost). When this delivery semantic is needed, the consumer has to take control of the offset committing.
    The commitment should happen after the processing of the message is done. So if the process fails, no offset will be committed. And on the next pull from the topic, the failed message will get redelivered again.
    Bear in mind that with this approach, the processing of a message may be successful but the committing fails for any reason. So then again the message will be redelivered. This is because our processing (for example, persisting in a database) and committing the offsets in Kafka are not in a single transaction. (The enable.auto.commit property should be false since we need to take control of the offset committing).
  • Exactly once. This semantic guarantees that the processing of the message happens only once. This is the most strong semantic. When we want to have exactly once semantics, the process and committing of the message should happen in a single transaction.
    For example, if we are persisting the message in PostgreSQL, we should also persist the offsets in the same database so we can easily wrap both of them in a single transaction. If the persisting of the offset fails, the persisting of the message will also roll backs and vice versa.

Creating the Consumer

akka {
loglevel = DEBUG
}
akka.management {
http {
hostname = "127.0.0.1"
hostname = ${?AKKA_MANAGEMENT_HTTP_HOST}
bind-hostname = "0.0.0.0"
port = 8558
port = ${?AKKA_MANAGEMENT_HTTP_PORT}
}
}
strimzi-consumer {
bootstrap-servers = "localhost:9092"
bootstrap-servers = ${?SC_BOOTSTRAP_SERVER}
group-id = "my-group"
group-id = ${?SC_GROUP_ID}
enable-auto-commit = "false" // false for at-least-once semantics
enable-auto-commit = ${?SC_AUTO_COMMIT}
auto-offset-reset = "latest" // other value: earliest
auto-offset-reset = ${?SC_OFFSET_RESET}
topic = "my-topic"
topic = ${?SC_TOPIC_NAME}
parallelism = 10 // batch processing of consumed messages
parallelism = ${?SC_PARALLELISM_NUM}
sleep-time-ms = 100 // simulates computation of a message
sleep-time-ms = ${?SC_SLEEP_TIME}
}
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.6")
$ sbt docker:publishLocal
$ sbt docker:stage
Deploying Strimzi Consumer app in Kubernetes

Conclusion

--

--

--

CTO at Daneshgar Technology Co. Ltd | Interested in Distributed Systems, Cloud Computing, Serverless computing, Data Stream Processing and Microservices

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Our Journey: Migrating from Adobe ColdFusion to Lucee

Part 2 Add OTP for Concurrency and Fault Tolerance

Learning Python via Match of the Day commentary

From Monolith to Microservices: the Serverless approach

Industry use cases of OpenShift

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sina Nourian

Sina Nourian

CTO at Daneshgar Technology Co. Ltd | Interested in Distributed Systems, Cloud Computing, Serverless computing, Data Stream Processing and Microservices

More from Medium

Istio quick guide

Kubernetes scaler for dedicated stateful servers

Kafka on Kubernetes

Rancher CIS operator