Streamlining Notification Processing with Kafka, MinIO, and Python

Step-by-step guide on how to process Kafka notifications in Python

Photo by Yannick Pipke on Unsplash

Event-driven architecture has gained widespread popularity for enabling loosely coupled applications to communicate effectively.

In this tutorial, we'll demonstrate this architecture by configuring Minio to send topic notifications via Kafka. Additionally, we'll develop a straightforward Kafka listener in Python to consume these event records.

Let’s get started!

Prepare The MinIO-Kafka Environment

First, let’s prepare our local development infrastructure. The easiest way to get started is by running a docker-compose.yml file:

version: "3"

volumes:
  data:
networks:
    kafka-net:
      driver: bridge
services:
  minio:
    image: minio/minio
    container_name: minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    restart: always
    volumes:
      - data:/data
    networks:
      - kafka-net
    depends_on:
      - kafka
    environment:
      MINIO_NOTIFY_KAFKA_ENABLE: 'on'
      MINIO_NOTIFY_KAFKA_BROKERS: "kafka:29092"
      MINIO_NOTIFY_KAFKA_TOPIC:  'my-notifications'
      MINIO_NOTIFY_WEBHOOK_QUEUE_DIR: /home/events

  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - kafka-net

  kafka:
    image: confluentinc/cp-kafka:7.0.0
    container_name: kafka
    depends_on:
      - zookeeper
    networks:
      - kafka-net
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

We create containers for minio, kafka, and zookeeper.

• The MinIO service acts as a producer and sends notification data to Kafka.

• Kafka is a streaming system that will consume MinIO event data and handle the records accordingly.

• Zookeeper is used to track the status of nodes in the Kafka cluster and maintain a list of Kafka topics, partitions, etc.

Important notes:

• MinIO’s environment configuration defines the Kafka properties. For example, broker port, notification topic name.

• The Kafka broker is exposed on port 9092. So, we can refer to it for local testing.

• All services should run in the same network, in this case, kafka-net.

Configure The MinIO Notifications