Kafka

Overview

Apache Kafka can support the transmission of massive amounts of data.It has extensive applications in both offline and real-time message proccessing business systems.

1. Log Collection Collect logs from various services and provide them to various consumers, such as Hadoop, Hbase and Solr,through kafka interface service.
2. Operational Metrics kafka is also frequently used to record operational monitoring data.This includes collecting data from various distributed applications and providing centralized feedback on various operations, such as alarms and reports.
3. Stream Processing Using Spark Streaming and Storm

feature

1. High Throughput and Low Latency Kafka can process hundreds of thousands of messages per second with a latency as low as a few milliseconds.
2. Scalability Kafka clusters support hot expansion, allowing for easy scalability.
3. Durability and Reliability Messages are persisted to local disk and data replication is supported to prevent data loss.
4. Fault Tolerance Kafka allows for node failures within the cluster (if the replication factor is n, then n-1 node failures are tolerated).
5. High Concurrency Kafka supports thousands of clients reading and writing simultaneously.

install

here we using docker compose to build the single kafka with zookeeper

version: "3"
services:
  zookeeper:
    image: wurstmeister/zookeeper

  kafka-1:
    container_name: kafka-1
    image: wurstmeister/kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      HOST_IP: 10.32.1.200
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: 10.32.1.200
      KAFKA_ADVERTISED_PORT: 9092
    volumes:
      - /etc/localtime:/etc/localtime
    depends_on:
      - zookeeper

docker-compose -f kafka.yml up -d

then exec docker ps ,we can see

that means we have install single kafka cluster success.

also, here I give the cluster docker-compose file

version: "3"
services:
  zookeeper:
    image: wurstmeister/zookeeper

  kafka-1:
    container_name: kafka-1
    image: wurstmeister/kafka
    ports:
      - 10903:9092
    environment:
      KAFKA_BROKER_ID: 1
      HOST_IP: 192.168.10.30
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: 192.168.10.30
      KAFKA_ADVERTISED_PORT: 10903
    volumes:
      - /etc/localtime:/etc/localtime
    depends_on:
      - zookeeper
  kafka-2:
    container_name: kafka-2
    image: wurstmeister/kafka
    ports:
      - 10904:9092
    environment:
      KAFKA_BROKER_ID: 2
      HOST_IP: 192.168.10.30
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: 192.168.10.30
      KAFKA_ADVERTISED_PORT: 10904
    volumes:
      - /etc/localtime:/etc/localtime
    depends_on:
      - zookeeper
  kafka-3:
    container_name: kafka-3
    image: wurstmeister/kafka
    ports:
      - 10905:9092
    environment:
      KAFKA_BROKER_ID: 3
      HOST_IP: 192.168.10.30
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST_NAME: 192.168.10.30
      KAFKA_ADVERTISED_PORT: 10905 #改
    volumes:
      - /etc/localtime:/etc/localtime
    depends_on:
      - zookeeper