Kafka

Introduction

• Created by Linkedin.

• It is an open source project maintained by Confluent, IBM, Cloudera.

• A distributed, resilient architecture, fault tolerant

• Easily scalable by adding brokers i.e., horizontal scaling.

  • Can scale to 100s of brokers

  • Can handle millions of requests/second

• Highly performant with latency of less than 10ms in real time.

Use cases

• Only used as transportation mechanism with some of uses cases being,

  • Messaging system

  • Activity tracking system

  • Application logs gathering

  • Gather metrics from different systems

  • Stream processing with Streaming API

  • Decoupling of system dependencies

  • Microservice pub-sub requirements

  • Integration with Spark, Flink, Storm, Hadoop etc