← Big Data & Hadoop learning pathBig Data & Hadoop · Module 8 of 10
Data ingestion and streaming
Move data into big data systems in batches and real time.
📖 Key ideas
- Batch ingestion moves data on a schedule.
- Streaming processes events continuously.
- Apache Kafka is a distributed event streaming platform.
- Producers write to topics; consumers read from them.
- Stream processors: Spark Structured Streaming, Flink, Kafka Streams.
💡 Real-life example: Kafka is a busy railway station: trains (events) arrive on platforms (topics) and passengers (consumers) pick them up.
KafkaDistributed event streaming platform
TopicNamed stream of events
ProducerWrites events
ConsumerReads events