Training Trains Internship
Student login
← Data Engineering & ETL learning path
Data Engineering & ETL · Module 8 of 10

Streaming pipelines

Build real-time data flows.

📖 Key ideas

  1. Streaming pipelines process events as they arrive.
  2. Kafka (or Pub/Sub, Kinesis) transports events.
  3. Processors: Flink, Spark Structured Streaming, Kafka Streams.
  4. Handle late and duplicate events.
  5. Use streaming only when freshness truly matters.
💡 Real-life example: Streaming is a live news ticker; batch is the morning newspaper.
EventRecord of something that happened
KinesisAWS streaming service
Late dataEvents arriving after expected time
Exactly-onceEach event processed once