← Apache Spark learning pathApache Spark · Module 7 of 10
Structured Streaming
Process real-time data with Spark.
📖 Key ideas
- Structured Streaming treats a stream as an unbounded table.
- Sources: Kafka, files, sockets.
- Same DataFrame API as batch.
- Output modes: append, update, complete.
- Checkpoints give fault tolerance; watermarks handle late data.
💡 Real-life example: Structured Streaming is a table that keeps growing row by row, and your query keeps updating.
Unbounded tableStream modelled as growing table
CheckpointSaved progress for recovery
WatermarkLimit for late data
Micro-batchSmall batch processed repeatedly