← Apache Spark learning pathApache Spark · Module 5 of 10
Partitioning and performance
Make Spark jobs fast and efficient.
📖 Key ideas
- Partitions control parallelism.
- repartition() reshuffles; coalesce() reduces partitions without full shuffle.
- Cache/persist reused DataFrames.
- Broadcast small tables in joins to avoid shuffles.
- Watch for data skew and too many small files.
💡 Real-life example: Partitions are lanes on a highway: too few cause traffic jams, too many waste road.
PartitionChunk of data processed by one task
cache()Keeps data in memory for reuse
Broadcast joinSends small table to all executors
Data skewUneven data across partitions