← Apache Spark learning pathApache Spark · Module 2 of 10
Spark architecture
Understand drivers, executors and how jobs run.
📖 Key ideas
- The driver program creates the SparkSession and plans work.
- Executors run tasks on worker nodes.
- A job splits into stages; stages split into tasks.
- Each task processes one partition.
- The cluster manager allocates resources.
💡 Real-life example: The driver is a head chef giving orders; executors are line cooks each preparing a portion.
DriverCoordinates the Spark application
ExecutorRuns tasks on workers
TaskUnit of work on one partition
StageSet of tasks without shuffle