Training Trains Internship
Student login
← Apache Spark learning path
Apache Spark · Module 1 of 10

Introduction to Spark

Apache Spark is a fast engine for large-scale data processing.

📖 Key ideas

  1. Spark processes data in memory across a cluster.
  2. Up to 100x faster than disk-based MapReduce for some workloads.
  3. APIs in Python (PySpark), Scala, Java, SQL and R.
  4. Modules: Spark SQL, Structured Streaming, MLlib, GraphX.
  5. Runs on YARN, Kubernetes, standalone or Databricks.
💡 Real-life example: Spark is a team of chefs who keep ingredients on the counter (memory) instead of walking to the store (disk).
SparkDistributed data processing engine
PySparkPython API for Spark
In-memory processingKeeping data in RAM
DatabricksManaged Spark platform