← Apache Spark learning pathApache Spark · Module 4 of 10
DataFrames and Spark SQL
Work with structured data using DataFrames and SQL.
📖 Key ideas
- spark.read.csv / parquet / json loads DataFrames.
- Operations: select, filter, withColumn, groupBy, agg, join.
- Register a view: df.createOrReplaceTempView("sales") then spark.sql(...).
- Catalyst optimises query plans.
- show() and printSchema() inspect data.
💡 Real-life example: Spark DataFrames are pandas tables that can stretch across hundreds of machines.
DataFrameDistributed table with schema
withColumnAdds or replaces a column
Temp viewName for SQL queries
CatalystSpark query optimiser