← Apache Spark learning pathApache Spark · Module 6 of 10
Reading and writing data
Load and save data in many formats and sources.
📖 Key ideas
- Formats: CSV, JSON, Parquet, ORC, Avro, Delta.
- Sources: HDFS, S3, GCS, ADLS, JDBC databases, Kafka.
- Write modes: overwrite, append, ignore, errorifexists.
- partitionBy("year","month") writes partitioned folders.
- Parquet is the default efficient choice.
💡 Real-life example: Choosing a file format is choosing a container: Parquet is a vacuum-sealed box, CSV is a paper bag.
ParquetColumnar file format
Write modeBehaviour when data exists
partitionByWrites folders by column values
JDBCConnects to relational databases