Training Trains Internship
Student login
← Data Engineering & ETL learning path
Data Engineering & ETL · Module 5 of 10

Data lakes and file formats

Store raw and processed data in lakes.

📖 Key ideas

  1. Data lakes store raw files cheaply in object storage (S3, GCS, ADLS).
  2. Organise by zones: raw/bronze, clean/silver, curated/gold.
  3. Use columnar formats: Parquet, ORC.
  4. Table formats (Delta, Iceberg, Hudi) add ACID and time travel.
  5. Catalogues track what data exists.
💡 Real-life example: A data lake is a big reservoir; table formats are the gates and meters that keep it organised.
Object storageCheap scalable file storage
ParquetColumnar file format
Apache IcebergOpen table format
Data catalogueInventory of datasets