All 60 tasks, level by level

The real graded Data Engineering & ETL tasks the portal assigns, grouped into three levels.

The portal assigns these Data Engineering & ETL tasks. Finish Level 1 before Level 2 and Level 2 before Level 3. Your mentor may mark some tasks as required for the certificate.

Level 1: Foundations (24 tasks)

Task	What to do
What data engineers do	Explain the data engineer role, batch vs streaming, and draw a simple data pipeline from source to dashboard.
Python for data	Read CSV, JSON and Excel files with Python and pandas, clean them and write them out as Parquet.
SQL refresher	Write 12 SQL queries with joins, aggregates, window functions and CTEs on a sample dataset.
File formats	Compare CSV, JSON, Parquet and Avro by size and read speed on the same data.
Extract from an API	Pull data from a public REST API with pagination and retries, and save the raw responses.
Transform and load	Clean the API data and load it into PostgreSQL with a repeatable script.
Data modelling	Design a star schema for an online shop with fact and dimension tables.
Idempotent loads	Make your load safe to re-run without creating duplicates (upserts or partition overwrite).
Logging and errors	Add logging and clear failure messages to your pipeline, and show a failed run.
Scheduling with cron	Schedule your pipeline to run daily with cron or Task Scheduler and keep a run log.
Data quality checks	Add checks for nulls, duplicates and value ranges that stop the pipeline when data is bad.
Docker for pipelines	Run your pipeline and its database together with docker-compose.
Data dictionary	Write a data dictionary for your warehouse tables with meanings and owners.
SQL style guide	Write a SQL style guide and refactor three queries to follow it.
Date dimension	Generate a date dimension table with Indian holidays and fiscal year columns.
Web scraping responsibly	Scrape a site that allows it (check robots.txt and terms), with rate limiting, and store the data.
JSON flattening	Flatten nested JSON API data into relational tables.
Deduplication	Remove duplicate customer records using fuzzy matching rules.
Pipeline README	Document how to run, test and troubleshoot your pipeline.
Cloud storage	Store raw files in cloud object storage organised by date partitions.
CSV pitfalls	Show problems with encodings, delimiters and quotes in CSV files and fix them.
Timezone handling	Convert timestamps from different sources into UTC and IST correctly.
Small data model review	Review a classmate's data model and suggest improvements.
Pipeline metrics	Log row counts, durations and failures for each run.

Level 2: Building real features (20 tasks)

Task	What to do
Apache Airflow	Rebuild your pipeline as an Airflow DAG with separate extract, transform and load tasks.
`dbt models`	Build staging and mart models with dbt, with tests and documentation.
Incremental loading	Change your pipeline to load only new or changed records using a watermark column.
Slowly changing dimensions	Implement a Type 2 slowly changing dimension for customer addresses.
Data warehouse in the cloud	Load your data into a free warehouse (BigQuery sandbox or similar) and query it.
Partitioning and clustering	Partition a large table by date and show how it reduces data scanned.
Apache Spark basics	Process a large dataset with PySpark: read, transform, aggregate and write Parquet.
Great Expectations	Add a data validation suite with Great Expectations (or Soda) and publish its report.
Change data capture concept	Explain CDC and demonstrate it with Debezium or by reading a database's change log.
Pipeline documentation	Document your pipelines with lineage diagrams, owners, schedules and SLAs.
Backfills	Run a historical backfill of your pipeline for 30 days safely.
Data contracts	Write a data contract between a source team and your pipeline, with schema and SLAs.
`dbt snapshots`	Track changes over time with dbt snapshots.
`dbt tests and docs`	Add unique, not null and relationship tests, and publish dbt docs.
Spark optimisation	Fix a slow Spark job using partitioning, broadcast joins and caching.
Parquet layout	Compare partitioning strategies for Parquet files by query speed.
Reverse ETL concept	Explain reverse ETL and push one warehouse table to a CRM or sheet.
API ingestion at scale	Ingest a paginated API with concurrency limits, retries and checkpointing.
Orchestrator sensors	Trigger a pipeline when a file arrives using a sensor or event trigger.
Data quality dashboard	Show data quality check results over time in a dashboard.

Level 3: Professional work and capstone (16 tasks)

Task	What to do
Streaming with Kafka	Produce and consume events with Kafka (or Redpanda) and write them to storage in near real time.
Stream processing	Compute a rolling 5-minute count from a stream with Spark Structured Streaming or Flink.
Data lake layers	Build bronze, silver and gold layers for one dataset and explain what each layer guarantees.
Lakehouse table format	Store data with Delta Lake or Apache Iceberg and show time travel and schema evolution.
Orchestration with retries and alerts	Add retries, timeouts and failure alerts (email or chat) to your Airflow DAG.
Cost and performance tuning	Tune a slow Spark or SQL job and report time and cost before and after.
Data governance	Add access control, PII masking and a data catalogue entry for your main tables.
Capstone: production pipeline	Deliver an end-to-end pipeline (ingest, validate, transform, warehouse, dashboard) with orchestration, tests, docs and monitoring.
Real-time dashboard	Feed a streaming pipeline into a live dashboard.
Exactly-once thinking	Explain at-least-once vs exactly-once processing and design for idempotent writes.
Schema evolution	Handle a new column and a renamed column in a source without breaking the pipeline.
Cost monitoring	Track warehouse query cost per user or pipeline and find the most expensive query.
Data mesh concept	Explain data mesh and design domain-owned data products for a company.
Privacy engineering	Implement data retention and deletion for personal data in your pipeline.
Pipeline incident review	Write a postmortem for a failed or late pipeline run.
Capstone 2: analytics platform	Build a small analytics platform for a business case with ingestion, lakehouse, dbt models, orchestration, quality checks and a BI dashboard.

← Tools and first-week setup Submitting tasks and earning your certificate →