← Data Engineering & ETL learning pathData Engineering & ETL · Module 2 of 10
Data sources and extraction
Pull data from databases, APIs, files and events.
📖 Key ideas
- Sources: OLTP databases, SaaS APIs, files (CSV/JSON), logs, event streams.
- Full extracts copy everything; incremental extracts copy only changes.
- Change Data Capture (CDC) streams database changes.
- Handle API pagination and rate limits.
- Tools: Airbyte, Fivetran, Debezium, custom scripts.
💡 Real-life example: Incremental extraction is reading only today's newspaper instead of reprinting the whole archive.
OLTPTransactional application database
Incremental loadLoad only new/changed data
CDCChange Data Capture
ConnectorTool linking a source to a pipeline