Piperider

Abstract

1. Introduction

PipeRider automatically compares your data to highlight the difference in impacted downstream dbt models so you can merge your Pull Requests with confidence.

the connection profiles in your dbt project to connect to the data warehouse

queries, comparing downstream data impact takes little time, speeding up your team's review time

2. Objective

Code review for data in dbt

This project demonstrates how Python can be applied to a real-world Data Science problem.

3. Key Features / Modules

• Model profiling: PipeRider can profile your dbt models and obtain
• Metric queries: PipeRider can integrate with dbt metrics and present
• HTML report: PipeRider generates a static HTML report each time it runs, which can be viewed locally or shared.
• Report comparison: You can compare two previously generated reports or use a single command to compare the
• CI integration: The key to CI is automation, and in the code review process, automating this workflow is even more

4. Technology Stack

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/InfuseAI/piperider.git
cd piperider

1. Install PipeRider
2. Add PipeRider tag on your model: Go to your dbt project, and add the PipeRider tag on the model you want to
3. Run PipeRider

pip install piperider[<connector>]

--models/staging/stg_customers.sql
   {{ config(
      tags=["piperider"]
   ) }}

   select ...

dbt list -s tag:piperider --resource-type model

piperider run

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Turn the analysis into an interactive dashboard
• Automate data refresh with a scheduled job
• Add a predictive model on top of the analysis

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. What is the source of the dataset and how was missing data handled?
2. Which exploratory analysis steps revealed the most useful insight?
3. Why were these particular charts chosen to present the data?
4. Which statistical or ML technique supports the conclusions?
5. How could the analysis be automated or refreshed with new data?

9. Source Code & License

This project is developed by InfuseAI and published on GitHub under the Apache License 2.0. Please follow the license terms and credit the original author when you use or modify this code.