Desbordante Core

Abstract

1. Introduction

Desbordante is a high-performance data profiler that is capable of discovering and validating many different patterns in data using various algorithms.

For some patterns Desbordante supports a dynamic task variant. The distinguishing feature of dynamic algorithms compared to classic (static) algorithms is that after a result is obtained, the table can be changed and a dynamic algorithm will update the result based just on those changes instead of processing the whole table again. As a result, they can be up to several orders of magnitude faster than classic (static) ones in some situations.

The Discovery task is designed to identify all instances of a specified pattern type of a given dataset.

2. Objective

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.

This project demonstrates how C++ can be applied to a real-world Data Science problem.

4. Technology Stack

• GNU GCC, version 14+, LLVM Clang, version 16+, or Apple Clang, version 16+
• CMake, version 3.25+
• Boost library built with compiler you're going to use (GCC or Clang), version 1.85-1.86, 1.88+

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Arduino IDE / PlatformIO or a C++ compiler (g++)
• Target board (e.g. Arduino, ESP32) where applicable
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/Desbordante/desbordante-core.git
cd desbordante-core

1. Python >=3.10

$ pip install desbordante

sudo apt update && sudo apt upgrade
sudo apt install g++ cmake ninja-build libboost-all-dev python3 python3-venv
export CXX=g++

wget https://archives.boost.io/release/1.89.0/source/boost_1_89_0.tar.gz
tar xzvf boost_1_89_0.tar.gz
cd boost_1_89_0 && ./bootstrap.sh
sudo ./b2 install --prefix=/usr/

xcode-select --install

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Turn the analysis into an interactive dashboard
• Automate data refresh with a scheduled job
• Add a predictive model on top of the analysis

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. What is the source of the dataset and how was missing data handled?
2. Which exploratory analysis steps revealed the most useful insight?
3. Why were these particular charts chosen to present the data?
4. Which statistical or ML technique supports the conclusions?
5. How could the analysis be automated or refreshed with new data?

9. Source Code & License

This project is developed by Desbordante and published on GitHub under the GNU Affero General Public License v3.0. Please follow the license terms and credit the original author when you use or modify this code.