Lumen

Abstract

1. Introduction

Lumen is a fully open-source and extensible agent based framework for chatting with data and for retrieval augmented generation (RAG). The declarative nature of Lumen's data model make it possible for LLMs to easily generate entire data transformation pipelines, visualizations and other many other types of output. Once generated the data pipelines and visual output can be easily serialized, making it possible to share them, to continue the analysis in a notebook and/or build entire dashboards.

Lumen sets itself apart from other agent based frameworks in that it focuses on being fully open and extensible. With powerful internal primitives for expressing complex data transformations the LLM can gain insights into your datasets out-of-the box and can be further tailored with custom agents, analyses and tools to empower even non-programmers to perform complex analyses without having to code. The customization makes it possible to generate any type of output, allow the user and the LLM to perform analyses tailored to your domain and look up additional information and context easily. Since Lumen is built on Panel it can render almost any type of output with little to no effort, ensuring that even the most esoteric usecase is easily possible.

The declarative Lumen data model further sets it apart from other tools, making it easy for LLMs to populate custom components and making it easy for the user to share the results. Entire multi-step data transformation pipelines be they in SQL or Python can easily be captured and used to drive custom visualizations, interactive tables and more. Once generated the declarative nature of the Lumen specification allows them to be shared, reproducing them in a notebook or composing them through a drag-and-drop interface into a dashboard.

2. Objective

Illuminate your data. Agent framework turning natural language into SQL, charts, dashboards and reports.

This project demonstrates how Python can be applied to a real-world Data Science problem.

4. Technology Stack

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/holoviz/lumen.git
cd lumen

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Turn the analysis into an interactive dashboard
• Automate data refresh with a scheduled job
• Add a predictive model on top of the analysis

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. What is the source of the dataset and how was missing data handled?
2. Which exploratory analysis steps revealed the most useful insight?
3. Why were these particular charts chosen to present the data?
4. Which statistical or ML technique supports the conclusions?
5. How could the analysis be automated or refreshed with new data?

9. Source Code & License

This project is developed by holoviz and published on GitHub under the BSD 3-Clause "New" or "Revised" License. Please follow the license terms and credit the original author when you use or modify this code.