Libre Crawl

Abstract

1. Introduction

LibreCrawl will always be free and open source. If it's replacing your $259/year Screaming Frog license, deepcrawl license or sitebulb license, buy me a coffee.

LibreCrawl crawls websites and gives you detailed information about pages, links, SEO elements, and performance. It's built as a web application using Python Flask with a modern web interface supporting multiple concurrent users.

A web-based multi-tenant crawler for SEO analysis and website auditing.

2. Objective

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!

This project demonstrates how Python, Flask can be applied to a real-world Digital Marketing & SEO problem.

3. Key Features / Modules

• Multi-tenancy - Multiple users can crawl simultaneously with isolated sessions
• Custom CSS styling - Personalize the UI with your own CSS themes
• Browser localStorage persistence - Settings saved per browser
• JavaScript rendering for dynamic content (React, Vue, Angular, etc.)
• SEO analysis - Extract titles, meta descriptions, headings, etc.
• Link analysis - Track internal and external links with detailed relationship mapping
• PageSpeed Insights integration - Analyze Core Web Vitals
• Multiple export formats - CSV, Excel (XLSX), JSON, or XML
• Issue detection - Automated SEO issue identification
• Real-time crawling progress with live statistics

4. Technology Stack

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/PhialsBasement/LibreCrawl.git
cd LibreCrawl

1. Checks for Docker - if found, runs LibreCrawl in a container (recommended)
2. If no Docker, checks for Python - if not found, downloads and installs it (Windows only temporairly disabled since it causes some bat issues)
3. Installs all dependencies automatically (pip install -r requirements.txt)
4. Installs Playwright browsers for JavaScript rendering
5. Starts LibreCrawl in local mode (no authentication)
6. Opens your browser to http://localhost:5000
7. Clone or download this repository
8. Install dependencies:

start-librecrawl.bat

chmod +x start-librecrawl.sh
./start-librecrawl.sh

pip install -r requirements.txt

playwright install chromium

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Export reports to Google Sheets or PDF
• Schedule weekly automated reports
• Add competitor comparison

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. Which marketing or SEO problem does this tool solve?
2. Which data sources or APIs does it use (Search Console, Analytics, social platforms)?
3. Which metrics or KPIs does it report and how are they calculated?
4. How could the output help a business make decisions?
5. How would you schedule it to run automatically?

9. Source Code & License

This project is developed by PhialsBasement and published on GitHub under the MIT License. Please follow the license terms and credit the original author when you use or modify this code.