Open SEO Crawler

Abstract

1. Introduction

Built for SEO professionals, web developers, and site owners who want a real technical SEO audit that stays on their machine. Use it as a free SEO crawler for a single site or a recurring website crawler across your whole portfolio.

→ One-line install on Linux, macOS, or Windows. Auto-starts on boot, auto-updates daily. Browser opens at http://localhost:5002/ when done.

2. Objective

Free SEO crawler & website audit tool — a self-hosted, open-source SEO spider that crawls any website for technical SEO issues. Concurrent, CMS-aware (Shopify/WordPress/Webflow), XLSX export. A free alternative to Screaming Frog.

This project demonstrates how JavaScript, Flask, Python can be applied to a real-world Digital Marketing & SEO problem.

4. Technology Stack

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Node.js (LTS) and npm
• A modern web browser
• VS Code or any code editor
• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/puneetindersingh/open-seo-crawler.git
cd open-seo-crawler

1. Verifies prerequisites (Python 3.10+, systemd, sudo, disk space, free port 5002, internet)
2. Installs python3 / python3-venv / git / curl via apt if missing
3. Clones the repo, creates a virtualenv, installs Python deps
4. Registers open-seo-crawler.service so the crawler starts on every boot
5. Registers open-seo-crawler-update.timer to git pull + restart 2 min after every boot and once daily (auto-rolls-back on any failure)
6. Prints the access URLs (http://localhost:5002/ plus your LAN IP) and auto-opens the browser if you're on a desktop
7. Saves the URLs to ~/open-seo-crawler/ACCESS_URLS.txt for later
8. Verifies prerequisites (macOS, Python 3.10+, disk space, free port 5002, internet)

git clone https://github.com/puneetindersingh/open-seo-crawler.git
cd open-seo-crawler
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python3 app.py

curl -fsSL https://raw.githubusercontent.com/puneetindersingh/open-seo-crawler/master/install.sh -o install.sh && chmod +x install.sh && ./install.sh

curl -fsSL https://raw.githubusercontent.com/puneetindersingh/open-seo-crawler/master/install.sh -o install.sh && chmod +x install.sh && ./install.sh --check

systemctl status open-seo-crawler                  # is it running?
sudo systemctl restart open-seo-crawler            # restart
sudo systemctl disable open-seo-crawler            # stop autostarting on boot
./install.sh --update-now                          # force a git-pull + restart now
systemctl list-timers | grep open-seo-crawler      # when's the next auto-update?
journalctl -u open-seo-crawler-update.service -n 50  # update history
sudo systemctl disable --now open-seo-crawler-update.timer  # turn auto-update off
tail -f /var/log/open-seo-crawler.log              # live app logs

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Export reports to Google Sheets or PDF
• Schedule weekly automated reports
• Add competitor comparison

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. Which marketing or SEO problem does this tool solve?
2. Which data sources or APIs does it use (Search Console, Analytics, social platforms)?
3. Which metrics or KPIs does it report and how are they calculated?
4. How could the output help a business make decisions?
5. How would you schedule it to run automatically?

9. Source Code & License

This project is developed by puneetindersingh and published on GitHub under the MIT License. Please follow the license terms and credit the original author when you use or modify this code.