Sitemap Generator Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

Digital Marketing & SEOPHPMIT

Abstract

Sitemap Generator Crawler is an open-source Digital Marketing & SEO project. PHP script to recursively crawl websites and generate a sitemap. Zero dependencies. It is built using PHP. Key capabilities include: Actually crawls webpages like Google would; Zero dependencies; Generates a seperate XML file which gets updated every time the script gets executed (Runnable via CRON). The complete source code is publicly available on GitHub under the MIT License, making it a useful reference for students building a Digital Marketing & SEO mini project or final-year project.

1. Introduction

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

2. Objective

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

This project demonstrates how PHP can be applied to a real-world Digital Marketing & SEO problem.

3. Key Features / Modules

  • Actually crawls webpages like Google would
  • Zero dependencies
  • Generates a seperate XML file which gets updated every time the script gets executed (Runnable via CRON)
  • Crawls faster than online services
  • Verbose logging
  • Completely usable through CLI
  • Non-restrictive licensing

4. Technology Stack

PHP

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

  • PHP 7.4+ with a web server (XAMPP / WAMP / LAMP)
  • MySQL / MariaDB
  • phpMyAdmin (optional)
  • Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/vezaynk/Sitemap-Generator-Crawler.git
cd Sitemap-Generator-Crawler
  1. Configure the crawler by modifying the config file sitemap.config.php
  2. Select the file to which the sitemap will be saved
  3. Select URL to crawl
  4. Configure blacklists, accepts the use of wildcards (example: http://example.com/private/ and .jpg)
  5. Generate sitemap
  6. Either send a GET request to this script or use it from the CLI as seen below
  7. A sitemap will be generated and saved
  8. Submit to Google

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

  • Export reports to Google Sheets or PDF
  • Schedule weekly automated reports
  • Add competitor comparison

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

  1. Which marketing or SEO problem does this tool solve?
  2. Which data sources or APIs does it use (Search Console, Analytics, social platforms)?
  3. Which metrics or KPIs does it report and how are they calculated?
  4. How could the output help a business make decisions?
  5. How would you schedule it to run automatically?

9. Source Code & License

This project is developed by vezaynk and published on GitHub under the MIT License. Please follow the license terms and credit the original author when you use or modify this code.

View Source on GitHubDownload ZIP

Want to build this as your internship project?

Work on a Digital Marketing & SEO project like this with mentor guidance, weekly reviews and an internship certificate from Training Trains, Erode — online or offline.

Apply for Digital Marketing & SEO Internship