Documentation Scraper with slurp-ai

Overview

slurp-ai scrapes documentation websites and compiles them into a single markdown file optimized for AI agent context. It is lightweight, fast, and deterministic - it does NOT use AI to scrape, it is FOR AI consumption.

CRITICAL: Run Outside Sandbox

All commands in this skill MUST be run outside the sandbox. Use dangerouslyDisableSandbox: true for all Bash commands including:

• which slurp (installation check)
• node analyze-sitemap.js (sitemap analysis)
• slurp (scraping)
• File inspection commands (wc, head, cat, etc.)

The sandbox blocks network access and file operations required for web scraping.

Pre-Flight: Check Installation

Before scraping, verify slurp-ai is installed:

which slurp || echo "NOT INSTALLED"

If not installed, ask the user to run:

npm install -g slurp-ai

Requires: Node.js v20+

Do NOT proceed with scraping until slurp-ai is confirmed installed.

Commands

Command	Purpose
slurp <url>	Fetch and compile in one step
slurp fetch <url> [version]	Download docs to partials only
slurp compile	Compile partials into single file
slurp read <package> [version]	Read local documentation

Output: Creates slurp_compiled/compiled_docs.md from partials in slurp_partials/.

CRITICAL: Analyze Sitemap First

Before running slurp, ALWAYS analyze the sitemap. This reveals the complete site structure and informs your --base-path and --max decisions.

Step 1: Run Sitemap Analysis

Use the included analyze-sitemap.js script:

node analyze-sitemap.js https://docs.example.com

This outputs:

• Total page count (informs --max)
• URLs grouped by section (informs --base-path)
• Suggested slurp commands with appropriate flags
• Sample URLs to understand naming patterns

Step 2: Interpret the Output

Example output:

📊 Total URLs in sitemap: 247

📁 URLs by top-level section:
   /docs                          182 pages
   /api                            45 pages
   /blog                           20 pages

🎯 Suggested --base-path options:
   https://docs.example.com/docs/guides/     (67 pages)
   https://docs.example.com/docs/reference/  (52 pages)
   https://docs.example.com/api/             (45 pages)

💡 Recommended slurp commands:

   # Just "/docs/guides" section (67 pages)
   slurp https://docs.example.com/docs/guides/ --base-path https://docs.example.com/docs/guides/ --max 80

Step 3: Choose Scope Based on Analysis

Sitemap Shows	Action
< 50 pages total	Scrape entire site: slurp <url> --max 60
50-200 pages	Scope to relevant section with --base-path
200+ pages	Must scope down - pick specific subsection
No sitemap found	Start with --max 30, inspect partials, adjust

Step 4: Frame the Slurp Command

With sitemap data, you can now set accurate parameters:

# From sitemap: /docs/api has 45 pages
slurp https://docs.example.com/docs/api/intro \
  --base-path https://docs.example.com/docs/api/ \
  --max 55

Key insight: Starting URL is where crawling begins. Base path filters which links get followed. They can differ (useful when base path itself returns 404).

Common Scraping Patterns

Library Documentation (versioned)

# Express.js 4.x docs
slurp https://expressjs.com/en/4x/api.html --base-path https://expressjs.com/en/4x/

# React docs (latest)
slurp https://react.dev/learn --base-path https://react.dev/learn

API Reference Only

slurp https://docs.example.com/api/introduction --base-path https://docs.example.com/api/

Full Documentation Site

slurp https://docs.example.com/

CLI Options

Flag	Default	Purpose
--max <n>	20	Maximum pages to scrape
--concurrency <n>	5	Parallel page requests
--headless <bool>	true	Use headless browser
--base-path <url>	start URL	Filter links to this prefix
--output <dir>	./slurp_partials	Output directory for partials
--retry-count <n>	3	Retries for failed requests
--retry-delay <ms>	1000	Delay between retries
--yes	-	Skip confirmation prompts

Compile Options

Flag	Default	Purpose
--input <dir>	./slurp_partials	Input directory
--output <file>	./slurp_compiled/compiled_docs.md	Output file
--preserve-metadata	true	Keep metadata blocks
--remove-navigation	true	Strip nav elements
--remove-duplicates	true	Eliminate duplicates
--exclude <json>	-	JSON array of regex patterns to exclude

When to Disable Headless Mode

Use --headless false for:

• Static HTML documentation sites
• Faster scraping when JS rendering not needed

Default is headless (true) - works for most modern doc sites including SPAs.

Output Structure

slurp_partials/              # Intermediate files
  └── page1.md
  └── page2.md
slurp_compiled/              # Final output
  └── compiled_docs.md       # Compiled result

Quick Reference

# 1. ALWAYS analyze sitemap first
node analyze-sitemap.js https://docs.example.com

# 2. Scrape with informed parameters (from sitemap analysis)
slurp https://docs.example.com/docs/ --base-path https://docs.example.com/docs/ --max 80

# 3. Skip prompts for automation
slurp https://docs.example.com/ --yes

# 4. Check output
cat slurp_compiled/compiled_docs.md | head -100

Common Issues

Problem	Cause	Solution
Wrong --max value	Guessing page count	Run analyze-sitemap.js first
Too few pages scraped	--max limit (default 20)	Set --max based on sitemap analysis
Missing content	JS not rendering	Ensure --headless true (default)
Crawl stuck/slow	Rate limiting	Reduce --concurrency 3
Duplicate sections	Similar content	Use --remove-duplicates (default)
Wrong pages included	Base path too broad	Use sitemap to find correct --base-path
Prompts blocking automation	Interactive mode	Add --yes flag

Post-Scrape Usage

The output markdown is designed for AI context injection:

# Check file size (context budget)
wc -c slurp_compiled/compiled_docs.md

# Preview structure
grep "^#" slurp_compiled/compiled_docs.md | head -30

# Use with Claude Code - reference in prompt or via @file

When NOT to Use

• API specs in OpenAPI/Swagger: Use dedicated parsers instead
• GitHub READMEs: Fetch directly via raw.githubusercontent.com
• npm package docs: Often better to read source + README
• Frequently updated docs: Consider caching strategy