WEB2 RECON PIPELINE

Full asset discovery from nothing to a prioritized URL list ready for hunting.

---

SETUP (one-time)

# 1. Set your Chaos API key (get free key at chaos.projectdiscovery.io)
export CHAOS_API_KEY="your-key-here"
# Add to ~/.zshrc or ~/.bashrc for persistence:
echo 'export CHAOS_API_KEY="your-key-here"' >> ~/.zshrc

# 2. Update nuclei templates (run weekly)
nuclei -update-templates

# 3. Configure subfinder with API keys for more sources
mkdir -p ~/.config/subfinder
cat > ~/.config/subfinder/config.yaml << 'EOF'
# Get free keys at: virustotal.com, securitytrails.com, censys.io, shodan.io
virustotal: [YOUR_VT_KEY]
securitytrails: [YOUR_ST_KEY]
censys_apiid: YOUR_CENSYS_ID
censys_secret: YOUR_CENSYS_SECRET
shodan: [YOUR_SHODAN_KEY]
EOF

# 4. Verify all tools installed
which subfinder httpx dnsx nuclei katana waybackurls gau dalfox ffuf anew gf interactsh-client

---

THE 5-MINUTE RULE

If a target shows nothing interesting after 5 minutes of recon, move on. Don't burn hours on dead surface.

5-minute kill signals:

• All subdomains return 403 or static marketing pages
• No API endpoints visible in URLs
• No JavaScript bundles with interesting endpoint paths
• nuclei returns 0 medium/high findings
• No forms, no authentication, no user data

---

MANUAL-ONLY PROGRAM OVERRIDE

Before running this pipeline, inspect the live bounty policy for automation restrictions. If automated tools, scripts, scanners, public URL indexes, or scanner-generated findings are prohibited:

1. Do not run the standard recon pipeline, bulk CT harvesting, httpx, katana, gau, waybackurls, ffuf, nuclei, scripted multi-host probes, or scanner wrappers against that program.
2. Create SCOPE_FENCE.md first, recording the exact prohibition and explicit third-party host exclusions.
3. Use a normal browser or intercepting proxy manually. Follow links and inspect requests generated by one researcher-controlled session.
4. Treat public certificate-transparency and archived-URL discoveries as prohibited too when the policy excludes search-engine/index/archive findings—not merely as low-confidence leads.
5. If prior automation was accidentally used before the policy was fully parsed, quarantine its outputs: do not test, report, or use those discoveries. Restart from manually reached first-party pages.
6. Honor operator-selected exclusions independently of program scope. Example: if the user says to skip native apps, remove Windows/macOS/Linux/iOS/Android workstreams immediately rather than keeping them as pending tasks.

This override outranks every command below.

STANDARD RECON PIPELINE

Pre-Hunt: Always Run First

TARGET="target.com"

# Step 0: Passive — crt.sh certificate transparency (no API key needed)
# NOTE: crt.sh is flaky (502s/timeouts) and large domains truncate JSON. Retry with a UA,
# and use regex fallback if json.load fails. See references/passive_ct_dns_brute_curl_fingerprint.md
curl -s "https://crt.sh/?q=%.${TARGET}&output=json" \
  | jq -r '.[].name_value' \
  | sed 's/\*\.//g' \
  | sort -u > /tmp/subs.txt
echo "[+] crt.sh: $(wc -l < /tmp/subs.txt) subdomains"

# Step 1: Chaos API (ProjectDiscovery — most comprehensive source)
curl -s "https://dns.projectdiscovery.io/dns/$TARGET/subdomains" \
  -H "Authorization: $CHAOS_API_KEY" \
  | jq -r '.[]' >> /tmp/subs.txt

echo "[+] Chaos returned $(wc -l < /tmp/subs.txt) subdomains"

# Step 2: subfinder (passive multi-source)
subfinder -d $TARGET -silent | anew /tmp/subs.txt
assetfinder --subs-only $TARGET | anew /tmp/subs.txt

echo "[+] Total subdomains after all sources: $(wc -l < /tmp/subs.txt)"

# Step 3: DNS resolution + live host check
cat /tmp/subs.txt | dnsx -silent | httpx -silent -status-code -title -tech-detect | tee /tmp/live.txt

echo "[+] Live hosts: $(wc -l < /tmp/live.txt)"

# Step 4: URL crawl
cat /tmp/live.txt | awk '{print $1}' | katana -d 3 -jc -kf all -silent | anew /tmp/urls.txt

# Step 5: Historical URLs
echo $TARGET | waybackurls | anew /tmp/urls.txt
gau $TARGET --subs | anew /tmp/urls.txt

echo "[+] Total URLs: $(wc -l < /tmp/urls.txt)"

# Step 6: Nuclei scan
nuclei -l /tmp/live.txt -t ~/nuclei-templates/ -severity critical,high,medium -o /tmp/nuclei.txt

Output to Organized Directory

TARGET="target.com"
RECON_DIR="recon/$TARGET"
mkdir -p $RECON_DIR

# All outputs go here:
/tmp/subs.txt         → $RECON_DIR/subdomains.txt
/tmp/live.txt         → $RECON_DIR/live-hosts.txt
/tmp/urls.txt         → $RECON_DIR/urls.txt
/tmp/nuclei.txt       → $RECON_DIR/nuclei.txt

---

ATTACK SURFACE TRIAGE

robots.txt → API Surface Discovery (HIGH SIGNAL)

Before any URL crawling, check robots.txt. It often reveals the entire authenticated API surface that search engines are told to skip — these are the most interesting endpoints.

curl -s "https://$TARGET/robots.txt"
# Look for Disallow entries — they reveal hidden API paths:
# Disallow: /webapi/          → API surface
# Disallow: /service_api/     → Service API
# Disallow: /my-servers       → Authenticated user pages
# Disallow: /my-orders
# Disallow: /transactions
# Disallow: /balance
# Disallow: /api-keys
# Disallow: /ssh-keys
# Disallow: /profile
# Disallow: /2fa              → 2FA endpoints
# Disallow: /hive-login       → Special auth flows
# Disallow: /docker-auth      → Infrastructure auth

# Real example (Clore.ai): robots.txt exposed 25+ API/auth endpoints
# that were NOT discoverable via crawling (auth-gated)

# After finding targets in robots.txt, probe for API access:
for path in webapi service_api my-servers my-orders balance transactions profile; do
  code=$(curl -s -o /dev/null -w "%{http_code}" "https://$TARGET/${path}" -H "Accept: application/json")
  echo "/${path}: HTTP ${code}"
done

# If path returns SPA HTML, try POST with JSON body:
curl -s -X POST "https://$TARGET/marketplace/servers" \
  -H "Content-Type: application/json" \
  -d '{}' -H "Accept: application/json"

# Also probe api.$TARGET subdomain for REST endpoints:
for sub in api dev staging admin mail cdn vpn; do
  curl -s -o /dev/null -w "%{http_code}" "https://${sub}.$TARGET/" --connect-timeout 5
  echo "  ${sub}.$TARGET"
done

Find Interesting Targets in URL List

# Parameters worth testing
cat /tmp/urls.txt | grep -E "[?&](id|user|file|path|url|redirect|next|src|token|key|api_key)=" | tee /tmp/interesting-params.txt

# API endpoints
cat /tmp/urls.txt | grep -E "/api/|/v1/|/v2/|/v3/|/graphql|/rest/|/gql" | tee /tmp/api-endpoints.txt

# File upload endpoints
cat /tmp/urls.txt | grep -E "upload|file|attachment|document|image|avatar|photo|media" | tee /tmp/uploads.txt

# Admin/internal paths
cat /tmp/urls.txt | grep -E "/admin|/internal|/debug|/test|/staging|/dev|/management|/console" | tee /tmp/admin-paths.txt

# Authentication endpoints
cat /tmp/urls.txt | grep -E "/oauth|/login|/auth|/sso|/saml|/oidc|/callback|/token" | tee /tmp/auth-paths.txt

gf Patterns (Quick Classification)

# Install gf patterns: https://github.com/tomnomnom/gf
cat /tmp/urls.txt | gf xss | tee /tmp/xss-candidates.txt
cat /tmp/urls.txt | gf ssrf | tee /tmp/ssrf-candidates.txt
cat /tmp/urls.txt | gf idor | tee /tmp/idor-candidates.txt
cat /tmp/urls.txt | gf sqli | tee /tmp/sqli-candidates.txt
cat /tmp/urls.txt | gf redirect | tee /tmp/redirect-candidates.txt
cat /tmp/urls.txt | gf lfi | tee /tmp/lfi-candidates.txt
cat /tmp/urls.txt | gf rce | tee /tmp/rce-candidates.txt

# User-controlled CSS surface (themes, profile pages, HTML email renderers,
# rich-text editors, PDF generators). gf has no pattern for this — grep manually:
cat /tmp/urls.txt | grep -iE "theme|profile|signature|customize|email|invoice|pdf|render|markdown" \
  | tee /tmp/css-injection-candidates.txt
# → if any hit, run web2-vuln-classes **CSS Injection**

---

SECRET SCANNING IN JS BUNDLES

Next.js deep-dive: For full extraction of Privy App IDs, WalletConnect project IDs, Thirdweb RPC URLs, QuikNode API keys, API base URLs, and auth flow reconstruction from webpack chunks, see references/nextjs_bundle_config_extraction.md.

Vue.js SPA deep-dive: For VUE_APP_* env var extraction, API route map from constants, RPC proxy abuse testing, SIWE auth patterns, and S3 upload pre-signed URL discovery, see references/vuejs_spa_bundle_extraction.md.

SecretFinder (API keys, tokens in JS bundles)

# Activate venv
source ~/tools/SecretFinder/.venv/bin/activate

# Scan a single JS file
python3 ~/tools/SecretFinder/SecretFinder.py -i "https://target.com/static/js/main.js" -o cli

# Scan all JS URLs found in recon
cat /tmp/urls.txt | grep "\.js$" | head -50 | while read url; do
  echo "=== $url ==="
  python3 ~/tools/SecretFinder/SecretFinder.py -i "$url" -o cli 2>/dev/null
done

deactivate

LinkFinder (Endpoints hidden in JS)

source ~/tools/LinkFinder/.venv/bin/activate

# Single JS file
python3 ~/tools/LinkFinder/linkfinder.py -i "https://target.com/app.js" -o cli

# All pages (crawls JS from HTML)
python3 ~/tools/LinkFinder/linkfinder.py -i "https://target.com" -d -o cli

deactivate

---

DIRECTORY FUZZING

ffuf — Standard Fuzzing

# Directory discovery on a live host
ffuf -u "https://target.com/FUZZ" \
     -w ~/wordlists/common.txt \
     -mc 200,201,204,301,302,307,401,403 \
     -ac \
     -t 40 \
     -o /tmp/ffuf-dirs.json

# API endpoint discovery
ffuf -u "https://target.com/api/FUZZ" \
     -w ~/wordlists/api-endpoints.txt \
     -mc 200,201,204,301,302 \
     -ac \
     -t 20

# IDOR fuzzing with authenticated request
# Create req.txt with Authorization: Bearer TOKEN
ffuf -request /tmp/req.txt \
     -request-proto https \
     -w <(seq 1 10000) \
     -fc 404 \
     -ac \
     -t 10

---

TARGET SCORING — GO / NO-GO

Score before spending time. Skip if score < 4.

Criterion	Points
Max bounty >= $5K	+2
Large user base (>100K) or handles money	+2
Program launched < 60 days ago	+2
Complex features: API, OAuth, file upload, GraphQL	+1
Recent code/feature changes (GitHub, changelog)	+1
Private program (less competition)	+1
Tech stack you know	+1
Source code available	+1
Prior disclosed reports to study	+1

< 4: Skip
4-5: Only if nothing better available
6-8: Good — spend 1-3 days
>= 9: Excellent — spend up to 1 week

Pre-Dive Hard Kill Signals

1. Max bounty < $500 → not worth your time
2. All recent reports are N/A or duplicate → hunters saturated it
3. Scope is only a static marketing page → no attack surface
4. Company < 5 employees with no revenue → won't pay
5. Explicitly excludes your planned bug class in rules

---

TECH STACK DETECTION (2 min)

# Response headers reveal backend
curl -sI https://target.com | grep -iE "server|x-powered-by|x-aspnet|x-runtime|x-generator"

# Common signals:
# Server: nginx + X-Powered-By: PHP/7.4 → PHP backend
# Server: gunicorn OR X-Powered-By: Express → Python/Node.js
# X-Powered-By: ASP.NET → .NET
# Server: Apache Tomcat → Java
# X-Runtime: Ruby → Ruby on Rails
# X-Powered-By: Sails <sailsjs.com> → Sails.js (Node.js + Vue)

# Framework from JS bundle paths:
# /_next/static/ → Next.js
# /_next/static/chunks/ → Next.js (100+ webpack chunks — each may contain config fragments)
#   → extract ALL chunk URLs from <script src=...> and RSC self.__next_f payload
#   → see references/nextjs_bundle_config_extraction.md for full deep-dive
# /static/js/main.chunk.js → CRA (React)
# /packs/ → Ruby on Rails + Webpacker
# /__nuxt/ → Nuxt.js (Vue)

# --- Sails.js specific signals (see references/sailsjs_recon_patterns.md) ---
# After detecting X-Powered-By: Sails, probe these immediately:
# 1. Check SAILS_LOCALS._environment in HTML source for dev mode
# curl -s https://target.com | grep -oP 'SAILS_LOCALS.*?};'
# 2. Check /api/health for info disclosure
# curl -s https://target/api/health
# 3. Check socket.io endpoint
# curl -s https://target/socket.io/?EIO=4\&transport=polling
# 4. Check docker IP leak in X-Server-IP header
# curl -sI https://target.com | grep X-Server-IP

Stack → Primary Bug Class Map

Stack	Hunt First	Hunt Second
Ruby on Rails	Mass assignment	IDOR (:id routes)
Django	IDOR (ModelViewSet, no object perms)	SSTI (mark_safe)
Flask	SSTI (render_template_string)	SSRF (requests lib)
Laravel	Mass assignment ($fillable)	IDOR (Eloquent, no ownership)
Express (Node.js)	Prototype pollution	Path traversal + debug surface (/_debug, /__debug__) → web2-vuln-classes "Error Disclosure / Debug Endpoints"
Spring Boot	Actuator endpoints → web2-vuln-classes "Error Disclosure / Debug Endpoints" for full surface	SSTI (Thymeleaf)
ASP.NET	ViewState deserialization	Open redirect (ReturnUrl)
Next.js	SSRF via Server Actions + /_next/data/ / /_next/static/chunks/ → web2-vuln-classes "Error Disclosure / Debug Endpoints"	Open redirect via redirect()
GraphQL	Introspection → auth bypass on mutations	IDOR via node(id:)
WordPress	Plugin SQLi	REST API auth bypass
SPA frameworks (React / Vue / Svelte / Angular)	DOM XSS sinks via state/router → web2-vuln-classes section 3 "postMessage Testing" for cross-frame entry points	Client-side route auth bypass (role check only in JS)

JS-Rendered Targets / Anti-Bot Crawling → crawl4ai-recon

For sites that block katana/hakrawler with Cloudflare, PerimeterX, DataDome, or heavy client-side rendering that katana cannot execute (React/Vue SPAs, webflow-style sites, login-walled pages), use the crawl4ai-recon skill. Crawl4AI spawns full Chromium with stealth heuristics — handles JS rendering, anti-bot bypass, structured (markdown/JSON) extraction.

When to pivot from katana to Crawl4AI:

• Katana returns mostly empty URL list on a known SPA target
• Target has Cloudflare challenge page on every request
• Manual Burp crawl needed — Crawl4AI automates the headless browser bit
• You want LLM-friendly output (markdown) to feed into agent for endpoint classification

Pipeline integration:

# Standard: katana first (cheap), then crawl4ai for JS-heavy / blocked targets
cat /tmp/live.txt | awk '{print $1}' | katana -d 3 -silent > /tmp/urls.txt
# If low yield (< 50 URLs), or target is known SPA / Cloudflare'd:
while read host; do
  crawl4ai-cli "https://$host" --depth 2 -o json -f "./crawl/${host//\//_}.json"
done < /tmp/live.txt

See crawl4ai-recon skill for install, anti-bot config, Docker server hardening (11 published GHSAs), and known CVE patterns to AVOID in your own deployment.
| Sails.js (Node.js + Vue) | Info disclosure via /api/health (_environment, memory, DB/Redis status) + dev mode surface (blueprint API, socket.io) → references/sailsjs_recon_patterns.md | Docker IP leak (X-Server-IP), user enumeration via login error messages, CORS wildcard |
| Express / NestJS | Swagger/OpenAPI docs (/docs, /docs/json, /api-docs) → references/express_nestjs_api_probe.md | Info disclosure via root endpoint ({"environment","mode","version"}), token-in-body auth (not header), sub-app structure (/bot-api, /poh-api, /webapi)

---

CONTINUOUS MONITORING SETUP

Set up once per target. Alerts you before other hunters.

New Subdomain Alerts (daily cron)

#!/bin/bash
TARGET="target.com"
KNOWN="/tmp/$TARGET-subs-known.txt"

subfinder -d $TARGET -silent > /tmp/$TARGET-subs-fresh.txt
curl -s "https://dns.projectdiscovery.io/dns/$TARGET/subdomains" \
  -H "Authorization: $CHAOS_API_KEY" \
  | jq -r '.[]' >> /tmp/$TARGET-subs-fresh.txt

# Diff against known
NEW=$(comm -23 <(sort /tmp/$TARGET-subs-fresh.txt) <(sort $KNOWN 2>/dev/null))

if [ -n "$NEW" ]; then
  echo "NEW SUBDOMAINS: $NEW"
  echo "$NEW" >> $KNOWN
fi

# Schedule: crontab -e → 0 8 * * * /bin/bash ~/monitors/subs-watch.sh

GitHub Commit Watch

#!/bin/bash
REPO="TargetOrg/target-app"
LAST_SHA="/tmp/$REPO-last-sha.txt"

CURRENT=$(curl -s "https://api.github.com/repos/$REPO/commits?per_page=1" | jq -r '.[0].sha')
KNOWN=$(cat $LAST_SHA 2>/dev/null)

if [ "$CURRENT" != "$KNOWN" ]; then
  echo "New commit on $REPO: $CURRENT"
  echo $CURRENT > $LAST_SHA
  # Get changed files
  curl -s "https://api.github.com/repos/$REPO/commits/$CURRENT" \
    | jq -r '.files[].filename' | grep -E "auth|middleware|route|permission|role|admin"
fi

# Schedule: */30 * * * * /bin/bash ~/monitors/github-watch.sh

---

PORT SCANNING (often skipped — don't skip)

# naabu — fast port scanner from ProjectDiscovery
# Finds non-standard ports: 8080, 8443, 3000, 8888, 9000, etc.
cat /tmp/live.txt | awk '{print $1}' | naabu -port 80,443,8080,8443,3000,4000,5000,8000,8888,9000,9090,9200,6379 -silent | tee /tmp/open-ports.txt

# Why this matters: admin panels, debug services, internal APIs often run on alt ports
# Example wins: :8080/actuator/env (Spring Boot), :9200/_cat/indices (Elasticsearch), :6379 (Redis)

SECRET SCANNING IN JS BUNDLES

# trufflehog — high-signal secret detection with entropy analysis
# Scans JS files and git repos
pip install trufflehog3 2>/dev/null || true
trufflehog filesystem --only-verified recon/$TARGET/ 2>/dev/null

# SecretFinder — manual JS bundle scan (already in tools/)
source ~/tools/SecretFinder/.venv/bin/activate
cat /tmp/urls.txt | grep "\.js$" | head -100 | while read url; do
  python3 ~/tools/SecretFinder/SecretFinder.py -i "$url" -o cli 2>/dev/null
done
deactivate

# Quick grep for common patterns in downloaded JS
wget -q -r -l 1 -A "*.js" -P /tmp/js-files/ "https://$TARGET" 2>/dev/null
grep -rn "api_key\|apiKey\|client_secret\|access_token\|private_key\|AWS_SECRET\|AKIA" /tmp/js-files/ 2>/dev/null

GITHUB DORKING FOR TARGET

# Search GitHub for hardcoded secrets before hunting the app
TARGET_ORG="TargetOrgName"  # Check their GitHub org

# Useful dorks (search on github.com):
# org:TARGET_ORG password
# org:TARGET_ORG api_key
# org:TARGET_ORG "Authorization: Bearer"
# org:TARGET_ORG .env
# org:TARGET_ORG "BEGIN RSA PRIVATE KEY"

# CLI with gh (GitHub CLI):
gh search code "api_key" --owner "$TARGET_ORG" --json path,repository 2>/dev/null | jq '.'
gh search code "password" --owner "$TARGET_ORG" --json path,repository 2>/dev/null | head -20

# GitDorker (if installed):
python3 ~/tools/GitDorker/GitDorker.py -t GITHUB_TOKEN -d ~/tools/GitDorker/Dorks/alldorksv3 -q "$TARGET" -org

30-MINUTE RECON PROTOCOL

Minutes 0-5: Read Program Page

Note:
- ALL in-scope assets (every domain listed)
- Out-of-scope list (read carefully — common trap)
- Safe harbor statement
- Impact types accepted (some exclude "low")
- Average bounty amount (signals program generosity)

Minutes 5-15: Asset Discovery

Run the standard pipeline above. Focus on live.txt output.

Minutes 15-25: Surface Map

Run gf patterns and the interesting-params grep above.

Minutes 25-30: Manual Exploration

Open Burp Suite. Browse the app with proxy on:

1. Register an account
2. Perform main user actions (create/read/update/delete resources)
3. Note all API calls in Burp history
4. Look for endpoints not in your URL list

After 30 min: Prioritize

Priority 1: API endpoints with ID parameters → IDOR candidates
Priority 2: File upload features → XSS/RCE candidates
Priority 3: OAuth/SSO flows → auth bypass candidates
Priority 4: Search/filter with user input → SQLi/SSRF/SSTI candidates
Priority 5: Admin/debug endpoints → auth bypass candidates

Framework-Specific References

See the references/ directory for framework-specific recon patterns:

File	Covers
references/express_nestjs_api_probe.md	Express/NestJS API probing — Swagger discovery, root endpoint info leak, token-in-body auth pattern, sub-app enumeration, auth bypass vectors
references/sailsjs_recon_patterns.md	Sails.js detection, dev mode surface, blueprint API probing, socket.io, /api/health leaks, Docker IP leak, CORS misconfig, user enumeration, Next.js build manifest analysis for hybrid stacks, Huawei Cloud OBS detection
references/nextjs_bundle_config_extraction.md	Deep-dive Next.js webpack bundle extraction — downloading all chunks, searching for Privy App ID, WalletConnect project ID, Thirdweb RPC URLs, QuikNode API keys, API base URLs, embedded env-var fallbacks, and auth flow reconstruction
references/vuejs_spa_bundle_extraction.md	Vue.js SPA extraction — VUE_APP_* env dump, API route map from constants, RPC proxy abuse testing, SIWE auth flow, S3 upload pre-signed URL discovery, /dd/ vs /dcd/ path conventions
references/payment-platform-sdk-recon.md	Payment-platform recon using live policy DOM extraction, CT-source fallbacks, production/staging fencing, web-bundle manifests, and public iOS/Android/React Native SDKs as authoritative endpoint/auth maps. Includes owned-account experiment design and false-positive kills.
references/web3-dapp-frontend-rpc-recon.md	Web3 SPA recon: distinguish URL routes from UI state, capture runtime chunks, map wallet/RPC/contract surfaces, verify deployments with eth_getCode, safely simulate write functions via eth_call/eth_estimateGas, and kill public-RPC/WAF/scanner false positives.
references/browser-gas-hunting-api-surface.md	Browser-side API surface extraction from JS bundles (performance.getEntriesByType), auth flow reverse-engineering (challenge/sign-in/sign_message pattern, EVM signature, Turnkey custodial vs self-custody), token extraction from localStorage, Cloudflare WARP for IP rotation, terminal output encoding workaround
references/passive_ct_dns_brute_curl_fingerprint.md	Passive-only recon with just curl + Python stdlib (no subfinder/httpx/dnsx): crt.sh flakiness/retry + truncated-JSON regex fallback, DNS-brute wordlist, wildcard vs 0.0.0.1-sinkhole vs internal-IP interpretation of resolution results, curl fingerprinting (status/server/title), and the Python-vs-ProjectDiscovery httpx gotcha.