# Black Sparrow > High-performance, local-first website crawler, 120-rule technical SEO audit engine, and AI-native auditor in Rust. ## Overview - [Black Sparrow — Fast, Local-First Website Crawler & AI-Native SEO Auditor](https://blacksparrowdev.vercel.app/raw/index.md): Modern, local-first website crawler, 120-rule technical SEO audit engine, and AI-native auditor built in Rust. ## Getting Started - [Introduction](https://blacksparrowdev.vercel.app/raw/docs/getting-started/introduction.md): Learn what Black Sparrow is, why it was built, and how it compares to traditional SEO crawlers. - [Installation](https://blacksparrowdev.vercel.app/raw/docs/getting-started/installation.md): Simple, step-by-step instructions to install Black Sparrow on Linux, macOS, and Windows. - [Quickstart Tutorial](https://blacksparrowdev.vercel.app/raw/docs/getting-started/quickstart.md): Run your first website audit in under 60 seconds with Black Sparrow. - [Core Concepts](https://blacksparrowdev.vercel.app/raw/docs/getting-started/core-concepts.md): Understand how Black Sparrow calculates health scores, organizes 4 severity tiers, and persists data. ## CLI Reference - [CLI Overview](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/overview.md): Complete overview of Black Sparrow commands, global options, and exit codes. - [Command: sparrow audit](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/audit.md): Complete guide to running full-site technical SEO audits with Black Sparrow. - [Command: sparrow inspect](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/inspect.md): Instant single-page developer X-ray for inspecting HTTP status, DOM metadata, and schemas. - [Command: sparrow issues](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/issues.md): Filter, search, and drill down into audit findings for any session. - [Command: sparrow report](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/reports.md): Export interactive HTML dashboards, Screaming Frog CSV suites, and Markdown reports. - [Commands: check-ai & schema](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/ai-and-schema.md): Audit website readiness for AI search engines (/llms.txt) and validate schema.org structured data. - [Commands: list, delete & clean](https://blacksparrowdev.vercel.app/raw/docs/cli-reference/session-management.md): Manage historical crawl sessions and maintain your local SQLite database. ## 120 SEO Rules - [Rules Catalog Overview](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/overview.md): Architecture of Black Sparrow's 120-rule technical SEO audit engine and scoring rubric. - [Rules: HTTP & Transport](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/http-and-status.md): Rules 101–108 covering status codes, redirects, SSL transport, and server response times. - [Rules: Meta & Headings](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/meta-and-headings.md): Rules covering , meta descriptions, OpenGraph tags, and H1–H6 heading hierarchy. - [Rules: Directives & Canonicals](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/directives-and-canonicals.md): Rules covering robots directives, noindex tags, X-Robots-Tag, and canonicalization integrity. - [Rules: Links & Anchors](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/links-and-anchors.md): Rules covering internal link health, broken external links, anchor text optimization, and nofollow usage. - [Rules: Images & Core Web Vitals](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/images-and-cwv.md): Rules covering image accessibility, missing dimensions (CLS), broken images, and modern image formats. - [Rules: International & Security](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/international-and-security.md): Rules covering hreflang reciprocity, multi-lingual SEO, and essential web security headers. - [Rules: Schema & Structured Data](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/schema-and-structured-data.md): Rules covering JSON-LD syntax validation, schema.org types, and Google Rich Results eligibility. - [Rules: Graph & Internal PageRank](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/graph-and-internal-pagerank.md): Post-crawl graph analysis covering orphan pages, excessive click depth, and link equity flow. - [Rules: AI Search & GEO Readiness](https://blacksparrowdev.vercel.app/raw/docs/seo-rules/ai-and-geo-readiness.md): Rules covering Generative Engine Optimization, /llms.txt discovery, and AI bot crawler access. ## Architecture - [Crawler Engine & AIMD Politeness](https://blacksparrowdev.vercel.app/raw/docs/architecture/crawler-and-aimd.md): Deep dive into Black Sparrow's asynchronous Tokio architecture and dynamic congestion control. - [Streaming Parser & Memory Efficiency](https://blacksparrowdev.vercel.app/raw/docs/architecture/streaming-parser.md): Why Cloudflare's lol_html streaming parser keeps memory usage under 50 MB RAM. - [Storage & SQLite Schema](https://blacksparrowdev.vercel.app/raw/docs/architecture/storage-and-sqlite.md): Local-first persistence layer, WAL-mode database architecture, and SQL query recipes. - [Headless Chrome & JavaScript SEO](https://blacksparrowdev.vercel.app/raw/docs/architecture/headless-chrome.md): Decoupled Chrome DevTools Protocol (CDP) for crawling client-side JavaScript SPAs. ## AI & Model Context Protocol (MCP) - [Model Context Protocol (MCP) Setup](https://blacksparrowdev.vercel.app/raw/docs/ai-and-mcp/mcp-setup.md): Connect Black Sparrow's native MCP server to Claude Desktop, Cursor, and Windsurf. - [Native MCP Tools Reference](https://blacksparrowdev.vercel.app/raw/docs/ai-and-mcp/mcp-tools.md): Complete specification of all 8 native MCP tools exposed by Black Sparrow. - [Generative Engine Optimization (GEO)](https://blacksparrowdev.vercel.app/raw/docs/ai-and-mcp/geo-optimization.md): How to optimize your website for AI search engines, ChatGPT, Perplexity, and Claude. ## Developer Guide - [Building from Source](https://blacksparrowdev.vercel.app/raw/docs/developer-guide/building-from-source.md): Setup your local Rust development environment and compile Black Sparrow. - [Testing & Benchmarks](https://blacksparrowdev.vercel.app/raw/docs/developer-guide/testing-and-benchmarks.md): How to run unit, integration, and doc tests using cargo-nextest and synthetic fixtures. - [Adding Custom SEO Rules](https://blacksparrowdev.vercel.app/raw/docs/developer-guide/adding-custom-rules.md): Step-by-step guide to implementing new in-flight or graph rules in Black Sparrow.