Streaming Parser & Memory Efficiency

Most web scrapers and crawlers use DOM tree parsers (such as Cheerio, BeautifulSoup, or Chromium DOM trees). To parse a page, they construct a full object hierarchy in RAM representing every HTML tag, attribute, and text node.

When crawling thousands of pages, this DOM-heavy approach requires gigabytes of memory.

Zero-Copy Streaming with lol_html#

Black Sparrow uses lol_html, an ultra-fast HTML5 parser developed by Cloudflare:

• Token-Level Streaming: As network bytes arrive via TCP, lol_html extracts <title>, <meta>, <h1>, and <a> tags in a single forward pass without buffering the entire document.
• Immediate Extraction: As soon as link href targets and image alt attributes are collected, raw HTML chunks are freed from memory.
• Zero Panics: The parser is 100% resilient against malformed, incomplete, or corrupted HTML sent by broken servers.

Memory Optimizations in Rust#

1. compact_str for Small Strings#

Most web URLs, HTML tag names, and MIME types are under 24 bytes. Traditional Rust String objects allocate 24 bytes on the stack plus a separate heap allocation.

Black Sparrow uses compact_str::CompactString. Any string of 24 bytes or less is stored inline on the stack, completely eliminating heap allocations for the vast majority of extracted tokens!

2. bitflags for Robots Directives#

Rather than storing boolean flags (is_noindex: bool, is_nofollow: bool), Black Sparrow packs all robots and indexing directives into a single 16-bit integer using the bitflags crate (RobotsFlags).

3. Locality-Sensitive 64-bit SimHash#

To detect near-duplicate pages without storing full body text in memory, Black Sparrow computes a 64-bit SimHash fingerprint of the page content. Two pages with a Hamming distance of 3 or less are flagged as duplicate content.

Empirical Benchmark#

Auditing 1,000 pages of typical documentation and e-commerce content: