Requirements: Crawl the Web - Web Crawler | OneNoughtOne

0/273

System Design (HLD)Web Crawler

Designing a Web Crawler at Scale

LevelAdvanced

Duration90 mins

TopicWeb Crawler

1 / 6

Requirements: Crawl the Web

The Engine Behind Search

When you type a query into Google and receive millions of relevant results in under 300 milliseconds, you're witnessing the culmination of one of the most ambitious distributed systems ever built. But before any page can be ranked, retrieved, or displayed—it must first be discovered.

This discovery is the domain of the web crawler, also known as a spider, bot, or web robot. A web crawler is a systematic, automated system that traverses the World Wide Web, downloading pages, extracting links, and building an index of discoverable content. It is the foundational component that makes search engines, price comparison sites, archival systems, and content aggregators possible.

Designing a web crawler presents unique engineering challenges that don't exist in typical application development. The 'database' you're querying—the internet itself—contains billions of dynamically changing pages, is spread across millions of servers you don't control, and requires you to respect the resources and policies of each individual site you visit. Get it wrong, and you'll either miss critical content or become an unwelcome burden on the web's infrastructure.

What You Will Learn

By the end of this page, you will understand: (1) The functional and non-functional requirements for a production-grade web crawler, (2) How to estimate scale for crawling billions of pages, (3) The unique constraints that make web crawling different from typical distributed systems, and (4) The core architecture decisions that shape crawler design.

What Is a Web Crawler?

A web crawler is an automated program that systematically browses the World Wide Web, typically for the purpose of indexing web content. Unlike a human browsing the web, a crawler operates at massive scale—potentially visiting billions of pages across millions of domains.

The fundamental loop of a web crawler:

Seed URLs: Start with an initial set of URLs (seeds)
Fetch: Download the content of a page via HTTP/HTTPS
Parse: Extract useful content and discover new URLs (links)
Store: Save the content for indexing or processing
Queue: Add newly discovered URLs to the frontier
Repeat: Continue until some termination condition

This deceptively simple loop becomes extraordinarily complex at scale. Each step involves distributed systems challenges: How do you store trillions of URLs efficiently? How do you avoid re-downloading unchanged pages? How do you respect each website's rate limits? How do you handle dynamic JavaScript-rendered content?

Web Crawler Use Cases
Use Case	Example	Key Requirements
Search Engine Indexing	Google, Bing, DuckDuckGo	Comprehensive coverage, freshness, ranking signals
Price Comparison	Google Shopping, Kayak	Frequent updates, structured data extraction
Web Archiving	Internet Archive, Common Crawl	Historical preservation, complete snapshots
SEO Tools	Ahrefs, Moz, Semrush	Link analysis, site structure, metadata
Research & Analytics	Academic crawlers, brand monitoring	Targeted crawls, sentiment analysis
Content Aggregation	News aggregators, RSS readers	Real-time updates, content classification

Crawler vs. Scraper

While often confused, crawlers and scrapers serve different purposes. A crawler focuses on discovering and traversing the web structure—finding URLs and building a graph of the web. A scraper focuses on extracting specific data from pages. In practice, production systems combine both: crawling to discover, scraping to extract. This module focuses primarily on the crawling aspects, though we'll address content extraction as well.

Functional Requirements

Before diving into architecture, we must precisely define what our web crawler needs to do. Functional requirements describe the core capabilities the system must provide.

For a production-grade web crawler targeting search engine scale, the functional requirements are extensive:

Core Functional Requirements

•URL Discovery — Given a set of seed URLs, discover all reachable pages by recursively following hyperlinks. The crawler should build a comprehensive map of the discoverable web.
•Page Fetching — Download web pages over HTTP/HTTPS, handling redirects, authentication challenges, and various content types (HTML, PDF, images, etc.).
•Content Storage — Store fetched content for downstream processing (indexing, analysis, archival). This includes both the raw HTML and extracted metadata.
•Link Extraction — Parse downloaded pages to extract hyperlinks for further crawling. Handle relative URLs, canonical URLs, and various link formats.
•Recrawl Scheduling — Revisit previously crawled pages to detect updates. Prioritize recrawls based on page importance and change frequency.
•Robots.txt Compliance — Respect each domain's robots.txt directives, honoring disallow rules and crawl-delay specifications.
•URL Normalization — Canonicalize URLs to avoid crawling the same content under different URL representations.
•Domain-Level Throttling — Limit the request rate to individual domains to avoid overloading target servers.

Extended Functional Requirements

•JavaScript Rendering — For Single Page Applications and JavaScript-heavy sites, render pages using a headless browser to discover dynamically-loaded content and links.
•Sitemap Processing — Parse XML sitemaps to discover URLs that may not be reachable through hyperlink traversal.
•DNS Resolution Caching — Cache DNS lookups to reduce latency and avoid overwhelming DNS infrastructure.
•Content Deduplication — Detect and avoid storing duplicate content that appears under different URLs (mirrors, URL parameters).
•Crawl Frontier Management — Maintain a prioritized queue of URLs to crawl, balancing breadth, depth, freshness, and importance.
•Metadata Extraction — Extract structured metadata (title, description, OpenGraph, schema.org) for downstream processing.
•Error Handling & Retry — Handle transient failures gracefully with exponential backoff and retry logic.
•Crawl Statistics & Monitoring — Provide real-time visibility into crawl progress, error rates, and throughput.

Requirements Clarification—Interview Strategy:

In a system design interview, you won't have time to address all requirements. The key is to:

Enumerate the full scope of requirements to demonstrate breadth of thinking
Prioritize which requirements to focus on given time constraints
Clarify any ambiguities with the interviewer (e.g., "Should we support JavaScript rendering?", "What's our target freshness SLA?")

For this module, we'll assume we're building a general-purpose web crawler for search engine indexing—the most demanding use case that subsumes most others.

Non-Functional Requirements

Non-functional requirements define the quality attributes of the system—how it should perform rather than what it should do. For a web crawler, these requirements are particularly challenging because you're interacting with an external, uncontrolled environment (the internet).

Performance Requirements

•Throughput — The crawler should sustain fetching 1,000+ pages per second at steady state, scaling to 10,000+ pages/second for large deployments. This translates to crawling billions of pages per month.
•Latency — Individual page fetches should complete within reasonable timeouts (5-30 seconds), but end-to-end crawl latency for a new URL should be minutes to hours, not days.
•Freshness — High-priority pages (news sites, frequently updated content) should be recrawled within hours. Lower-priority pages can be recrawled weekly or monthly.
•Coverage — The crawler should achieve comprehensive coverage of the discoverable web, minimizing the 'dark web' of pages reachable only through non-standard means.

Scalability Requirements

•Horizontal Scalability — The system must scale horizontally to handle the web's size. No single machine can crawl the entire web, so we need a distributed architecture that adds capacity by adding nodes.
•URL Frontier Scale — The URL frontier (queue of URLs to crawl) must handle billions of URLs efficiently, supporting priority-based selection and deduplication.
•Storage Scale — Content storage must accommodate petabytes of crawled data, with efficient retrieval for indexing and analysis.
•Per-Domain Parallelism — While respecting per-domain rate limits, the system should maximize parallelism across different domains to fully utilize available bandwidth.

Reliability Requirements

•Fault Tolerance — Individual node failures should not halt the crawl. The system should recover gracefully and redistribute work to healthy nodes.
•No URL Loss — Once a URL is discovered, it should not be lost due to system failures. The frontier must be durable.
•Consistency — The system should avoid duplicate crawls of the same URL at the same time, which wastes resources and may violate politeness policies.
•Graceful Degradation — Under resource pressure, the system should prioritize high-value pages and gracefully reduce throughput rather than failing catastrophically.

The Politeness Constraint

Unlike most distributed systems, a web crawler has an explicit politeness requirement: it must not harm the servers it crawls. This means respecting robots.txt, implementing per-domain rate limiting (typically 1 request per 1-10 seconds per domain), and using appropriate User-Agent identification. Violating politeness can get your crawler blocked, blacklisted, or even result in legal action. This constraint fundamentally shapes the architecture.

Scale Estimation: Back-of-Envelope Calculations

Understanding the scale of the web is essential for designing a crawler. Let's work through the numbers to appreciate the engineering challenge.

The Size of the Web:

The indexed web contains approximately 50+ billion pages (various estimates range from 40-130 billion)
There are approximately 200+ million active domains
The average web page size is approximately 2-3 MB (including all resources), though HTML alone averages ~100-200 KB
The web grows at approximately 1-3% per month in terms of new pages
Page content changes at varying rates: news might change hourly; corporate pages might change monthly

Scale Estimation for a Search Engine Crawler
Metric	Estimate	Derivation
Target Pages to Crawl	50 billion pages	Comprehensive coverage goal
Crawl Cycle Duration	30 days (average)	Balance freshness with resources
Pages Per Day	~1.67 billion	50B ÷ 30 days
Pages Per Second	~19,000 pages/second	1.67B ÷ 86,400 seconds
HTML Content Per Page	~150 KB average	Compressed HTML only
Storage Per Day (HTML)	~250 TB/day	1.67B × 150KB
Storage Per Month	~7.5 PB/month	250TB × 30 days
Outbound Bandwidth	~30 Gbps sustained	250TB ÷ 86,400s × 8 bits

The Politeness Bottleneck:

Here's where web crawling becomes uniquely constrained. Even though we want to fetch 19,000 pages per second, we cannot hit any single domain that fast.

Typical politeness policies:

Maximum 1 request per second per domain (aggressive)
1 request per 5-10 seconds per domain (conservative)
Must honor Crawl-delay directive in robots.txt

Implications:

With 200 million domains and 1 request/second/domain, theoretical max throughput is 200M pages/second—far more than we need
But domains are not equally sized: the top domains (Wikipedia, Amazon, news sites) have millions of pages each
Crawling Wikipedia at 1 page/second would take ~2 months for its 60M+ pages
This creates a long tail problem: small sites are easy to crawl completely; large sites require extended crawl periods

The parallelism equation:

To achieve 19,000 pages/second while respecting 1 request/second/domain:

We need to be crawling at least 19,000 different domains concurrently
This requires sophisticated scheduling to balance domain coverage with overall throughput

Scale Dictates Architecture

These numbers make it clear why web crawlers are inherently distributed systems. No single machine can handle 30+ Gbps of network I/O, store petabytes of content, or manage billions of URLs in a queue. The architecture must partition work across hundreds or thousands of machines while maintaining coordination for politeness, deduplication, and priority scheduling.

Unique Challenges of Web Crawling

Web crawling presents a unique combination of challenges that distinguish it from typical distributed systems. Understanding these challenges is essential for making sound architectural decisions.

External System Challenges

•Uncontrolled Data Source — Unlike internal systems, you don't control the web. Pages can disappear, move, or change format without notice.
•Adversarial Environment — Some sites actively try to block crawlers, serve different content to bots, or trap crawlers in infinite loops (spider traps).
•Heterogeneous Content — Pages use different encodings, formats, HTML versions, and rendering technologies (JavaScript frameworks, WebAssembly, etc.).
•Network Unreliability — You're making millions of requests to servers across the globe, through varying network conditions, with unpredictable latency and failure rates.

Internal System Challenges

•Massive URL Frontier — Storing and efficiently querying billions of URLs while maintaining priority ordering and deduplication is a significant data structure challenge.
•Distributed Coordination — Multiple workers must coordinate to respect per-domain rate limits without central bottlenecks or races.
•Duplicate Detection — The same content appears under many URLs. Detecting duplicates at scale requires probabilistic data structures and careful design.
•Crawl Prioritization — With more URLs than you can crawl, you must intelligently prioritize. This requires understanding page importance before you've visited the page.

Spider Traps and Pathological Cases

•Infinite Calendar Loops — Calendar pages that generate links for every day into the infinite future (e.g., /events/2025/01/01, /events/2025/01/02, ... forever).
•Session ID URLs — Sites that embed session IDs in URLs, creating infinite unique URLs for the same content (/page?sid=abc123, /page?sid=def456, ...).
•Symbolic Link Cycles — Directory listings with symbolic links that create infinite directory depth (/foo/bar/foo/bar/foo/bar/...).
•Dynamically Generated Content — Pages that generate new links containing random or incremented values, creating an unbounded crawl space.
•Soft 404s — Pages that return HTTP 200 with 'Page Not Found' content, causing the crawler to waste resources on non-existent pages.

The Crawler's Dilemma

Web crawlers face a fundamental tension: being too aggressive wastes resources and harms servers; being too conservative misses content and allows freshness to suffer. Every design decision involves balancing completeness vs. politeness, freshness vs. efficiency, and breadth vs. depth. There is no universally 'correct' answer—only tradeoffs appropriate to your specific use case.

Core Architectural Decisions

Before diving into detailed component design, let's outline the high-level architectural decisions that shape a production web crawler.

Key Architectural Choices

•Distributed Architecture — Given the scale, a distributed architecture with multiple crawler workers is mandatory. Work is partitioned across workers, typically by domain or URL hash.
•URL Frontier as Central Queue — The URL frontier acts as the distributed task queue. It must support priority-based scheduling, domain-level grouping, and efficient deduplication.
•Domain-Based Sharding — To enforce politeness, URLs are typically sharded by domain. Each domain is 'owned' by specific workers who manage its crawl rate.
•Separate Fetching and Processing — Page fetching (I/O bound) is separated from content processing (CPU bound), allowing independent scaling.
•Content Store Separation — Fetched content is stored in a separate content store (object storage, HDFS) rather than inline in the frontier database.
•Centralized DNS and Robots.txt Caching — To avoid redundant external calls, DNS lookups and robots.txt files are cached centrally and shared across workers.

High-Level Architecture Overview:

┌─────────────────────────────────────────────────────────────────────────────┐
│                           WEB CRAWLER ARCHITECTURE                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌─────────────┐    ┌─────────────────────────────────────────────────┐   │
│   │ Seed URLs   │───▶│              URL FRONTIER                        │   │
│   └─────────────┘    │  ┌─────────────────────────────────────────┐    │   │
│                      │  │ Priority Queues (by domain)             │    │   │
│   ┌─────────────┐    │  │ - high-priority-queue                   │    │   │
│   │ Sitemaps    │───▶│  │ - medium-priority-queue                 │    │   │
│   └─────────────┘    │  │ - low-priority-queue                    │    │   │
│                      │  └─────────────────────────────────────────┘    │   │
│                      │  ┌─────────────────────────────────────────┐    │   │
│                      │  │ Seen URLs (deduplication)               │    │   │
│                      │  │ - Bloom filter / database               │    │   │
│                      │  └─────────────────────────────────────────┘    │   │
│                      └─────────────────────────────────────────────────┘   │
│                                         │                                   │
│                                         ▼                                   │
│                      ┌─────────────────────────────────────────────────┐   │
│                      │            CRAWLER WORKERS                      │   │
│                      │  ┌─────────┐ ┌─────────┐ ┌─────────┐           │   │
│                      │  │Worker 1 │ │Worker 2 │ │Worker N │  ...      │   │
│                      │  └─────────┘ └─────────┘ └─────────┘           │   │
│                      │       │           │           │                 │   │
│                      │       └───────────┼───────────┘                 │   │
│                      └───────────────────┼─────────────────────────────┘   │
│                                          │                                  │
│                                          ▼                                  │
│   ┌─────────────────────────────────────────────────────────────────────┐  │
│   │                         THE INTERNET                                 │  │
│   │  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐       │  │
│   │  │ Site A  │ │ Site B  │ │ Site C  │ │ Site D  │ │ Site E  │ ...  │  │
│   │  └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘       │  │
│   └─────────────────────────────────────────────────────────────────────┘  │
│                                          │                                  │
│                                          ▼                                  │
│   ┌─────────────────────────────────────────────────────────────────────┐  │
│   │                      PROCESSING PIPELINE                             │  │
│   │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────┐   │  │
│   │  │ Link Extractor│──│ Content Parser│──│ URL Normalizer        │   │  │
│   │  └───────────────┘  └───────────────┘  └───────────────────────┘   │  │
│   │         │                   │                     │                 │  │
│   │         ▼                   ▼                     ▼                 │  │
│   │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────┐   │  │
│   │  │ New URLs ─────┼──│ Content Store │  │ Back to Frontier      │   │  │
│   │  └───────────────┘  └───────────────┘  └───────────────────────┘   │  │
│   └─────────────────────────────────────────────────────────────────────┘  │
│                                                                             │
│   ┌─────────────────────────────────────────────────────────────────────┐  │
│   │                      SUPPORT SERVICES                                │  │
│   │  ┌────────────┐  ┌────────────────┐  ┌────────────────────────┐    │  │
│   │  │ DNS Cache  │  │ Robots.txt Svc │  │ Politeness Scheduler   │    │  │
│   │  └────────────┘  └────────────────┘  └────────────────────────┘    │  │
│   └─────────────────────────────────────────────────────────────────────┘  │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

This architecture separates concerns cleanly: the URL Frontier manages what to crawl next; Crawler Workers handle the actual fetching; the Processing Pipeline extracts links and content; and Support Services provide shared infrastructure for DNS, robots.txt, and rate limiting.

API Design

While a web crawler is primarily an internal system, it still requires well-defined APIs for operation and integration. Let's define the key interfaces.

Crawler Control API

// ============================================================
// CRAWLER CONTROL API
// ============================================================
 
// Submit seed URLs to start or extend a crawl
POST /api/v1/seeds
Request:
{
  "urls": [
    "https://example.com",
    "https://another-site.org"
  ],
  "priority": "high",       // high, medium, low
  "crawl_depth": 10,        // maximum link depth from seed
  "options": {
    "follow_robots": true,
    "render_javascript": false,
    "extract_metadata": true
  }
}
Response:
{
  "job_id": "crawl-2024-001",
  "urls_queued": 2,
  "estimated_completion": "2024-01-15T12:00:00Z"
}
 
// Query crawl status
GET /api/v1/crawls/{job_id}/status
Response:
{
  "job_id": "crawl-2024-001",
  "status": "running",
  "urls_queued": 150000,
  "urls_crawled": 45230,
  "urls_failed": 1203,
  "bytes_downloaded": "125.4 GB",
  "started_at": "2024-01-13T08:00:00Z",
  "estimated_completion": "2024-01-15T12:00:00Z"
}
 
// Retrieve crawled content
GET /api/v1/content/{url_hash}
Response:
{
  "url": "https://example.com/page",
  "crawled_at": "2024-01-14T10:30:00Z",
  "http_status": 200,
  "content_type": "text/html",
  "content_hash": "sha256:abc123...",
  "content_url": "s3://crawl-data/2024/01/abc123.html.gz",
  "metadata": {
    "title": "Example Page",
    "description": "An example web page",
    "links_found": 45,
    "canonical_url": "https://example.com/page"
  }
}

Internal Component APIs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
// ============================================================
// URL FRONTIER API (Internal)
// ============================================================
 
// Add URLs to the frontier
AddURLs(urls: []URL, priority: Priority, source: string) -> AddResult
 
// Get next URLs to crawl for a specific domain partition
GetNextURLs(partition: int, limit: int) -> []CrawlTask
 
// Mark URL as crawled (success or failure)
MarkComplete(url: URL, result: CrawlResult) -> void
 
// Check if URL has been seen
IsURLSeen(url: URL) -> bool
 
// ============================================================
// POLITENESS SCHEDULER API (Internal)
// ============================================================
 
// Check if we can crawl this domain now
CanCrawl(domain: string) -> (bool, waitTime: Duration)
 
// Record a request to a domain (updates rate limiting state)
RecordRequest(domain: string, timestamp: Time) -> void
 
// Get crawl delay for a domain (from robots.txt)
GetCrawlDelay(domain: string) -> Duration
 
// ============================================================
// CONTENT STORE API (Internal)
// ============================================================
 
// Store crawled content
Store(url: URL, content: []byte, metadata: Metadata) -> ContentID
 
// Retrieve content by URL or content hash
Get(id: ContentID) -> (content: []byte, metadata: Metadata)
 
// Check if content already exists (for deduplication)
Exists(contentHash: string) -> (bool, existingURL: URL)

Summary and Next Steps

We've established a comprehensive foundation for designing a production web crawler. Let's consolidate the key insights before diving into detailed component design.

Key Takeaways

•Web crawlers are discovery engines — They systematically traverse the web to build an index of discoverable content, powering search engines and countless other applications.
•Scale is extreme — Crawling billions of pages requires distributed systems that handle petabytes of data, billions of URLs, and thousands of requests per second.
•Politeness is paramount — Unlike most systems, crawlers must actively limit their throughput to avoid harming the servers they interact with. This constraint fundamentally shapes the architecture.
•The web is adversarial — Spider traps, dynamic content, and hostile countermeasures require robust defenses and heuristics to avoid wasting resources.
•Core components are identifiable — URL Frontier, Crawler Workers, Processing Pipeline, and Support Services form the backbone of any production crawler.
•Tradeoffs are everywhere — Coverage vs. politeness, freshness vs. efficiency, breadth vs. depth—every decision involves explicit tradeoffs that depend on your use case.

What's Next:

In the following pages, we'll dive deep into each core component:

Page 2: URL Frontier Management — How to store, prioritize, and deduplicate billions of URLs efficiently
Page 3: Politeness Policies — Implementing robots.txt compliance and rate limiting at scale
Page 4: Duplicate Detection — Detecting duplicate content using fingerprinting and probabilistic data structures
Page 5: Distributed Crawling — Scaling horizontally while maintaining coordination and consistency
Page 6: Content Extraction — Parsing HTML, extracting links, and handling diverse content types

Page Complete

You now have a comprehensive understanding of web crawler requirements, scale, and architectural foundations. The following pages will transform these requirements into detailed component designs, each addressing specific challenges of crawling the web at scale.