OneNoughtOne
0/97
Loading content...
Design a web search engine like Google that crawls the entire internet, builds an inverted index of hundreds of billions of pages, and returns the most relevant results for any text query in under 500ms — at a scale of 8.5 billion searches per day. This encompasses web crawling, indexing, query understanding, multi-signal ranking (BM25 + PageRank + ML), snippet generation, and serving with massive scatter-gather fan-out.
| Metric | Value |
|---|---|
| Indexed web pages | 100+ billion |
| Queries per day | 8.5 billion |
| Peak queries per second | 200,000 |
| Index size (compressed) | ~100 petabytes |
| Index shards | 10,000+ |
| Index shard replicas | 3× per data centre |
| Data centres | 20+ globally |
| Crawler pages fetched per day | 1+ billion |
| Unique terms in index | 1+ trillion |
| Average query latency | < 500ms |
| Result cache hit rate | ~30% (head queries) |
Relevance: Results must be highly relevant — BM25 for lexical matching, PageRank for authority, BERT for semantic understanding; hundreds of ranking signals; ML re-ranker for final ordering
Latency: < 500ms end-to-end — despite fanning out to thousands of index shards; achieved via in-memory Tier 1 index, early termination, hedged requests, and aggressive caching
Freshness: Breaking news searchable within minutes via realtime supplemental index (Kafka pipeline); base index rebuilt periodically for full corpus optimisation
Scale: 100K–200K QPS, each query fans to 10,000+ shards; geo-distributed complete index replicas; tail latency managed with hedged requests and partial results
Fault tolerance: Any shard or data centre can fail without affecting availability; query serves partial results on timeout; replicated index across data centres
Comprehensiveness: Crawl and index the entire public web; prioritise high-quality and frequently-changing pages; avoid spam and low-quality content
Web search: given a text query, return a ranked list of the most relevant web pages from a corpus of hundreds of billions of indexed pages, in < 500ms
Web crawling: continuously discover and fetch web pages across the entire internet; handle billions of pages; respect robots.txt; prioritise important/fresh pages; detect and avoid crawler traps
Indexing: parse crawled HTML, extract text/metadata, build an inverted index mapping every word to the list of documents containing it; support phrase queries, boolean operators, and field-specific search (title, URL)
Ranking: rank search results by relevance using hundreds of signals — link analysis (PageRank), content relevance (TF-IDF / BM25), user engagement (click-through rate), freshness, page quality, mobile-friendliness, and more
Query understanding: interpret user intent — spell correction ('did you mean'), synonym expansion, entity recognition ('Apple' = company vs fruit based on context), query rewriting, and knowledge graph lookups
Snippet generation: for each result, display a title, URL, and a contextual snippet (excerpt from the page highlighting the query terms); featured snippets for direct-answer queries
Autocomplete: as the user types, suggest completions based on popular queries, personalised history, and trending topics (refer H0013 — Search Autocomplete)
Image / video / news search: support vertical search across different content types; image search with visual similarity; news search with recency ranking; video search with thumbnail previews
Knowledge panel: for entity queries (people, places, companies), display a structured knowledge card with summary, facts, images, and related entities sourced from the knowledge graph
Personalisation and localisation: personalise results based on user location, language, search history, and device; localise results (local businesses, maps, currency) based on geo-IP
Non-functional requirements define the system qualities critical to your users. Frame them as 'The system should be able to...' statements. These will guide your deep dives later.
Think about CAP theorem trade-offs, scalability limits, latency targets, durability guarantees, security requirements, fault tolerance, and compliance needs.
Frame NFRs for this specific system. 'Low latency search under 100ms' is far more valuable than just 'low latency'.
Add concrete numbers: 'P99 response time < 500ms', '99.9% availability', '10M DAU'. This drives architectural decisions.
Choose the 3-5 most critical NFRs. Every system should be 'scalable', but what makes THIS system's scaling uniquely challenging?
The complete explanation is included with Career and All Access. Compare it with your approach once you’ve worked through the problem.
Your system-design submission history is available after you sign in.
Sign In