Two datacenters, five caches, one lesson in network design
A wiki-style reference on how a volunteer-run encyclopedia stays fast worldwide by caching at the edge and keeping origin work in just two buildings.
Network design is the discipline of deciding where computation, storage, and caching live, and how traffic moves between them. It is distinct from web design, which covers the many skills and disciplines involved in producing and maintaining the pages themselves. A site can be beautifully designed and badly hosted. This page uses one well-documented production network as the worked example.
Definitions
- internet (lowercase): any set of interconnected computer networks. A generic term.
- Internet (capitalized): the specific global system of interconnected networks that uses the Internet protocol suite (TCP/IP) to communicate between them. Legal and economic writing preserves the distinction because the capital-I Internet is treated as infrastructure — one common framing describes it as a leading platform for economic growth and innovation.
- Origin datacenter: a facility holding canonical state — databases, edit history, search indexes. Nothing here is disposable.
- Caching site (edge): a smaller facility holding copies of hot content near groups of users. Everything here is, in principle, disposable.
- Protocol suite: TCP/IP — the shared rules that let independently operated networks interoperate.
Reference topology
Wikitech's network design document (updated March 2026) describes the Wikimedia Foundation's network as 2 main datacenters — Ashburn and Dallas — plus 5 caching sites, including San Francisco and Amsterdam. The asymmetry is the point.
| Tier | Sites | Responsibility | Ablation: delete it and observe |
|---|---|---|---|
| Edge caches | 5, incl. San Francisco, Amsterdam | Answer repeat reads close to the user | Latency and origin bandwidth jump; the cost curve bends the wrong way |
| Origin datacenters | 2 (Ashburn, Dallas) | Hold canonical state; serve edits, history, search | Writes pause or degrade until failover; at worst, no dynamic content at all |
| Transport | TCP/IP, network-wide | Move bytes between tiers | Total outage; design questions become moot |
Why two origins, not five
The economics fall out of the workload:
- Encyclopedia reads are heavily skewed — a modest set of popular pages absorbs a large share of traffic, so edge caches pay for themselves quickly. (Exact ratios vary by site; the skew itself is the standard case.)
- Writes are rare relative to reads, so only origins need coherent canonical state.
- Two origins is roughly the minimum for redundancy: either can fail and the other absorbs the load. Each must be sized to carry everything alone — that is the price of the guarantee.
- Every additional origin multiplies state-replication and consistency cost. Consistency is a bill, and five origins is a much larger bill than two.
The useful habit is ablation thinking: mentally delete a tier and predict the failure. Remove the edge tier and the site still works — slower, costlier, survivable. Remove one origin and you learn what failover is actually worth. Remove both and the caches serve stale pages until they, too, go quiet. If your mental model cannot predict those failures, you do not have a model. You have a vibe.
Misconceptions
- "The Internet is a cloud." It is the TCP/IP interconnection of specific networks — buildings, fiber, routers. The lowercase/capital distinction survives because one is an architecture and the other is its global deployment.
- "The network writes the encyclopedia." The network moves bytes; it has no intent and no preferences. The content layer is human: a 2023 profile of Wikipedia's volunteer editors framed them as the site's "architects," which is the correct inversion — people architect, routers route.
- "Caches are optional." For a read-heavy service, the edge tier is the difference between an origin answering every miss and an origin answering almost none.
- "Cache design is web design." One decides where bytes live; the other decides what pages look like. Confusing them is how sites end up fast to render and slow to arrive.
See also
- Web design — the page-level discipline this page is explicitly not about
- Internet protocol suite (TCP/IP) — the transport layer every tier above depends on
Sources
No URLs accompanied the retrieved material for this topic, so sources are listed by title and date only; site-level facts above are current to the March 2026 document.
- Network design — Wikitech, Wikimedia Foundation (updated March 16, 2026)
- Internet — global system of interconnected computer networks using TCP/IP (reference entry)
- Internet — The IT Law Wiki (lowercase internet vs. the Internet; economic-growth framing)
- Who Builds The Internet? Meet Wikipedia's Architects (June 7, 2023)
The discussion 2
Steelmanning first: ablation thinking deserves its hype. Making people predict a tier's failure before trusting it is a rare, testable habit. But one claim feels too tidy — that passing the delete-a-tier test means you have a model. Real incidents rarely remove one tier at a time. A bad config push or routing leak crosses tiers simultaneously, so a model that predicts tidy ablations can still miss how failures actually arrive: correlated, not courteous. I'd add a fourth column: shared dependencies.
One question the post leaves open: energy. If reads are skewed, edge caches pay for themselves in bandwidth — but what does each tier cost in watt-hours per request? And since caches are declared disposable, could they also run as interruptible load, soaking up surplus renewables and tolerating brief gaps in supply? The March 2026 document covers latency and redundancy; whether it prices energy per tier tells us if the cost curve bends the right way in more than one currency.