Mini-CDN
CDN simulation in Go: one origin, three edge caches, a router with consistent hashing (or round-robin), partial-failure ring sync, load shedding, TTL + LRU disk cache, Prometheus/Grafana.
Problem
To understand how a CDN behaves on a miss, when an entry expires, when the cache is full, when the origin or an edge is down, and when the router is overloaded, I built a small one end to end in Go.
Architecture
- Clients talk to the router on :8080. The router picks a healthy edge (consistent hash by path, or round-robin) and reverse-proxies. Over the in-flight budget → 503 with X-Load-Shed: 1 and Retry-After.
- Hash ring: FNV-1a with virtual nodes (HASH_VNODES, default 100). Ring membership follows /healthz probes every 2 s via SetNodes. Distinct host:port values stay distinct when several edges share a hostname.
- GET and HEAD try one other healthy edge when the selected edge fails before writing a response (X-Edge-Failover). POST is not retried.
- Each edge has a disk cache with an in-memory index. On a miss, the origin body is teed to the client and a temp file, then atomically renamed. Serve stale on origin failure (X-Cache: STALE) only if the client response has not started.
- Metrics include cdn_hash_ring_members, cdn_load_shed_total, cdn_edge_healthy, cache hits/misses/stale serves, and response duration histograms. Grafana is provisioned; Prometheus scrapes router, edges, and origin.
Design decisions
Consistent hashing with virtual nodes
ROUTER_STRATEGY=consistent (default in compose) maps each path onto healthy edges via FNV-1a virtual nodes so the same file prefers one edge.
Trade-off: Better cache locality than round-robin; membership changes remaps some keys. Round-robin remains available for demos.
Health-synced ring and GET/HEAD failover
Unhealthy edges leave the ring. If the selected edge fails before a response starts, GET/HEAD retry one other healthy edge once.
Trade-off: Partial failure remaps traffic without a full outage; POST is not retried to avoid duplicate side effects.
Load shedding via MAX_INFLIGHT
Router and edges refuse work above the in-flight budget with 503, X-Load-Shed, and Retry-After instead of queueing unboundedly.
Trade-off: Protects latency under overload; clients must back off or retry.
Stream while caching
On a miss, the origin body is copied to the response and a temp file together, then renamed into place.
Trade-off: No extra wait for the first client, and a partial download never becomes a cache entry. Mid-body origin errors are not followed by a second body.
Path safety first
CleanRequestPath rejects '..' segments and SafeJoin keeps every disk path inside the cache root, both with unit tests.
Trade-off: A small amount of extra code that closes path-traversal bugs on both origin and edges.
Stack
What I'd improve next
- Single-flight concurrent misses for the same key, so a cold file hits the origin once per edge.
- An O(1) LRU (map + doubly linked list) instead of the linear scan.
- A repeatable load test that reports hit ratio and latency.
Results
- edge cache nodes behind a consistent-hash / round-robin router
- 3