Deep dives
How real systems are built and how they fail: architecture, fault tolerance, postmortems, databases and performance work, from the engineering blogs of companies like Cloudflare, Netflix and Stripe and from people who write about distributed systems. Each article is read before it is kept; launches, marketing and hiring posts are left out.
October 2026
- Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
A cross-tenant KV cache leak in long-context inference: how 155 responses exposed other customers' data, the containment timeline, and the SGLang fixes made afterward.
- A faster way to compute signed 64-bit floor division by constants
A new algorithm for signed 64-bit floor division by compile-time constants, with per-divisor throughput and latency benchmarks across ARM and x86.
- Making Failure Diagnosis Reliable with Pensive AI
How Netflix made automated failure diagnosis dependable enough to trust during incidents, and the checks that keep its root-cause conclusions reliable.
- Heavenly delights of hashing numbers under EQUALP
How ECL made number hashing consistent under EQUALP so mathematically equal values collide correctly, with benchmarks of the resulting hash-table speedups.
- Twelve years of distributed SQLite
A twelve-year retrospective on building rqlite, showing why putting a Raft log in front of SQLite makes a CP system and what that choice forced on reads and snapshots.
- Memory is slow, Disk is fast
Benchmarks show a disk-based structure on NVMe beating an in-RAM AVL tree in both memory and speed, arguing hardware trends make textbook Big O intuitions misleading.
- Origin Range Requests: Serving large files without range-request tuning
Explains why caching HTTP range requests naively wastes CDN bandwidth, and how Cloudflare's rebuilt design halved p99 time-to-first-byte for object storage.
- Bringing post-quantum cryptography to geographically fenced key management
Describes how Cloudflare wraps decryption keys so only servers in a chosen geography can unwrap them, combining post-quantum crypto with regional data-sovereignty constraints.
- HPTS 2026 (Day 2 and Day 3)
Conference notes summarizing systems research talks on GPU databases, datacenter power, object storage, failures, and consensus, with the trade-offs each design makes.
- How close can pure FP get to C?
A hands-on account of tuning a DEFLATE compressor in Roc, showing how close a functional language can get to hand-tuned C, with benchmark numbers.
- UUIDv7 doesn't have the properties you think it does
Explains which ordering, uniqueness, and monotonicity properties UUIDv7 actually guarantees, and where the assumptions developers make about them break.
- Animations for System Builders: Metastability and Lost Writes
Shows how animations build intuition for failure modes like metastable overload and how a Raft cluster can lose writes when one node's storage lies about fsync.
- Single-tenant Titus: VM-level isolation on a multi-tenant container platform
How Netflix gives tenants single-tenant VM-level isolation on a shared multi-tenant container platform, and the design trade-offs that choice forced.
- We gave AI agents a mobile dev environment
Building a remote service so many parallel AI-agent builds, simulators, and tests can run off-laptop, and the Mac-vs-Linux tooling constraints that shaped it.
- The bridge to agent-scale Git infrastructure
How GitHub replicated nearly all repositories to Azure, why per-reference quorums finish the rest, and how object storage lets read throughput scale independently.
- Cloudflare deploys protections against php2shell vulnerability
Coordinated response to the php2shell PHP flaws: why WAF rules needed request-plus-response evidence, and how attackers mutated payloads to evade protections.
- 8 eMudhra certificates are compromised. Here's what you need to know
How Cloudflare responded to compromised intermediate certificates: prioritising affected services, refreshing shared trust stores, and rolling out patched clients and tunnels at scale.
- Cross-store product clustering across millions of stores
How Shopify groups product listings across millions of stores using tiered matching models, graph partitioning, and learned cluster-merge decisions at scale.
- Jepsen: Orkes Conductor 5.3.2
Jepsen's consistency analysis of Orkes Conductor, fault-injecting a distributed workflow engine to find correctness bugs in its coordination.
- Introducing tsrs: A port of TypeScript to Rust
Benchmarks and trade-offs behind porting the TypeScript compiler to Rust, showing how they got large speed and memory wins without forking the language.
- A Very Fast IPv4 Parser, in Swift
A from-scratch IPv4 parser in Swift reaching ~3ns per address, covering the bit tricks, vectorisation, and benchmark methodology behind the result.
- Building Overexposed: understanding application exposure
Shopify rebuilt its exposure scanner around cloud inventory and live probing with policy-mapped verdicts, after config-file scanning missed running services.
- Cutting our inference bill by 68% by being obsessive about optimization
Cloudflare cut its AI code reviewer's inference bill 68% via token usage profiling, prompt caching, and model routing, with real numbers and trade-offs.
- The Thin Orchestrator: How We Made Our AI Agents More Reliable by Giving the Router Less to Do
Meta keeps agent orchestration thin, routing work and enforcing reliability checks while specialized agents reason, improving accuracy and efficiency.
- Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster
Meta details Prometheus, its gigawatt-scale supercluster: liquid cooling, networking, and the engineering behind scaling toward a million GPUs.
- Tor VPN Beta: What we've learned building our own VPN for Android from scratch
Tor explains lessons from building an Android VPN client from scratch, covering networking internals, protocol choices, and platform constraints.
- Superoptimizing an FPU emulator
Shows how to brute-force search instruction sequences for a branchless formula replacing a branchy float bit-twiddling routine, and how to validate candidates with test cases.
- Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale
Describes how Netflix built an end-to-end knowledge graph to unify observability metadata and query it across systems at scale.
- The road to iroh 1.0
Traces four years of design decisions behind iroh, moving from IPFS to a QUIC-based peer-to-peer blob transfer protocol, including rejected approaches.
- 2026 World Cup: Breaking Records, Spiky Charts and Reliability at Scale
How Meta's infrastructure absorbed record World Cup traffic spikes, and how pairing game events to egress surges revealed when load actually arrives.
- End-to-end encrypted relays for OCapN
How to build end-to-end encrypted relays for a capability-based network, covering relay design and the trade-offs of routing encrypted peer traffic.
- TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
How Netflix moved time-series reads from Kafka/Flink streams to Cassandra cold reads with tiered storage, and the trade-offs involved.
- Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II
How Netflix re-architected its ads event pipeline to handle live traffic, with concrete engineering decisions for scaling real-time events.
- Netflix Stratum Media Processing: Automated Container Right-Sizing
How Netflix automated container right-sizing for media processing to cut compute waste, including the measurements and approach behind it.
- Reliable GPU compute for large training & inference jobs
How Netflix makes GPU compute reliable for large training and inference jobs, covering failure modes and resilience at scale.
- From 2,500 requests to 24: Rebuilding star history without the stargazers
How GitHub rebuilt star history to cut 2,500 requests down to 24, showing the API and database trade-offs they chose.
- Multi-region disaster recovery for Amazon API Gateway WebSocket APIs
Shows how to design multi-region failover for WebSocket APIs, combining DNS failover, DynamoDB Global Tables and health probes to restore sessions.
- Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug
A hunt for a rare 0.001% query-caching error in a MySQL proxy, showing how wrong protocol assumptions create subtle production bugs.
- Under the hood, off the hot path: How D1 scales to millions of databases
Explains how Cloudflare split D1's control plane from the edge data plane to provision and manage millions of small databases without slowing queries.
- Meta’s Generative Ads Model (GEM): Scaling the Central Model and Compute Engine for Frontier Ads AI
Covers how Meta built and scaled GEM, an LLM-style ads recommendation model, including compute engine choices for training and serving at frontier scale.
- Rewinding the exponential: How we keep hardware usage down
Describes how Cloudflare cut hardware use in its data pipeline by reworking throughput and processing while preserving data quality.
- Bulk Executor for Amazon DynamoDB, Part 2: Lessons we learned the hard way
Second part of a DynamoDB bulk tool: the design tradeoffs, surprises and workarounds the team hit building bulk operations on AWS Glue.
- A slightly faster slimmer curl
How curl sped up base64, URL, and date parsing by replacing lookup functions, with measured speedups.
- Speeding up the Unreal editor by… not using unnecessarily secure hash methods
How replacing a cryptographically secure hash with a faster non-crypto hash sped up Unreal editor shader compilation.
- Shipping Media Over QUIC at Meta Scale for Instagram Live Ingest
How Meta deployed Media over QUIC for Instagram Live ingest, consolidating separate ingest protocols into one common transport.
- Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations
How Meta safely uses idle inference capacity for ads ranking via two elasticity levers, isolation, and graceful load shedding.
- How shredding JSON is giving Logfire 1000x query speedups
Explains dynamic shredding, an optimization that rewrites semi-structured attribute data so previously timing-out queries finish in under a second.
- Atlas Infinite: MongoDB's Disaggregated Architecture
Describes how MongoDB disaggregated compute from storage in Atlas while keeping WiredTiger efficient, and the trade-offs that design required.
- Consistency is not a localized property
Argues consistency is an end-to-end property no single component can own, using Kafka's exactly-once work to show why someone must own the global guarantee.
- Making np.searchsorted up to 25× Faster in NumPy 2.5
Shows vectorizing NumPy's binary search with branch elimination and cache-friendly layouts to replace per-key loops, reaching a 25x speedup.
- Vectorized CLZ and CTZ
Explains how to vectorize count-leading-zeros and count-trailing-zeros with SIMD, the bit tricks involved, and the resulting speedups.
- Spinlocks Considered Harmful (2020)
Explains why spinlocks hurt throughput and latency under contention, and which scheduling-aware alternatives avoid those costs.
- B-Trees Are Back: Engineering Fast and Pageable Node Layouts
Shows cache- and page-aware node layouts that make B-trees fast and pageable, and the trade-offs against classic designs.
- The history of the Hetzner Cloud network stack
Traces how Hetzner evolved its cloud network stack to a custom-orchestrated Open vSwitch design and the trade-offs of owning each layer.
- The Performance Cost of RwLock in Our Read-Heavy Workload
Compares RwLock against atomic replacement in a read-heavy concurrent service, with measurements showing where each approach wins or costs.
- When Submarine Cables Go Dark: Understanding and Preparing for the Risks of Taiwan's International Internet Disconnection
Examines how submarine cable cuts could disconnect Taiwan and what resilience measures reduce the risk of major internet outages.
- Building an evidence-grounded agentic security operations harness on Cloudflare
Shows how to build an AI-agent security ops system by separating deterministic evidence collection from model inference so recommendations stay grounded.
- How fast is Python 3.15?
Measures Python 3.15's runtime performance against earlier versions, with concrete benchmark numbers and analysis.
- Janet on x32: 32-bit Pointers, 64-bit Speed, 25% Less RAM
Details how Janet's runtime uses 32-bit pointers on 64-bit systems to cut memory use 25% while keeping full speed.
- Building Git infrastructure for agent-scale development
Describes how GitHub is rebuilding its Git infrastructure in place to handle agent-scale development without disrupting live service.
- The keys to the Internet change on October 11. Are you ready?
Explains how DNSSEC's root key-signing key rollover works, why resolvers must trust the new key, and how operators verify readiness.
- Beyond synthetic testing: Capturing and replaying real database workloads at Airbnb
Shows how Airbnb captures and replays real production database workloads to test upgrades instead of relying only on synthetic benchmarks.
- Metrics Board: Building an Agent-ready Metrics Layer
Describes Pinterest's metrics layer architecture for defining and serving metrics consistently to both people and agents.
- NTS: Authenticated Time at Meta
Explains how Meta serves authenticated time with NTS, using stateless derived cookie keys so devices can verify time was not tampered with.
- Iroh global content discovery
Dissects BitTorrent's Mainline DHT and transfer protocol to design permissionless global content discovery, covering DHT design and the trade-offs involved.
- Upstream Rust maintenance report
A maintainer's account of real Rust toolchain performance work: shrinking the target directory 5-35%, speeding up the parser and Polonius, and measuring compiler performance on ARM.
- Architect highly available Oracle Database on Amazon EVS and FSx for ONTAP
Design trade-offs for running Oracle on VMware-in-AWS: bare metal sizing, separate vSAN and NFS storage tiers, and SnapMirror cross-region DR planning.
- Streamline: custom video pipelines with Cloudflare Stream and Workers
A video pipeline built from long-lived containers, Durable Objects orchestration, and Workers control signaling, showing how to design media processing decoupled from request lifecycles.
- 2026 Birthday week: network performance update
How Cloudflare measures latency against the 1,000 largest networks, a new Challenge Pages methodology, and where network improvements moved real connection times.
- The Second Golden Spike: Memory Safety Across the Valen/Rust Boundary
Explains how the Valen language upholds memory safety and borrow checking across a direct Rust boundary, and the design trade-offs of merging two safety models.
- Announcing Cloudflare K2: serverless event streams
Explains how K2 builds a durable partitioned event log on R2 object storage to decouple producers and consumers and never drop events.
- Introducing Workers KV Instant — powered by Quicksilver
Shows how Cloudflare reused its internal Quicksilver store to give Workers KV instant global replication with sub-2ms p99 reads and no cold reads.
September 2026
- We used a database as a message queue. Now we use Kafka
Why running task queues inside FoundationDB put read load on user requests, and how moving async work to Kafka cut that load and custom code.
- Running multi-day AZ evacuation drills with ARC Zonal Shift
How to run a 48-72 hour single-AZ evacuation drill to expose autoscaling, DNS, and connection failures short failover tests miss.
- Cloudflare Containers, rebuilt to scale agent sandboxes
How rearchitecting a container runtime and scheduler for on-demand sandboxes cut median startup from about 4s to 648ms.
- Evolving our calendar assistant Reclaim to be AI-native without starting over
How Dropbox added natural-language scheduling to Reclaim by layering AI over existing calendar logic rather than rebuilding, and the trade-offs that forced.
- Preventing quantum downgrade attacks against IPsec
Explains quantum downgrade attacks on IPsec and the IETF mitigation Cloudflare implemented, showing how protocol peers are tricked into weaker cryptography.
- Building a post-quantum certificate authority with Merkle Tree Certificates
Describes Merkle Tree Certificates, a Web PKI redesign that makes transparency first-party and scales post-quantum signatures efficiently for certificate issuance.
- Using AI to chart a course for our post-quantum migration
Explains how Cloudflare plans a platform-wide post-quantum cryptography migration, from discovering where crypto is used to tracking progress and sequencing upgrades.
- Supporting native Rust in Workers with the new Emscripten target for wasm-bindgen
How a new Emscripten target unlocks previously unsupported Rust libraries and Tokio async on Workers, and the wasm-bindgen limits that had to be solved.
- The road to the agentic browser: A Kitesurf update
What it takes to build a Workers-based browser engine for agents: DOM performance work, WebMCP support, and rendering, backed by test-suite numbers.
- Four months of VoidZero at Cloudflare: making the open-source JavaScript toolchain faster for all humans and agents
How the VoidZero team sped up JavaScript compilation, linting and testing, including a 10x faster React compiler, and the trade-offs behind those gains.
- Tuning a Server for Benchmarking
Shows how to configure and tune a server so benchmark measurements are accurate and reproducible instead of distorted by the host.
- Trading a Cloud Identity for Your Own: Workload Attestation on Managed Compute
How Netflix issues and verifies its own workload identities on managed compute instead of trusting cloud provider identity.
- Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework
How Pinterest finalizes partitions in its next-generation DB ingestion framework, covering the coordination and correctness problems at scale.
- Improving site performance by shipping more CSS
How GitHub replaced CSS-in-JS across github.com with static CSS and cut rendering and runtime cost, with the migration tradeoffs.
- How Cloudflare addressed a cross-tenant data exposure vulnerability in Containers
An incident postmortem of a cross-tenant disk data exposure in multi-tenant containers, explaining dm-thin storage behavior, impact, and the fix.
- Rendering huge pull requests in the GitHub Copilot app
How GitHub rebuilt its diff renderer to open million-line pull requests with hundreds of inline comments without freezing the UI.
- I want my mesh networks to be signed, not encrypted
Argues mesh networks should authenticate and sign traffic rather than only encrypt it, illustrating a concrete networking design trade-off.
- Latest BGP hijack targets hosting software vendor
A real BGP hijack incident shows how internet routing can be subverted and what it exposes about trust in the protocol.
- We just shipped support for the ugliest part of HTTP: Vary
Explains the HTTP Vary header, why caches mishandle it, and how to normalize, pass through, or bypass cache keys for ambiguous content negotiation.
- Helix: The internal tool powering our Shopify app's native migration
How Shopify's Helix tool drives LLM rewrites of its app into Swift and Kotlin using small checkpoints and strict quality gates.
- In Search of a Compositional Theory of Self-Stabilization
Explains why composing self-stabilizing systems is hard and how that lens helps diagnose and prevent metastable failures in distributed systems.
- Open-Sourcing Rebalancer: A Generic, High-Performance Library for Solving Assignment Problems
How Meta designed a generic assignment-problem solver for resource allocation, separating problem specification, storage, solving, and debugging.
- Inside Petal: Building the World’s First Petabit-Class Transoceanic Subsea Cable
The engineering behind a petabit-class transoceanic subsea cable, including deploying multi-core fiber at scale over 7,000 km.
- How Notion handles concurrent editing with CRDTs
Explains how Notion models shared documents as CRDTs, the concrete concurrency problems that creates, and the trade-offs chosen.
- Saving another 100TB of RAM with math (and Rust)
How Cloudflare reclaimed over 100TB of RAM by fixing memory-hungry consistent hashing in Rust, walking through the math, profiling, and algorithm changes.
- Leave the Class Path in the Rearview Mirror
Why Netflix moved its JVM services off the traditional classpath, what it uses instead, and the tradeoffs involved.
- The guest journey, updated in real time: extending Airbnb’s sequence recommender with Chronon
How Airbnb extended its sequence recommender to refresh the guest journey in real time with Chronon, and the streaming pipeline trade-offs involved.
- Beyond the model: Engineering AI infra with scientific judgement
How Airbnb engineers build and operate AI infrastructure, using scientific judgement to decide what to build versus buy and how to validate it.