← Signals

Deep dives

How real systems are built and how they fail: architecture, fault tolerance, postmortems, databases and performance work, from the engineering blogs of companies like Cloudflare, Netflix and Stripe and from people who write about distributed systems. Each article is read before it is kept; launches, marketing and hiring posts are left out.

October 2026

  1. Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

    A cross-tenant KV cache leak in long-context inference: how 155 responses exposed other customers' data, the containment timeline, and the SGLang fixes made afterward.

    Reliability & postmortems · Cloudflare · Oct 26
  2. A faster way to compute signed 64-bit floor division by constants

    A new algorithm for signed 64-bit floor division by compile-time constants, with per-divisor throughput and latency benchmarks across ARM and x86.

    Performance · Lobsters · Oct 26 · discussion
  3. Making Failure Diagnosis Reliable with Pensive AI

    How Netflix made automated failure diagnosis dependable enough to trust during incidents, and the checks that keep its root-cause conclusions reliable.

    Reliability & postmortems · Netflix · Oct 26
  4. Heavenly delights of hashing numbers under EQUALP

    How ECL made number hashing consistent under EQUALP so mathematically equal values collide correctly, with benchmarks of the resulting hash-table speedups.

    Performance · Lobsters · Oct 26 · discussion
  5. Twelve years of distributed SQLite

    A twelve-year retrospective on building rqlite, showing why putting a Raft log in front of SQLite makes a CP system and what that choice forced on reads and snapshots.

    Distributed systems · Lobsters · Oct 24 · discussion
  6. Memory is slow, Disk is fast

    Benchmarks show a disk-based structure on NVMe beating an in-RAM AVL tree in both memory and speed, arguing hardware trends make textbook Big O intuitions misleading.

    Performance · Lobsters · Oct 23 · discussion
  7. Origin Range Requests: Serving large files without range-request tuning

    Explains why caching HTTP range requests naively wastes CDN bandwidth, and how Cloudflare's rebuilt design halved p99 time-to-first-byte for object storage.

    Performance · Cloudflare · Oct 23
  8. Bringing post-quantum cryptography to geographically fenced key management

    Describes how Cloudflare wraps decryption keys so only servers in a chosen geography can unwrap them, combining post-quantum crypto with regional data-sovereignty constraints.

    Architecture · Cloudflare · Oct 23
  9. HPTS 2026 (Day 2 and Day 3)

    Conference notes summarizing systems research talks on GPU databases, datacenter power, object storage, failures, and consensus, with the trade-offs each design makes.

    Distributed systems · Murat Demirbas · Oct 23
  10. How close can pure FP get to C?

    A hands-on account of tuning a DEFLATE compressor in Roc, showing how close a functional language can get to hand-tuned C, with benchmark numbers.

    Performance · Lobsters · Oct 23 · discussion
  11. UUIDv7 doesn't have the properties you think it does

    Explains which ordering, uniqueness, and monotonicity properties UUIDv7 actually guarantees, and where the assumptions developers make about them break.

    Databases · Lobsters · Oct 23 · discussion
  12. Animations for System Builders: Metastability and Lost Writes

    Shows how animations build intuition for failure modes like metastable overload and how a Raft cluster can lose writes when one node's storage lies about fsync.

    Reliability & postmortems · Marc Brooker · Oct 23
  13. Single-tenant Titus: VM-level isolation on a multi-tenant container platform

    How Netflix gives tenants single-tenant VM-level isolation on a shared multi-tenant container platform, and the design trade-offs that choice forced.

    Architecture · Netflix · Oct 22
  14. We gave AI agents a mobile dev environment

    Building a remote service so many parallel AI-agent builds, simulators, and tests can run off-laptop, and the Mac-vs-Linux tooling constraints that shaped it.

    Infrastructure · Shopify · Oct 22
  15. The bridge to agent-scale Git infrastructure

    How GitHub replicated nearly all repositories to Azure, why per-reference quorums finish the rest, and how object storage lets read throughput scale independently.

    Architecture · GitHub · Oct 21
  16. Cloudflare deploys protections against php2shell vulnerability

    Coordinated response to the php2shell PHP flaws: why WAF rules needed request-plus-response evidence, and how attackers mutated payloads to evade protections.

    Reliability & postmortems · Cloudflare · Oct 21
  17. 8 eMudhra certificates are compromised. Here's what you need to know

    How Cloudflare responded to compromised intermediate certificates: prioritising affected services, refreshing shared trust stores, and rolling out patched clients and tunnels at scale.

    Infrastructure · Cloudflare · Oct 20
  18. Cross-store product clustering across millions of stores

    How Shopify groups product listings across millions of stores using tiered matching models, graph partitioning, and learned cluster-merge decisions at scale.

    Architecture · Shopify · Oct 20
  19. Jepsen: Orkes Conductor 5.3.2

    Jepsen's consistency analysis of Orkes Conductor, fault-injecting a distributed workflow engine to find correctness bugs in its coordination.

    Distributed systems · Lobsters · Oct 20 · discussion
  20. Introducing tsrs: A port of TypeScript to Rust

    Benchmarks and trade-offs behind porting the TypeScript compiler to Rust, showing how they got large speed and memory wins without forking the language.

    Performance · Cloudflare · Oct 20
  21. A Very Fast IPv4 Parser, in Swift

    A from-scratch IPv4 parser in Swift reaching ~3ns per address, covering the bit tricks, vectorisation, and benchmark methodology behind the result.

    Performance · Lobsters · Oct 20 · discussion
  22. Building Overexposed: understanding application exposure

    Shopify rebuilt its exposure scanner around cloud inventory and live probing with policy-mapped verdicts, after config-file scanning missed running services.

    Infrastructure · Shopify · Oct 19
  23. Cutting our inference bill by 68% by being obsessive about optimization

    Cloudflare cut its AI code reviewer's inference bill 68% via token usage profiling, prompt caching, and model routing, with real numbers and trade-offs.

    Performance · Cloudflare · Oct 19
  24. The Thin Orchestrator: How We Made Our AI Agents More Reliable by Giving the Router Less to Do

    Meta keeps agent orchestration thin, routing work and enforcing reliability checks while specialized agents reason, improving accuracy and efficiency.

    Reliability & postmortems · Meta · Oct 19
  25. Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster

    Meta details Prometheus, its gigawatt-scale supercluster: liquid cooling, networking, and the engineering behind scaling toward a million GPUs.

    Infrastructure · Meta · Oct 19
  26. Tor VPN Beta: What we've learned building our own VPN for Android from scratch

    Tor explains lessons from building an Android VPN client from scratch, covering networking internals, protocol choices, and platform constraints.

    Infrastructure · Lobsters · Oct 19 · discussion
  27. Superoptimizing an FPU emulator

    Shows how to brute-force search instruction sequences for a branchless formula replacing a branchy float bit-twiddling routine, and how to validate candidates with test cases.

    Performance · Lobsters · Oct 18 · discussion
  28. Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale

    Describes how Netflix built an end-to-end knowledge graph to unify observability metadata and query it across systems at scale.

    Infrastructure · Netflix · Oct 16
  29. The road to iroh 1.0

    Traces four years of design decisions behind iroh, moving from IPFS to a QUIC-based peer-to-peer blob transfer protocol, including rejected approaches.

    Infrastructure · Lobsters · Oct 16 · discussion
  30. 2026 World Cup: Breaking Records, Spiky Charts and Reliability at Scale

    How Meta's infrastructure absorbed record World Cup traffic spikes, and how pairing game events to egress surges revealed when load actually arrives.

    Reliability & postmortems · Meta · Oct 15
  31. End-to-end encrypted relays for OCapN

    How to build end-to-end encrypted relays for a capability-based network, covering relay design and the trade-offs of routing encrypted peer traffic.

    Distributed systems · Lobsters · Oct 15 · discussion
  32. TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

    How Netflix moved time-series reads from Kafka/Flink streams to Cassandra cold reads with tiered storage, and the trade-offs involved.

    Databases · Netflix · Oct 14
  33. Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II

    How Netflix re-architected its ads event pipeline to handle live traffic, with concrete engineering decisions for scaling real-time events.

    Distributed systems · Netflix · Oct 14
  34. Netflix Stratum Media Processing: Automated Container Right-Sizing

    How Netflix automated container right-sizing for media processing to cut compute waste, including the measurements and approach behind it.

    Infrastructure · Netflix · Oct 14
  35. Reliable GPU compute for large training & inference jobs

    How Netflix makes GPU compute reliable for large training and inference jobs, covering failure modes and resilience at scale.

    Reliability & postmortems · Netflix · Oct 14
  36. From 2,500 requests to 24: Rebuilding star history without the stargazers

    How GitHub rebuilt star history to cut 2,500 requests down to 24, showing the API and database trade-offs they chose.

    Performance · GitHub · Oct 14
  37. Multi-region disaster recovery for Amazon API Gateway WebSocket APIs

    Shows how to design multi-region failover for WebSocket APIs, combining DNS failover, DynamoDB Global Tables and health probes to restore sessions.

    Architecture · AWS Architecture · Oct 14
  38. Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug

    A hunt for a rare 0.001% query-caching error in a MySQL proxy, showing how wrong protocol assumptions create subtle production bugs.

    Reliability & postmortems · Cloudflare · Oct 14
  39. Under the hood, off the hot path: How D1 scales to millions of databases

    Explains how Cloudflare split D1's control plane from the edge data plane to provision and manage millions of small databases without slowing queries.

    Databases · Cloudflare · Oct 13
  40. Meta’s Generative Ads Model (GEM): Scaling the Central Model and Compute Engine for Frontier Ads AI

    Covers how Meta built and scaled GEM, an LLM-style ads recommendation model, including compute engine choices for training and serving at frontier scale.

    Architecture · Meta · Oct 13
  41. Rewinding the exponential: How we keep hardware usage down

    Describes how Cloudflare cut hardware use in its data pipeline by reworking throughput and processing while preserving data quality.

    Performance · Cloudflare · Oct 13
  42. Bulk Executor for Amazon DynamoDB, Part 2: Lessons we learned the hard way

    Second part of a DynamoDB bulk tool: the design tradeoffs, surprises and workarounds the team hit building bulk operations on AWS Glue.

    Databases · AWS Architecture · Oct 13
  43. A slightly faster slimmer curl

    How curl sped up base64, URL, and date parsing by replacing lookup functions, with measured speedups.

    Performance · Lobsters · Oct 13 · discussion
  44. Speeding up the Unreal editor by… not using unnecessarily secure hash methods

    How replacing a cryptographically secure hash with a faster non-crypto hash sped up Unreal editor shader compilation.

    Performance · Lobsters · Oct 12 · discussion
  45. Shipping Media Over QUIC at Meta Scale for Instagram Live Ingest

    How Meta deployed Media over QUIC for Instagram Live ingest, consolidating separate ingest protocols into one common transport.

    Infrastructure · Meta · Oct 12
  46. Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations

    How Meta safely uses idle inference capacity for ads ranking via two elasticity levers, isolation, and graceful load shedding.

    Architecture · Meta · Oct 12
  47. How shredding JSON is giving Logfire 1000x query speedups

    Explains dynamic shredding, an optimization that rewrites semi-structured attribute data so previously timing-out queries finish in under a second.

    Performance · Lobsters · Oct 11 · discussion
  48. Atlas Infinite: MongoDB's Disaggregated Architecture

    Describes how MongoDB disaggregated compute from storage in Atlas while keeping WiredTiger efficient, and the trade-offs that design required.

    Databases · Murat Demirbas · Oct 11
  49. Consistency is not a localized property

    Argues consistency is an end-to-end property no single component can own, using Kafka's exactly-once work to show why someone must own the global guarantee.

    Distributed systems · Lobsters · Oct 11 · discussion
  50. Making np.searchsorted up to 25× Faster in NumPy 2.5

    Shows vectorizing NumPy's binary search with branch elimination and cache-friendly layouts to replace per-key loops, reaching a 25x speedup.

    Performance · Lobsters · Oct 10 · discussion
  51. Vectorized CLZ and CTZ

    Explains how to vectorize count-leading-zeros and count-trailing-zeros with SIMD, the bit tricks involved, and the resulting speedups.

    Performance · Lobsters · Oct 9 · discussion
  52. Spinlocks Considered Harmful (2020)

    Explains why spinlocks hurt throughput and latency under contention, and which scheduling-aware alternatives avoid those costs.

    Performance · Lobsters · Oct 9 · discussion
  53. B-Trees Are Back: Engineering Fast and Pageable Node Layouts

    Shows cache- and page-aware node layouts that make B-trees fast and pageable, and the trade-offs against classic designs.

    Databases · Lobsters · Oct 9 · discussion
  54. The history of the Hetzner Cloud network stack

    Traces how Hetzner evolved its cloud network stack to a custom-orchestrated Open vSwitch design and the trade-offs of owning each layer.

    Infrastructure · Lobsters · Oct 8 · discussion
  55. The Performance Cost of RwLock in Our Read-Heavy Workload

    Compares RwLock against atomic replacement in a read-heavy concurrent service, with measurements showing where each approach wins or costs.

    Performance · Lobsters · Oct 8 · discussion
  56. When Submarine Cables Go Dark: Understanding and Preparing for the Risks of Taiwan's International Internet Disconnection

    Examines how submarine cable cuts could disconnect Taiwan and what resilience measures reduce the risk of major internet outages.

    Reliability & postmortems · Lobsters · Oct 7 · discussion
  57. Building an evidence-grounded agentic security operations harness on Cloudflare

    Shows how to build an AI-agent security ops system by separating deterministic evidence collection from model inference so recommendations stay grounded.

    Architecture · Cloudflare · Oct 7
  58. How fast is Python 3.15?

    Measures Python 3.15's runtime performance against earlier versions, with concrete benchmark numbers and analysis.

    Performance · Lobsters · Oct 7 · discussion
  59. Janet on x32: 32-bit Pointers, 64-bit Speed, 25% Less RAM

    Details how Janet's runtime uses 32-bit pointers on 64-bit systems to cut memory use 25% while keeping full speed.

    Performance · Lobsters · Oct 6 · discussion
  60. Building Git infrastructure for agent-scale development

    Describes how GitHub is rebuilding its Git infrastructure in place to handle agent-scale development without disrupting live service.

    Architecture · GitHub · Oct 6
  61. The keys to the Internet change on October 11. Are you ready?

    Explains how DNSSEC's root key-signing key rollover works, why resolvers must trust the new key, and how operators verify readiness.

    Infrastructure · Cloudflare · Oct 6
  62. Beyond synthetic testing: Capturing and replaying real database workloads at Airbnb

    Shows how Airbnb captures and replays real production database workloads to test upgrades instead of relying only on synthetic benchmarks.

    Databases · Airbnb · Oct 6
  63. Metrics Board: Building an Agent-ready Metrics Layer

    Describes Pinterest's metrics layer architecture for defining and serving metrics consistently to both people and agents.

    Architecture · Pinterest · Oct 6
  64. NTS: Authenticated Time at Meta

    Explains how Meta serves authenticated time with NTS, using stateless derived cookie keys so devices can verify time was not tampered with.

    Infrastructure · Meta · Oct 6
  65. Iroh global content discovery

    Dissects BitTorrent's Mainline DHT and transfer protocol to design permissionless global content discovery, covering DHT design and the trade-offs involved.

    Distributed systems · Lobsters · Oct 4 · discussion
  66. Upstream Rust maintenance report

    A maintainer's account of real Rust toolchain performance work: shrinking the target directory 5-35%, speeding up the parser and Polonius, and measuring compiler performance on ARM.

    Performance · Lobsters · Oct 3 · discussion
  67. Architect highly available Oracle Database on Amazon EVS and FSx for ONTAP

    Design trade-offs for running Oracle on VMware-in-AWS: bare metal sizing, separate vSAN and NFS storage tiers, and SnapMirror cross-region DR planning.

    Architecture · AWS Architecture · Oct 2
  68. Streamline: custom video pipelines with Cloudflare Stream and Workers

    A video pipeline built from long-lived containers, Durable Objects orchestration, and Workers control signaling, showing how to design media processing decoupled from request lifecycles.

    Architecture · Cloudflare · Oct 2
  69. 2026 Birthday week: network performance update

    How Cloudflare measures latency against the 1,000 largest networks, a new Challenge Pages methodology, and where network improvements moved real connection times.

    Performance · Cloudflare · Oct 2
  70. The Second Golden Spike: Memory Safety Across the Valen/Rust Boundary

    Explains how the Valen language upholds memory safety and borrow checking across a direct Rust boundary, and the design trade-offs of merging two safety models.

    Architecture · Lobsters · Oct 1 · discussion
  71. Announcing Cloudflare K2: serverless event streams

    Explains how K2 builds a durable partitioned event log on R2 object storage to decouple producers and consumers and never drop events.

    Distributed systems · Cloudflare · Oct 1
  72. Introducing Workers KV Instant — powered by Quicksilver

    Shows how Cloudflare reused its internal Quicksilver store to give Workers KV instant global replication with sub-2ms p99 reads and no cold reads.

    Infrastructure · Cloudflare · Oct 1

September 2026

  1. We used a database as a message queue. Now we use Kafka

    Why running task queues inside FoundationDB put read load on user requests, and how moving async work to Kafka cut that load and custom code.

    Distributed systems · Lobsters · Sep 30 · discussion
  2. Running multi-day AZ evacuation drills with ARC Zonal Shift

    How to run a 48-72 hour single-AZ evacuation drill to expose autoscaling, DNS, and connection failures short failover tests miss.

    Reliability & postmortems · AWS Architecture · Sep 30
  3. Cloudflare Containers, rebuilt to scale agent sandboxes

    How rearchitecting a container runtime and scheduler for on-demand sandboxes cut median startup from about 4s to 648ms.

    Architecture · Cloudflare · Sep 30
  4. Evolving our calendar assistant Reclaim to be AI-native without starting over

    How Dropbox added natural-language scheduling to Reclaim by layering AI over existing calendar logic rather than rebuilding, and the trade-offs that forced.

    Architecture · Dropbox · Sep 29
  5. Preventing quantum downgrade attacks against IPsec

    Explains quantum downgrade attacks on IPsec and the IETF mitigation Cloudflare implemented, showing how protocol peers are tricked into weaker cryptography.

    Infrastructure · Cloudflare · Sep 29
  6. Building a post-quantum certificate authority with Merkle Tree Certificates

    Describes Merkle Tree Certificates, a Web PKI redesign that makes transparency first-party and scales post-quantum signatures efficiently for certificate issuance.

    Architecture · Cloudflare · Sep 29
  7. Using AI to chart a course for our post-quantum migration

    Explains how Cloudflare plans a platform-wide post-quantum cryptography migration, from discovering where crypto is used to tracking progress and sequencing upgrades.

    Infrastructure · Cloudflare · Sep 29
  8. Supporting native Rust in Workers with the new Emscripten target for wasm-bindgen

    How a new Emscripten target unlocks previously unsupported Rust libraries and Tokio async on Workers, and the wasm-bindgen limits that had to be solved.

    Infrastructure · Cloudflare · Sep 28
  9. The road to the agentic browser: A Kitesurf update

    What it takes to build a Workers-based browser engine for agents: DOM performance work, WebMCP support, and rendering, backed by test-suite numbers.

    Infrastructure · Cloudflare · Sep 28
  10. Four months of VoidZero at Cloudflare: making the open-source JavaScript toolchain faster for all humans and agents

    How the VoidZero team sped up JavaScript compilation, linting and testing, including a 10x faster React compiler, and the trade-offs behind those gains.

    Performance · Cloudflare · Sep 28
  11. Tuning a Server for Benchmarking

    Shows how to configure and tune a server so benchmark measurements are accurate and reproducible instead of distorted by the host.

    Performance · Lobsters · Sep 26 · discussion
  12. Trading a Cloud Identity for Your Own: Workload Attestation on Managed Compute

    How Netflix issues and verifies its own workload identities on managed compute instead of trusting cloud provider identity.

    Infrastructure · Netflix · Sep 25
  13. Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework

    How Pinterest finalizes partitions in its next-generation DB ingestion framework, covering the coordination and correctness problems at scale.

    Databases · Pinterest · Sep 25
  14. Improving site performance by shipping more CSS

    How GitHub replaced CSS-in-JS across github.com with static CSS and cut rendering and runtime cost, with the migration tradeoffs.

    Performance · GitHub · Sep 25
  15. How Cloudflare addressed a cross-tenant data exposure vulnerability in Containers

    An incident postmortem of a cross-tenant disk data exposure in multi-tenant containers, explaining dm-thin storage behavior, impact, and the fix.

    Reliability & postmortems · Cloudflare · Sep 24
  16. Rendering huge pull requests in the GitHub Copilot app

    How GitHub rebuilt its diff renderer to open million-line pull requests with hundreds of inline comments without freezing the UI.

    Performance · GitHub · Sep 23
  17. I want my mesh networks to be signed, not encrypted

    Argues mesh networks should authenticate and sign traffic rather than only encrypt it, illustrating a concrete networking design trade-off.

    Infrastructure · Lobsters · Sep 23 · discussion
  18. Latest BGP hijack targets hosting software vendor

    A real BGP hijack incident shows how internet routing can be subverted and what it exposes about trust in the protocol.

    Infrastructure · Lobsters · Sep 23 · discussion
  19. We just shipped support for the ugliest part of HTTP: Vary

    Explains the HTTP Vary header, why caches mishandle it, and how to normalize, pass through, or bypass cache keys for ambiguous content negotiation.

    Infrastructure · Cloudflare · Sep 22
  20. Helix: The internal tool powering our Shopify app's native migration

    How Shopify's Helix tool drives LLM rewrites of its app into Swift and Kotlin using small checkpoints and strict quality gates.

    Architecture · Shopify · Sep 21
  21. In Search of a Compositional Theory of Self-Stabilization

    Explains why composing self-stabilizing systems is hard and how that lens helps diagnose and prevent metastable failures in distributed systems.

    Distributed systems · Murat Demirbas · Sep 21
  22. Open-Sourcing Rebalancer: A Generic, High-Performance Library for Solving Assignment Problems

    How Meta designed a generic assignment-problem solver for resource allocation, separating problem specification, storage, solving, and debugging.

    Architecture · Meta · Sep 21
  23. Inside Petal: Building the World’s First Petabit-Class Transoceanic Subsea Cable

    The engineering behind a petabit-class transoceanic subsea cable, including deploying multi-core fiber at scale over 7,000 km.

    Infrastructure · Meta · Sep 21
  24. How Notion handles concurrent editing with CRDTs

    Explains how Notion models shared documents as CRDTs, the concrete concurrency problems that creates, and the trade-offs chosen.

    Distributed systems · Lobsters · Sep 20 · discussion
  25. Saving another 100TB of RAM with math (and Rust)

    How Cloudflare reclaimed over 100TB of RAM by fixing memory-hungry consistent hashing in Rust, walking through the math, profiling, and algorithm changes.

    Performance · Cloudflare · Sep 18
  26. Leave the Class Path in the Rearview Mirror

    Why Netflix moved its JVM services off the traditional classpath, what it uses instead, and the tradeoffs involved.

    Architecture · Netflix · Sep 18
  27. The guest journey, updated in real time: extending Airbnb’s sequence recommender with Chronon

    How Airbnb extended its sequence recommender to refresh the guest journey in real time with Chronon, and the streaming pipeline trade-offs involved.

    Architecture · Airbnb · Sep 17
  28. Beyond the model: Engineering AI infra with scientific judgement

    How Airbnb engineers build and operate AI infrastructure, using scientific judgement to decide what to build versus buy and how to validate it.

    Infrastructure · Airbnb · Sep 15