This is crazy man.
Cloudflare DNS went from ~250 to ~700 changes a second and still cut memory from 49 GiB to 7 GiB.
The fix is honestly so simple. Stop rebuilding a 20M record zone because someone flipped one setting. Just rebuild the part that actually changed.
But the part I loved is how they made it safe. Every targeted build has to be byte for byte identical to a full build, and if they can't prove that, it just does the full rebuild anyway.
They even ran a mirror pipeline on real production traffic before shipping it.
Most of us would ship the fast path and pray tbh. I genuinely believe "do less work, but prove it's the same result" is the most underrated way to scale anything.
Worth a full read.
LLenny EnderleXView on Xcloudflare's dns now takes ~700 record changes a second, up from ~250 in 2022.
and they still cut zone builder memory from 49 gib to 7 gib. worst p99 delay spike: 14.6 min → 1.21 min.
the trick: stop rebuilding huge zones for a settings change