Skip to content
INTERNET INFRASTRUCTURE & NETWORKS

Cloudflare Overhauls DNS Propagation Pipeline, Cutting Peak Memory Usage by 85 Percent

Cloudflare has announced a major architectural overhaul of its DNS propagation pipeline, drastically reducing memory consumption, accelerating update speeds, and addressing the scaling challenges posed by massive enterprise domains. The update marks a significant milestone in how the global infrastructure provider handles billions of Domain Name System records across its extensive edge network.

When Cloudflare last detailed its DNS propagation pipeline in 2022, its central "Zone Builder" service processed roughly 250 DNS record changes every second. Today, that volume has nearly tripled to approximately 700 record changes per second, accompanied by an additional 100 secondary events, such as zone setting adjustments and product configurations.

However, company engineers note that raw event volume was never the primary hurdle. Instead, the true engineering bottleneck lay in the computational complexity of certain events and how that workload scales with the physical size of a zone. Some customer zones hosted on Cloudflare now exceed 20 million individual DNS records, with projections pointing toward hundreds of millions in the near future.

How DNS Changes Reach the Edge

For any domain, a DNS zone represents the collective repository of its records and settings. Cloudflare relies on a complex, multi-stage pipeline to translate any modification made to a zone into actionable data accessible across its global network.

When a user updates a DNS record or modifies a zone setting via the API or dashboard, the event is processed through a series of internal services and Kafka messaging queues. The DNS Kafka Aggregator compiles these signals before passing them to the Zone Builder. Rather than directly copying raw database rows to the edge, the Zone Builder combines database records with specific zone settings, applies complex DNS serving rules, and organizes the output into a specialized format designed for rapid retrieval by authoritative DNS servers.

Historically, the original Zone Builder operated on a simple principle: it completely ignored the specific nature of an event and instead rebuilt the desired state from scratch by reading the current database. While this approach guaranteed accuracy, it required reading, encoding, and comparing every single record while holding the entire zone in memory simultaneously. For smaller domains, this method was inexpensive. For massive enterprise zones with tens of millions of records, however, the data could no longer fit into memory, leading to severe resource contention and pipeline bottlenecks.

The Challenge of Zone Settings and Targeted Builds

The engineering team previously solved record-level scaling by introducing per-record builds, which proved to be significantly faster than full-zone reconstructions. On a test zone containing one million records, per-record builds operated thousands of times faster than traditional full builds. The obvious next step was to extend this targeted approach to zone settings.

How we halved DNS build delay and cut memory use by 85%

However, proving when it was safe to target specific settings proved to be an intricate engineering challenge. While a record event clearly identifies what changed, a zone-level event can originate from a wide array of Cloudflare products, often without explicitly detailing the exact nature of the modification. Furthermore, these settings can alter data shared by multiple zones that share the same domain name.

Choosing too much work wastes valuable computing capacity, while choosing too little risks leaving stale or incorrect DNS data exposed at the edge. To solve this safely, Cloudflare developed a targeted build planner capable of selecting the smallest possible build that could be mathematically proven safe, paired with a unified scheduler that intelligently combines concurrent work destined for the same zone.

Minimizing Resource Consumption and Eliminating Bottlenecks

The implementation of the targeted build planner and unified scheduler has fundamentally transformed pipeline efficiency. Zone setting changes no longer force the system to rebuild zones containing tens of millions of records. Together, these architectural improvements have slashed peak memory utilization by 85 percent, bringing the aggregate working-set memory down from an average of 49 gigabytes to just 7 gigabytes, with peak usage dropping from 79 gigabytes to 12 gigabytes.

This newfound headroom allowed engineers to safely shorten necessary resilience waits within the pipeline. By refining how the system handles database replication lag and zone rebuild intervals, Cloudflare halved the median build completion delay from roughly 7.7 seconds down to approximately 3.3 seconds. More importantly, the tail latency saw dramatic improvements, with the worst-case p99 delay dropping twelvefold from 14.6 minutes down to roughly 1.21 minutes during peak traffic periods.

To ensure absolute reliability during the rollout, Cloudflare deployed a mirror pipeline that consumed production Kafka streams and database states while writing to an isolated Quicksilver configuration environment that served no customer traffic. A dedicated comparison engine continuously sampled zones for discrepancies, allowing engineers to verify that targeted builds produced byte-for-byte identical outputs compared to legacy full builds.

While full builds have not been completely eliminated—as they remain necessary for first-time zone publications and broad structural changes that impact an entire domain—their overall frequency has dropped by roughly 40 percent. By ensuring that large zones are no longer subjected to redundant full-scale reconstructions, Cloudflare has significantly hardened its infrastructure against future growth, ensuring rapid and reliable DNS propagation for millions of internet properties worldwide.