Operating at internet scale often means managing resources that stretch the limits of human intuition. For Cloudflare, maintaining a global network comprising thousands of servers equipped with petabytes of RAM and millions of CPU cores requires relentless efficiency. Every available resource is pushed to its absolute limit to handle massive volumes of web traffic. Yet, even with such vast infrastructure, hardware remains finite. When an organization demands that every core service run on every available network node, there is simply no room for wasted space or inefficient code.
At this extraordinary scale, even minor performance tweaks are heavily magnified. Improvements that yield a meager one percent gain are met with celebration because they translate into massive real-world savings. Recently, the engineering teams at Cloudflare achieved a dramatic reduction in the memory footprint of a core internal service by refining a single foundational algorithm. This optimization successfully reclaimed more than 100 terabytes of RAM globally across the company’s edge infrastructure. This massive memory recovery builds directly on the heels of a separate optimization effort by the DNS team, which successfully shed another 100 terabytes of memory the previous month.
Waste Not: Tracking Down Excessive Memory
Ensuring equitable resource sharing across numerous engineering teams in a massive organization is a persistent challenge. At Cloudflare, this delicate balance is maintained through the continuous monitoring and analysis performed by the internal Performance team. The story of this latest memory breakthrough began with an internal support ticket filed by engineer Ivan, who flagged a distinct operational anomaly. The ticket highlighted excessive memory usage originating from pingora-ketama within the Pingora Backend Router, affectionately known as PBR.
The investigation revealed that Cloudflare’s internal load-balancing service was consuming significantly more memory than anticipated. Specifically, the bloated memory footprint was traced directly to data structures associated with pingora-ketama, an open-source library utilized by the company to handle consistent hashing. To understand how the engineering team addressed this unexpected resource consumption, it is necessary to examine the fundamental mechanics of consistent hashing, its specific role within the Pingora Backend Router, and how it gradually became so memory-hungry.
The Mechanics of Consistent Hashing
Consistent hashing is a widely adopted algorithmic method used for distributing tasks and requests across multiple servers in a way that minimizes disruption when servers are dynamically added or removed from a network. Internally, Cloudflare utilizes this method to route cacheable HTTP requests to specific servers based on their unique URLs. This precise routing strategy ensures that the network maintains only a single copy of a file stored per data center, providing a stable, predictable mechanism to locate any cached file instantly.
The core concept behind consistent hashing relies on hash functions. While a hash function can accept virtually any kind of input data—such as an IP address or a cache key—its output is strictly limited to a single unsigned integer, typically spanning 32, 64, or 128 bits depending on the specific function used. This mathematical constraint allows systems to relate tasks and servers to one another in a reliable, repeatable manner.
In theoretical discussions, the output space of a hash function is traditionally visualized as a continuous, circular ring that wraps around from its maximum numerical value back to zero. While this ring depiction provides helpful visualizations, it can sometimes make simple integer ranges appear unnecessarily complex. For practical engineering analysis, the 32-bit output of the hash function can be effectively represented as a straightforward linear number line.
When a set of servers and a set of tasks are mapped onto this number line based on the cryptographic hashes of their representative values, assigning tasks to servers becomes a straightforward matter of identifying the first server located immediately to the left of each task. However, this foundational approach reveals a structural flaw: the random nature of hash distributions means that individual server ranges can vary wildly in size. Because the fraction of total requests a server handles is directly proportional to the size of its assigned range on the number line, uneven ranges lead directly to imbalanced server workloads.
Math, Variance, and the Consequences of Randomness
To quantify the distribution of workloads across servers, engineers must rely on fundamental statistical concepts: expected value and standard deviation. In simplified terms, the expected value establishes the center point where measurements based on a distribution will cluster, while the standard deviation indicates how closely most measurements adhere to that central point.
For consistent hashing, these statistical factors can be calculated for the fractional size of the range associated with any single server out of a total population of servers. For instance, in a scenario featuring 100 servers, the expected workload for any individual server centers around one percent of the total traffic. The standard deviation under these conditions suggests that most range lengths will fall very close to that expected percentage of the total length.
However, looking at the raw standard deviation alone can be misleading. Engineers must scale the standard deviation against the expected value to determine the true magnitude of error relative to the target size, a metric known as the coefficient of variation. When calculated for a standard deployment of 100 servers using basic consistent hashing, the coefficient of variation approaches nearly 100 percent. This statistical reality means that some servers are forced to work twice as hard as intended by handling vastly disproportionate volumes of requests, while other servers sit nearly idle.
The Traditional Remedy: Adding More Hashes
The fundamental simplicity of consistent hashing is a double-edged sword. It remains easy to understand and implement because every operational component is translated into easily relatable hashes along a single number line. Consequently, any optimization to the system must also be expressed in terms of that same number line, meaning the primary solution to consistent hashing imbalances is to introduce additional hashes.
To combat imbalanced workloads, systems can assign multiple hash points to represent each individual server instead of relying on a single hash. Intuitively, while an individual random range exhibits a high standard deviation, combining multiple independent ranges together causes their cumulative size to even out. This behavior aligns directly with the law of large numbers, wherein aggregated random segments compensate for one another, balancing out overly short or excessively long spans.
The limitation of this approach is that it relies on large numbers to be effective. In mainstream web servers like NGINX, the baseline number of hashes per server is hardcoded to 160, and Pingora adopted this same default value. Applying 160 points per server instead of just one dramatically improves the coefficient of variation, dropping the error margin from roughly 99 percent down to about eight percent.
Weighting Workloads and Managing Exponential Growth
While a constant number of hashes per server helps distribute workloads evenly, network operators frequently encounter situations where uniform distribution is undesirable. Within Cloudflare’s infrastructure, certain servers feature significantly more storage capacity than others. Ideally, the volume of requests routed to a given server should remain directly proportional to its available disk space.
This requirement is fulfilled by the ketama algorithm, which scales the number of hashes assigned to a server based on an assigned weight. If an operator wants a specific server to handle a multiple of the traffic directed to a standard server, the algorithm simply multiplies the number of associated hashes by that weighting factor. For storage-heavy nodes, disk capacity serves as the natural weight. In compute-intensive environments, weights might instead reflect CPU or GPU core counts.
However, a final architectural challenge remains: in real-world operations, not every server can handle every request. Compliance mandates, geo-restrictions, and specific caching features mean that only designated subsets of servers are permitted to process particular requests. Unlike simple load balancing adjustments, this complexity cannot be resolved by merely adding more hashes to a single ring. Instead, engineers must deploy entirely separate hash rings, with every distinct combination of operational features potentially requiring its own dedicated ring.
This architectural requirement triggers an exponential explosion of data structures. With multiple distinct operational features, the number of required consistent hash rings scales exponentially into dozens of separate structures. This massive proliferation of individual hash points required to maintain required functionality was the root cause of the excessive memory consumption—reaching up to six gigabytes in certain instances—discovered by the engineering team.
Implementing Storage Improvements at the Structural Level
The breakthrough solution to this memory bloat came from targeted structural optimizations. An initial insight focused on the internal Rust data structures utilized by the Pingora Backend Router to store hash points. Previously, a standard point structure comprised a 32-bit integer for the hash value and another 32-bit integer acting as an index pointing to the corresponding server stored in a separate array. This layout consumed eight bytes of memory per point, with four bytes allocated to the hash and four bytes allocated to the server index.
The structural inefficiency became clear when considering that PBR would never need to coordinate more than roughly 65,000 servers simultaneously, meaning a 32-bit integer for the server index was entirely wasteful. Replacing the index type with a 16-bit integer could theoretically reduce the index size by half. However, simply changing the data type in Rust does not automatically reduce the memory footprint due to strict memory alignment rules. Rust mandates that the size of any data structure in memory must remain a multiple of the size of its most aligned field. Because the 32-bit hash field remained the largest component at four bytes, the structure retained a minimum size of eight bytes.
To bypass this hardware alignment constraint without triggering controversial compiler flags, the engineering team refactored the structure to store the hash and the index as a raw byte array accessed via explicit getter methods. This compact representation reduced the memory footprint dedicated to consistent hashing by an impressive 25 percent.
Rethinking Hash Density and Eliminating Redundancy
To achieve even greater memory reductions, the team returned to the underlying mathematics of hash distribution. While traditional assumptions suggested that piling on more hashes continuously improved load distribution, a rigorous mathematical derivation proved otherwise. Plotting the coefficient of variation against the number of hashes per server revealed a diminishing returns curve: each incremental drop in the error margin required an order of magnitude increase in the number of hashes.
In practical terms, utilizing a high weighting factor for storage nodes could result in tens or even hundreds of thousands of hashes per server. Mathematical analysis demonstrated that the vast majority of those extra hashes provided negligible reductions in error—often less than one percent improvement while consuming immense amounts of RAM. Furthermore, as the density of 32-bit hash values increases, the statistical probability of hash collisions rises rapidly due to principles akin to the birthday paradox. Collisions introduce unpredictable errors because they cause intended contributions to server volume to be silently dropped.
Armed with these mathematical insights, the engineering team realized they could slash the number of generated hashes per server by 90 percent without incurring any measurable degradation in routing accuracy or load distribution.
Executing a Safe Global Migration
Transitioning a massive global network to a radically smaller hash ring introduced a formidable operational risk. Because changing the hash ring alters the routing destination for cacheable requests, a sudden, unfiltered global cutover would instantly invalidate the vast majority of cached content across the network. Such an event would transform an internal memory optimization into a catastrophic surge in origin server traffic.
To prevent this, the engineering team designed a careful, multi-layered migration strategy. For an extended period, the Pingora Backend Router maintained both versions of the load-balancing rings simultaneously in memory: the legacy ketama ring and the new, streamlined ring. Utilizing Cloudflare’s existing migration routing framework, incoming requests evaluated a stable hash-based decision rule to select which ring would determine the backend server. This design ensured that rollout decisions remained consistent per request while providing an immediate rollback path if any anomalies surfaced.
The rollout proceeded methodically through isolated validation locations before expanding to progressively larger clusters of data centers, eventually scaling across the entire global network. By independently controlling traffic volume and geographic scope, the team successfully avoided the cache churn that would have accompanied a traditional global percentage rollout.
Continuous monitoring tracked backend selection traces, ring version metrics, connection errors, process memory consumption, startup times, cache hit rates, and origin server traffic. Once the migration successfully reached full adoption, the legacy hash ring paths were permanently decommissioned.
Monitoring dashboards confirmed the anticipated results immediately following the removal of the redundant hash structures. The optimization successfully dropped used memory globally by 100 terabytes, achieving a massive resource reclamation without disrupting edge operations or impacting origin servers.
The refined ring architecture, featuring compacted storage formats, accelerated sorting methods, and scalable node hash density, is available within the open-source ecosystem through the pingora-ketama crate. By pairing rigorous mathematical analysis with systems-level engineering, the project demonstrates how deep investigations into foundational algorithms can unlock substantial operational efficiencies at internet scale.
Leave a Reply