Cloudflare has officially introduced Meerkat, an experimental distributed consensus service engineered to manage the company’s internal global control plane. Breaking away from industry conventions, Meerkat is powered by QuePaxa, a novel consensus algorithm published in 2023 by researchers Tennage, Băsescu, and their colleagues. While mainstream distributed systems typically rely on well-established algorithms like Raft, Multi-Paxos, or Viewstamped Replication, Cloudflare’s unique global deployment footprint required an architecture capable of handling the volatility and variable latency of the wide-area public Internet.
Most traditional consensus algorithms are deeply understood, feature mature implementations, and perform reliably within local data centers or tightly controlled networks. However, these systems often struggle when deployed across wide-area networks (WANs) characterized by volatile network connectivity and unpredictable latency. Common consensus protocols fundamentally rely on a single, highly available leader process operating with predictable timing. If this leader experiences higher-than-normal latency, goes offline, or suffers a network disconnection, the entire consensus system inevitably stalls or becomes unavailable until a replacement is elected. For Cloudflare, operating an edge network spanning continents, such pauses are unacceptable.
What Makes Meerkat Unique at Global Scale
At its core, Meerkat is a distributed system wherein a group of machine replicas must agree upon an ordered log of events, with each event determined by a consensus algorithm. Consensus mechanisms allow a group of independent machines to agree on a single shared truth, ensuring they never decide on conflicting values, even if individual machines crash or disconnect. Meerkat leverages this consensus-powered log to expose high-level applications, such as a distributed key-value store, where entries represent read and write operations. Because the underlying algorithm guarantees that replica logs never diverge, every node constructs an identical key-value state, placing Meerkat in the same functional category as systems like Consul, etcd, and Apache ZooKeeper.
Unlike its peers, however, Meerkat was designed specifically for global scale. Clusters may span multiple continents, bound by network links that frequently exhibit high and fluctuating latency. Traditional consensus systems are rarely optimized for this environment, typically requiring strict, low-latency network architectures to maintain optimal performance.
The Challenges of Consensus at a Global Scale
When network conditions degrade without physical machine crashes, many traditional consensus algorithms suffer from disproportionate performance drops or complete unavailability. Cloudflare has experienced this vulnerability firsthand. In 2020, a partial failure of a core network switch caused a consensus-powered internal system to severely degrade, slowing the operator dashboard down by 80 times for a duration of six hours. While the system preserved all data without corruption, it became entirely unavailable for write operations until engineers resolved the underlying network issue—an outcome that highlights the brittleness of traditional architectures under real-world stress.
Ideally, a resilient consensus algorithm should continue making progress despite sudden latency spikes and severed connections, rather than stumbling when healthy machines are simply isolated by the network. This challenge is magnified at a global scale, where faults frequently originate in the underlying transport network rather than local server hardware. Two primary traits of traditional consensus algorithms contribute to this vulnerability: destructive interference and error-prone fault detection.
The classic Paxos algorithm, for instance, relies on roles known as proposers and acceptors. To reach agreement, a proposer broadcasts a ballot number to a majority of acceptors during a Prepare phase, followed by the actual proposal during an Accept phase. Complications arise when multiple proposers submit concurrent proposals, interleaving higher ballot numbers between phases and mutually blocking one another from achieving majority support. This phenomenon, known as destructive interference, intensifies as the number of concurrent proposers grows. Formally, Paxos lacks liveness guarantees without assuming that all messages arrive within a bounded timeframe—an assumption known as partial synchrony.
To mitigate this competition, most deployed algorithms—including Raft, Multi-Paxos, and Viewstamped Replication—introduce a designated leader proposer. By centralizing proposing power into a single replica, these protocols eliminate competing proposals and optimize decision-making down to a single network round-trip under normal conditions. The trade-off, however, is that the leader must remain reachable. When a leader fails, slows down, or gets disconnected, the system stalls until remaining replicas detect the failure and elect a successor.
Unstable Leader Recovery and Fault Detection
Ironically, the leader recovery process itself often reintroduces destructive interference into otherwise stable systems. In Raft, for example, replicas that suspect a leader failure announce themselves as candidates and solicit majority votes. When multiple replicas detect a failure simultaneously, they often compete for votes, resulting in mutual blocks and failed elections. Even after a temporary network outage subsides, elections can persist as candidates repeatedly block each other, creating a metastable failure pattern where downtime outlasts the initial network glitch.

Compounding this issue is the inherent difficulty of fault detection in asynchronous networks. Because a machine cannot directly observe the state of a remote server, it must rely on expected message arrival times, such as heartbeats. In volatile wide-area networks, however, message delays and transient disconnections are indistinguishable from server crashes. When a fault detector erroneously concludes that a healthy leader has failed, it triggers an unnecessary election, abruptly halting the entire system until a new leader is established. Furthermore, to prevent trigger-happy elections, fault detectors must enforce conservative timeouts—often set to five to ten times the baseline network round-trip time—meaning that even accurate fault detection is inherently slow over high-latency global links.
Why QuePaxa Fits Cloudflare’s Architecture
Recognizing that strict leaders and accurate timeouts inevitably cause stalls over the public Internet, Cloudflare sought an algorithm designed specifically for asynchronous environments. Published in 2023, QuePaxa emerged as the ideal candidate.
QuePaxa achieves high performance during favorable network conditions while gracefully degrading during latency spikes and server failures through three key architectural choices: making the leader entirely optional, removing reliance on traditional fault detection, and designing for constructive interference among proposers.
Similar to Paxos, QuePaxa utilizes proposers and recorders. However, instead of allowing proposers to lock each other out indefinitely, QuePaxa assigns randomized priorities to proposals. The optional leader is assigned an exclusive maximum priority. During execution, proposers submit proposals to recorders, which propagate them across the cluster. When a proposer receives a proposal with a higher priority than its own, it adopts the higher-priority proposal and broadcasts it instead. Consequently, proposers actively cooperate to converge on the highest-priority proposal they collectively recognize, preventing mutual deadlocks.
While QuePaxa is a randomized algorithm requiring multiple rounds in contentious scenarios, it ensures eventual agreement. The system maintains an optional leader to provide a fast path, allowing the leader to decide in a single round trip when its proposal reaches a majority first. Clients do not need to route requests exclusively through the leader; any proposer can initiate a proposal. If the leader goes down, clients seamlessly continue making requests to other proposers without waiting for a formal recovery phase or triggering destructive elections.
To exploit the leader’s fast path without creating a single point of failure, clients hedge their bets by broadcasting requests to all replicas, while non-leader replicas observe a short local timeout to give the leader priority. Unlike the conservative timeouts in Raft, QuePaxa’s hedging timeout is not a fault-detection mechanism; shortening it simply increases active proposer participation rather than freezing the system.
How QuePaxa Operates Under the Hood
QuePaxa executes in rounds composed of four distinct phases. In Phase 0, proposers broadcast their adopted proposals, randomizing priorities to satisfy theoretical constraints while giving the leader a fast-path decision if it achieves majority support. Because of majority intersection—the geometric certainty that any two majorities of recorders share at least one node—any subsequent proposer querying a majority will inevitably encounter the leader’s proposal and adopt it.
If concurrent proposers block the fast path, the protocol proceeds to Phase 1, where proposers spread their adopted proposals to a majority of recorders, ensuring that lower-priority proposals cannot be erroneously decided later. Phase 2 allows proposers to check if a higher-priority proposal has gained sufficient support, enabling safe decisions. Finally, Phase 3 acts as a cleanup phase, ensuring that any residual information is carried forward into the next round so that work is never discarded.
Production Results and Next Steps
Evaluating Meerkat across global deployments spanning nearly all continents, Cloudflare observed that the system maintains continuous request processing even during deliberate leader crashes, disconnections, or performance degradations. Approximately 95% of decisions are resolved by the leader in the first phase, maintaining stable throughput. During rare instances where decisions require multiple phases, throughput drops momentarily to a fraction of its normal rate, but the system continues operating without halting. Furthermore, operators can dynamically tune and switch leadership roles without risking cluster stability.
Because QuePaxa is less battle-tested in production than Raft or Paxos, and because Cloudflare is extending its original crash-stop design to support full crash recovery, the engineering team is subjecting the implementation to rigorous formal verification, simulation, and fault-injection testing. Future technical updates from Cloudflare will detail the formal verification of the Rust implementation, advanced failure testing, and benchmarks gathered while transforming an academic protocol into a global production system.