Cloudflare has officially announced that Hyperdrive support for MySQL is now generally available (GA), marking a major milestone for developers building high-performance, globally distributed applications on serverless infrastructure. Hyperdrive, Cloudflare’s database connection pooler designed specifically for Workers, initially launched with support exclusively for PostgreSQL. Following the debut of the MySQL protocol support beta last year, engineering teams have spent months refining the infrastructure to match the reliability and robustness of the company’s existing PostgreSQL offerings.
According to Cloudflare, the journey from beta to general availability ultimately boiled down to resolving 20 nuanced bug fixes under the hood. During the beta testing phase, the primary objective was to validate that real-world customer usage consistently met production availability targets of 99.99%. While tracking how often client connections from a Worker terminated in Hyperdrive-related errors, developers identified subtle gaps in reliability. Coupled with direct feedback from beta participants, these metrics led engineers to uncover a rare and complex bug hidden deep within the production environment. Understanding the nature of this challenge requires examining how Hyperdrive operates at scale across a global network.
What is Hyperdrive?
Hyperdrive functions as a distributed database connection pooler that bridges Cloudflare Workers with traditional SQL databases. By holding warm SQL connections open across its vast global network, Hyperdrive allows numerous Worker requests to share these connections efficiently. Beyond mere connection pooling, Hyperdrive incorporates the ability to cache common queries within the global Content Delivery Network (CDN) cache. This dual functionality significantly reduces latency for end users while simultaneously alleviating heavy load on origin databases.
Establishing a fresh connection to an SQL database is an inherently expensive operation, typically requiring a minimum of five network round trips. On a serverless compute platform like Cloudflare Workers, initiating a brand-new database connection for every incoming request is entirely impractical. Not only would this add unacceptable latency penalties to each request, but maintaining an exorbitant number of open database connections would also incur unsustainable per-connection resource costs. While MySQL generally fares better than PostgreSQL in this regard because it does not allocate an entire operating system process per connection, typical serverless deployments can still exhaust system resources rapidly when subjected to high volumes of concurrent connections.
A Journey to Stability
Throughout the beta testing period, engineers tackled a variety of issues ranging from straightforward adjustments to intricate protocol discrepancies. One of the earliest problems reported by developers was that emojis written to the database were not being returned correctly. Investigation revealed this was an issue rooted in character set encoding. Hyperdrive negotiates connection encoding during the initial startup handshake, during which it historically negotiated utf8. As developers discovered, however, utf8 in MySQL is actually a deprecated alias for utf8mb3, which fails to encompass the entirety of the Unicode standard. Specifically, utf8mb3 supports only one to three bytes per Unicode code point, whereas full UTF-8 requires up to four bytes. By updating the handshake process to explicitly negotiate utf8mb4, the engineering team successfully resolved the emoji encoding bug.
A secondary encoding-related complication arose later when the team integrated connection cleanup via COM_RESET_CONNECTION. This lightweight MySQL command clears any persistent session state within a given connection, making it an ideal mechanism for Hyperdrive to ensure that session-level data never leaks between checkouts. However, introducing this cleaning step inadvertently triggered further encoding issues. Engineers learned that COM_RESET_CONNECTION does not preserve the character set encoding negotiated during the initial startup handshake; instead, it resets the connection back to the server’s default configuration. Consequently, if Hyperdrive negotiated utf8mb4 on a MySQL server instance that defaulted to latin1, running the reset command would silently downgrade the connection encoding to latin1. The solution required enforcing an explicit command to reset the encoding back to utf8mb4 immediately following every connection reset.
A Hard-to-Find Bug
As the beta progressed, Cloudflare’s highest-volume MySQL customer encountered an elusive yet persistent issue: a small fraction of queries resulted in empty responses even though non-empty data was always expected. Engineers initiated a rigorous investigation by combing through extensive telemetry data and analyzing user logic. They elevated the trace rate for the affected Hyperdrive instance in hopes of capturing the precise execution flow of the error, and thoroughly audited network infrastructure for potential anomalies. Because the error manifested in approximately 0.001% of requests, isolating relevant end-to-end execution traces proved exceptionally difficult.
The investigation finally gained traction when engineers noticed a vital pattern: the bug occurred exclusively on Hyperdrive instances that had caching enabled. This common denominator pointed suspicion directly toward Hyperdrive’s MySQL caching mechanisms.
Hyperdrive Caching
To understand the root cause of the caching anomaly, it is necessary to examine Hyperdrive’s split architecture and its interaction with the CDN cache. Every Hyperdrive instance operates using two distinct nodes: a Client node and an Endpoint node. The Client node executes on the same machine as the user’s Worker script, manages local caching when enabled, and forwards queries to the Endpoint node when necessary. Meanwhile, the Endpoint node maintains the persistent connection pool and is situated in the data center located geographically closest to the origin database.
Each node leverages the Cloudflare CDN cache native to the data center in which it is running. Consequently, when both the Client and Endpoint nodes operate within the same data center, they share a common cache. This shared topology is generally beneficial for overall performance. Hyperdrive creates a normalized hash of the query’s wire messages and stores the resulting bytes directly in the cache. If a Client running in the same data center as its Endpoint receives an incoming request, and the Endpoint has recently cached the results of an identical query, the Client can serve the response instantly from the shared data center cache. For this architecture to function reliably, however, the system must guarantee that identical bytes are cached for the same response, regardless of whether the data was originally stored by the Client or the Endpoint.
MySQL Client/Server Protocol
Faced with the challenge of debugging the elusive caching bug without direct execution traces, the engineering team turned their attention to the underlying MySQL client-server protocol. They hypothesized that their codebase might be relying on assumptions derived from PostgreSQL’s protocol that did not accurately translate to MySQL handling.
The MySQL protocol governs how clients communicate with servers, a dynamic that becomes complex within database connection pooling architectures where multiple components act as both clients and servers. In this ecosystem, the user’s Worker acts as a client interacting with a MySQL client library, connecting to the Hyperdrive Client, which acts as a server. The Hyperdrive Client then communicates with the Hyperdrive Endpoint, which acts as a client connecting to the actual MySQL origin server.
The protocol begins with a Connection Phase, during which the client and server negotiate a mutually agreeable authentication method and feature capabilities. Once established, they transition to the Command Phase, where the client transmits SQL statements for execution while the server responds and awaits subsequent commands. Capabilities are exchanged during the initial phase to ensure protocol evolution while preserving backward compatibility. If clients and servers utilize unsupported features, a protocol desynchronization occurs, resulting in unpredictable behavior.
Capability negotiation presents a unique challenge for Hyperdrive because it acts as a server to the Worker while simultaneously acting as a client to the origin database. To correctly parse messages from the origin server, Hyperdrive employs a fixed set of capability flags. Because these capabilities may differ from those expected by the Worker’s client library, Hyperdrive must translate all origin responses to match the capabilities anticipated by the Worker.
During the debugging process, engineers discovered that mysql2, a popular MySQL client library for Node.js, does not support the CLIENT_DEPRECATE_EOF capability. This specific capability alters how result sets are terminated. Without it, the server transmits an explicit end-of-file (EOF) packet between the column definitions and the data rows, followed by another EOF packet at the very end of the result set. When CLIENT_DEPRECATE_EOF is negotiated, the metadata EOF packet is omitted entirely, and the final EOF packet is replaced with a standard OK packet. While MySQL server versions 5.7.5 and later support this newer behavior, it is only utilized when explicitly negotiated by the client.
Although Hyperdrive supports CLIENT_DEPRECATE_EOF, the Worker’s MySQL driver often does not. To bridge this gap, Hyperdrive dynamically adjusts its behavior to support the capabilities expected by the Worker client, rewriting result sets to include EOF packets and translating OK packets into EOFs whenever required.
After examining telemetry, documentation, and source code, engineers finally identified the missing piece of the puzzle: a faulty assumption baked into the Hyperdrive Client caching logic. When implementing client-side caching for PostgreSQL, engineers had assumed they could safely cache the exact bytes written back to the Worker. While this assumption held true within the context of PostgreSQL’s wire protocol, it failed completely under MySQL due to capability mismatches. Because the Endpoint node always operates using Hyperdrive’s native capabilities while the Client node writes back bytes translated to match the Worker’s specific capabilities, the two nodes were caching fundamentally different data formats for the same underlying queries.
This disparity explained the unexpected empty query responses and connection hangs. The failures occurred specifically when the Client and Endpoint nodes operated within the same data center, shared a local cache, and one node attempted to read a value previously cached by the other.
Once the failure mode was identified, implementing a permanent solution was straightforward. Hyperdrive now ensures that a canonical version of the query result bytes—encoded strictly using Hyperdrive’s native capabilities—is always written to the cache, regardless of which node stored the data. Following the deployment of this fix, the empty row incidents ceased entirely, and the success rate for client connections successfully reached the target of 99.99%.
Developers can begin utilizing Hyperdrive support for MySQL immediately. Instances can be provisioned for existing MySQL-compatible databases directly through the Cloudflare dashboard or Wrangler command-line tool, or integrated seamlessly with new PlanetScale databases.
Leave a Reply