Skip to content
INTERNET INFRASTRUCTURE & NETWORKS

Cloudflare Fixes Container Storage Vulnerability Reported by Security Researchers

Cloudflare has successfully remediated a multi-tenant storage isolation vulnerability affecting its Cloudflare Containers and Cloudflare Sandboxes services. The flaw, which could have potentially allowed a paid account holder to recover residual disk blocks left behind by other customers on the same shared host, was discovered and responsibly disclosed by Oren Yomtov, a security researcher from Accomplish, via Cloudflare’s bug bounty program on September 4, 2026.

According to Cloudflare, a comprehensive review of historical disk input-output telemetry has revealed no evidence that customer data was compromised or that the vulnerability was exploited maliciously in the wild. The only activity identified that matched the researchers’ technique came from the reporting team and Cloudflare engineers conducting authorized validation tests.

The discovery highlights the intricate security challenges inherent in modern multi-tenant cloud architectures, where disparate customer workloads are dynamically packed onto shared underlying hardware infrastructure. Cloudflare Containers run workloads across a managed multi-tenant infrastructure, automatically provisioning them to eligible servers without allowing customers to select their specific underlying host hardware.

The successful remediation was developed and deployed in close collaboration with Yomtov and the Accomplish security research team, whose detailed technical report and controlled testing methodology enabled Cloudflare to validate the issue swiftly and apply a fleet-wide fix without requiring any configuration changes from customers.

How Container Storage Allocation Works

To understand how the vulnerability functioned, it is necessary to examine the underlying storage mechanics utilized by Cloudflare’s container architecture. Cloudflare Containers rely on the Linux kernel’s device mapper thin provisioning technology, known as dm-thin, to supply each individual container with its own dedicated writable root disk.

Within this architecture, each container operates inside an isolated virtual machine powered by the Firecracker virtual machine monitor. Firecracker presents the thin-provisioned root disk to the guest virtual machine as a block device designated as /dev/vdc.

Thin provisioning is a storage virtualization technique designed to optimize hardware utilization by allocating physical storage space dynamically only when a virtual disk actually writes data to a previously unmapped region. Within the affected storage pools, Cloudflare utilized a standard thin-block size of 64 kilobytes.

When a thin volume backing a container’s root disk was deleted after the workload finished executing, its physical blocks were returned to a shared storage pool that serviced workloads belonging to multiple distinct customer accounts. However, the configuration of the affected storage pools included a specific option designated as skip_block_zeroing.

Under normal circumstances, storage virtualization layers securely zero out newly allocated blocks before exposing them to a tenant to prevent data remnants from bleeding across boundaries. With the skip_block_zeroing option enabled, the dm-thin subsystem bypassed this crucial zeroing process for newly allocated blocks. Consequently, when a previously utilized 64-kilobytethin block was reassigned to a new container, a full-block write would completely overwrite its contents, but a smaller, partial-block write would only alter the specific portion being written. The remainder of the 64-kilobyte block could potentially retain residual data left behind by the block’s previous owner.

How the Exploit Technique Worked

The vulnerability could not be exploited simply by reading unmapped regions of a newly provisioned thin disk. When a container attempted to read an unmapped region, the dm-thin subsystem correctly returned streams of zeroes without actually allocating a physical block from the shared storage pool, keeping tenant data safely concealed.

To bypass this limitation, the researchers developed a proof-of-concept technique that identified 64-kilobyte-aligned regions corresponding to free space within the guest operating system’s ext4 filesystem. The proof of concept then wrote a single aligned 4-kilobyte block into each of those targeted regions.

When such a targeted write reached a previously unmapped thin block, the dm-thin manager was forced to allocate a physical 64-kilobyte block from the shared storage pool to satisfy the write request. Because the skip_block_zeroing configuration option was active, the 4-kilobyte write only replaced that exact fraction of the storage block, leaving the remaining 60 kilobytes intact with whatever data happened to reside there from a previously hosted container.

A subsequent raw-device read performed by the container could therefore observe bytes that the new container had never actually written to disk. The proof-of-concept script utilized by the researchers gathered aggregate counts, block offsets, sizes, checksum validation results, and truncated hash prefixes to document the behavior.

Despite recovering raw blocks to validate the structural integrity of the flaw, the telemetry and materials provided to Cloudflare contained no third-party filenames, user identifiers, cryptographic credentials, hostnames, IP addresses, or readable recovered content values. Furthermore, the researchers confirmed that all recovered data under their operational control was kept strictly confidential and securely destroyed following their submission to the bug bounty program.

Validation Across Production Placements

To rigorously distinguish storage blocks belonging to their own test filesystem from unrelated blocks originating from other tenants, the researchers leveraged ext4 directory block checksum features. Specifically, when an ext4 filesystem utilizes the metadata_csum feature, directory block checksums incorporate internal values directly associated with the filesystem instance and its specific inodes.

By testing their methodology against blocks they had deliberately created and subsequently deleted within their own controlled test filesystem, the researchers confirmed that their attribution logic correctly identified all test blocks. When scaling the validation across live infrastructure, the researchers observed residual storage material across multiple production placements, underlying server nodes, and geographic regions.

The types of residual material observed in these blocks included underlying directory structures, internal database pages, and structurally complete SQLite database fragments. The researchers emphasized that their testing scripts were engineered to output strictly aggregate counts and format verification checks rather than exposing actual recovered file contents.

Cloudflare confirmed that the materials submitted by the research team contained no recovered content values or third-party identifiers. Adhering strictly to Cloudflare’s HackerOne disclosure policy, the researchers subsequently verified that all recovered data had been permanently and securely deleted.

Potential Impact and Limitations

The security implications of the vulnerability centered on the potential crossing of multi-tenant isolation boundaries within Cloudflare’s shared infrastructure. Had an unauthorized actor leveraged the technique maliciously, a customer operating a Workers Paid account could theoretically have recovered residual data residing on storage blocks previously utilized by other customers’ containers on the same underlying physical host.

Successful exploitation could have theoretically exposed sensitive information, including filesystem metadata, directory hierarchies, database pages, and application data. However, the practical utility and targeting capabilities of the attack were severely constrained.

An attacker could not deliberately target a specific victim customer, a specific workload, a particular host machine, or specific data files. Data exposure relied entirely on probabilistic workload placement by Cloudflare’s orchestration scheduler and the specific release and reassignment patterns of dm-thin blocks.

Crucially, the researchers did not demonstrate any capability to modify another customer’s active data, nor did they cause any operational disruption or availability impact to workloads running on the platform.

Cloudflare Mitigation and Infrastructure Remediation

Upon receiving the responsible disclosure, Cloudflare immediately initiated a two-phase mitigation strategy to secure its container and sandbox infrastructure fleet-wide.

The first step involved removing the skip_block_zeroing parameter from the dm-thin storage pool configurations across the entire production fleet. This action successfully restored the default behavior of the device mapper subsystem, ensuring that all newly allocated storage blocks are rigorously zeroed out before being exposed to any container environment. This immediate configuration change effectively neutralized the reported exploit vector, a fact independently verified by the research team using their proof-of-concept scripts.

However, Cloudflare engineers recognized that zeroing newly allocated blocks did not automatically sanitize blocks that were already actively mapped into existing thin devices. These pre-existing mappings persisted within running container disks and within each host’s internal cache of prepared dm-thin snapshots utilized for Open Container Initiative image layers.

To ensure complete remediation, Cloudflare proceeded to retire all running container disks and purged all cached image snapshots that had been generated prior to the mitigation. The engineering team systematically drained host nodes during scheduled off-peak windows, restarted the underlying virtual machines on each affected host, and flushed the local image caches. This ensured that all active disks and cached image layers were entirely recreated using properly zeroed allocations. Cloudflare confirmed that this exhaustive cleanup operation has been fully completed across the entire Containers fleet.

Investigation Confirms No Malicious Exploitation

As an integral part of its incident response protocol, Cloudflare initiated a deep retrospective investigation to determine whether any external workloads had exhibited behavior consistent with the reported exploitation technique prior to the patch.

Security analysts reviewed historical disk input-output telemetry retained from the container infrastructure, utilizing the researchers’ proof-of-concept mechanics and internal reproduction data as reference signatures. Because the proof-of-concept technique produced a distinct relationship between small write operations and subsequent raw reads of unmapped regions, engineers were able to construct precise detection signatures.

These detection signatures were applied across all available historical telemetry logs. The analysis successfully identified activity that directly corresponded to the authorized validation testing conducted by the Accomplish researchers and Cloudflare engineers, but uncovered no additional activity matching the exploit pattern from any other source. Cloudflare found no evidence indicating that the vulnerability had ever been discovered or exploited by malicious threat actors.

Ongoing Customer Protection and Commitment to Transparency

With the fleet-wide patches and infrastructure cleanup fully complete, Cloudflare confirmed that its customers remain fully protected, and no further actions or configurations are required by users of Cloudflare Containers or Sandboxes.

Cloudflare leadership expressed gratitude to Oren Yomtov and the Accomplish security research team for their technical rigor, responsible disclosure practices, and collaborative approach throughout the remediation process. The company reiterated its encouragement for the broader security community to report potential vulnerabilities through its bug bounty program to help maintain the integrity and safety of its global platform.

Emphasizing that user trust remains foundational to the operation of customer infrastructure, Cloudflare affirmed its ongoing commitment to prioritizing platform security and maintaining operational transparency whenever technical challenges arise.

Leave a Reply

Your email address will not be published. Required fields are marked *