Skip to content
INTERNET INFRASTRUCTURE & NETWORKS

Cloudflare Discloses and Resolves Tenant Isolation Vulnerability in Workers AI GLM-5.3 Model

Cloudflare has released a comprehensive security report detailing a tenant isolation vulnerability that affected the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3). The issue, which stemmed from a configuration mismatch and a bug in the open-source inference engine SGLang, allowed a small number of users to receive responses containing data from previous requests made by other customers. Cloudflare has confirmed that the vulnerability has been fully remediated, the model has been restored to service, and no other models hosted on Workers AI were impacted.

The security flaw was initially brought to Cloudflare’s attention on September 8, 2026, through its HackerOne bug bounty program by security researcher Gadi Evron from Knostic. According to Cloudflare’s investigation, the issue had been active since the model’s launch on August 27, 2026. Between August 27 and September 9, a total of 155 responses were affected. Notably, every affected response was triggered by prompts exceeding 681,280 tokens.

An analysis of the affected requests’ timing and volume aligned with ordinary long-context usage patterns rather than targeted exploitation. Cloudflare found no evidence that the bug was deliberately exploited by malicious actors before it was patched. The disclosure was coordinated alongside SGLang, the open-source inference engine utilized by Cloudflare, and researchers to outline the timeline, the underlying mechanics of the bug, and subsequent hardening measures deployed across production environments.

What Happened

The GLM text generation model, developed by Z.ai and optimized for agentic coding workflows, is among the various models hosted on Cloudflare’s Workers AI platform. Following the report submitted via HackerOne on September 8, Cloudflare successfully reproduced the issue the following morning. Within 40 minutes of declaring an internal security incident, the company took the model offline, deployed a corrected configuration alongside a patched inference engine, and safely restored public access on September 9.

The cross-request KV cache exposure was caused by the convergence of two distinct issues. First, there was a configuration mismatch involving SGLang. Cloudflare deployed GLM-5.3 with a configured context window of 1,048,576 tokens, but the Key-Value (KV) cache pool allocated on the prefill GPUs possessed space for only 681,280 tokens. Cloudflare engineering teams were aware of this size disparity prior to launch, anticipating that the inference engine would gracefully reject any prompt exceeding the available capacity and return a client-facing error. Because standard pre-launch long-context tests returned successful HTTP status codes, the discrepancy was not initially flagged as a critical risk.

The second factor was a flaw in how SGLang processed over-length requests when operating in a disaggregated prefill and decode architecture. When a prompt exceeded the limits of the KV cache pool, SGLang’s scheduler internally rejected the request, substituting it with a one-token placeholder to allow it to clear the standard batch processing pipeline without allocating physical memory. In a single-node setup, this mechanism functioned as intended, safely generating an error for the client.

However, in Cloudflare’s disaggregated deployment mode, admission queues on both the prefill and decode nodes monitored whether a request was finished rather than explicitly verifying whether it had been rejected. Consequently, the rejected request was treated as active. The prefill node generated a KV cache for the placeholder prompt and transferred it to the decode node via remote direct memory access. Assuming that the incoming transfer represented the full prompt and had overwritten previous memory states, the decode node generated a response while attending over stale cache pages left behind by earlier user sessions. This resulted in responses to extremely long prompts occasionally containing fragments of data from other users.

Immediate Remediation and System Restoring

Upon disabling the model, Cloudflare instituted immediate corrective measures before returning GLM-5.3 to public availability. Engineers adjusted the GLM-5.3 configuration to match the parameters used by GLM-5.2, expanding the GPU memory allocation so that the prefill KV pool can comfortably handle roughly 2.04 million tokens—more than double the maximum context length of 1,048,576 tokens. By ensuring that any valid prompt fitting the context window automatically fits the cache pool, the rejection path is bypassed for valid requests, preventing the bug from being triggered.

To eliminate reliance on pool sizing alone, Cloudflare also patched the inference engine to ensure the rejection path is inherently safe. Because factors such as quantization, tensor parallelism, and CUDA graph adjustments can unpredictably alter pool sizes, relying strictly on configuration dimensions remains fragile. Cloudflare engineers patched the engine so that requests flagged for pending rejection are retired immediately at the disaggregation admission queue before any bootstrap, allocation, or KV transfer occurs.

Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

This approach was contributed upstream to the SGLang project as pull request #38704, and the maintainers subsequently integrated similar logic in pull request #38935. Cloudflare validated both configuration updates and engine patches using the researcher’s reproduction script and internal canary markers before restoring public access to the model.

Background on Workers AI Architecture and SGLang Integration

Workers AI executes GLM-5.3 alongside SGLang utilizing a disaggregated configuration architecture. In this setup, a dedicated pool of prefill GPUs processes the incoming prompt to generate the model’s KV cache. That cache is subsequently transferred over remote direct memory access, powered by Mooncake, to a separate pool of decode GPUs responsible for generating the final text response.

Within GPU memory, the KV cache consists of a shared pool of fixed-size pages assigned to requests upon admission and released upon completion. To maximize computational performance, these memory pages are not automatically zeroed out between requests; instead, incoming requests simply overwrite existing pages upon allocation. Consequently, the inference engine bears the responsibility of maintaining accurate ownership records for each memory page across both local nodes and remote prefill nodes that retain simultaneous references.

Furthermore, SGLang calculates the total pool size at startup based on remaining GPU memory after loading model weights, while the advertised context length operates as an independent configuration parameter. The lack of an automated enforcement mechanism to align these two parameters necessitated the rigorous deployment checks that Cloudflare has since implemented.

Ongoing Hardening and Industry Collaboration

The incident highlighted a broader class of KV cache memory-safety challenges related to page lifetimes and cross-node races. These include scenarios where a memory page is prematurely freed, reused, or read while active references still exist within single-node operations or across disaggregated nodes communicating via RDMA.

In the wake of the incident, Cloudflare accelerated collaborative efforts with the core engineering team at RadixArk, the maintainers of SGLang, to implement systemic protections across all production deployments. These improvements include enhanced request cancellation protocols in sgl-router, strict validation of transfer checksums on decode nodes before text generation begins, deferred KV release mechanisms, and refined tracking of abort notifications.

Cloudflare continues to advance long-term architectural defenses, including the development of integrity-by-construction prototypes that utilize physical KV page tagging to ensure an engine immediately fails any request attempting to read unowned memory pages. Additionally, the company is exploring the application of formal verification methods to mathematically prove the safety and consistency of distributed RDMA-based KV cache transfer protocols, aiming to eliminate entire categories of distributed state divergence bugs.

The coordinated disclosure highlights the value of proactive security research and rapid industry collaboration. Cloudflare has expressed gratitude to researcher Gadi Evron for providing a clear reproduction script and responsibly coordinating the disclosure, as well as to the SGLang engineering team for swiftly integrating upstream patches into recent engine releases.