Engineering teams deploying large language models at scale frequently encounter a quiet financial trap: as underlying AI models become more capable, sophisticated, and intelligent, the cost of utilizing them tends to skyrocket. Cloudflare recently experienced this firsthand with its internal AI code reviewer. What began in April as a palatable, cost-effective operation averaging $1.19 per review climbed steeply, reaching an average of $2.46 by the end of August. Yet, through an aggressive, data-driven optimization campaign launched in mid-September, the company managed to reverse this trend entirely.
Today, a high-effort code quality review at Cloudflare costs just $0.77—a dramatic 68% reduction from the peak average of $2.43 recorded during the company’s two most expensive weeks prior to the overhaul. Beyond merely slashing expenses, the engineering team’s optimization efforts inadvertently accelerated the system. The median review time plummeted from 409 seconds down to just 92 seconds, vastly improving the developer experience. Roughly 40% of these total cost savings stemmed from direct code modifications and architectural optimizations within the reviewer, while the remaining majority was unlocked through strategic experimentation with different underlying language models and execution effort tiers.
How a Routine Review Doubled in Cost Over Months
Tracking the trajectory of these expenses reveals a steady upward creep over the spring and summer months. Back in March, the average review hovered at $1.22. By June, that figure edged up to $1.44, rose to $1.74 in July, and peaked at $2.32 in August. A minor driver of this increase was a shift in the distribution of workloads: more reviews landed in the resource-intensive "full review" tier, rising from 59% in March to 65% in August. Cloudflare attributes this to the rapid advancement of foundational AI models, which has naturally empowered engineers to produce larger, more complex code changes that demand deeper scrutiny.
However, the primary culprit behind the soaring bills was model migration. At the beginning of August, Cloudflare transitioned all its reviewers onto Anthropic models because thorough evaluations proved they outperformed competing options on this specific workload. The internal standards reviewer, Codex, was migrated to Claude Opus 4.7 due to its superior reliability and performance. Consequently, Codex’s operational cost surged from about $0.25 per review to $0.88 in August, temporarily making it more expensive than the coordinator agent orchestrating the workflow.
Simultaneously, token consumption surged. While the system’s prompt cache hit rate improved from 85.7% in April to 97% in September, the actual number of tokens processed per review more than tripled, climbing from 0.91 million to 2.99 million. Even with highly discounted cache read rates, tripling the raw volume of tokens naturally inflated overall expenditure.
Uncovering Where the Tokens Were Disappearing
To reverse these costs, Cloudflare tasked its autonomous agents in mid-September with tracking down every single token and trimming consumption aggressively. Prior to this intervention, the AI review budget was divided among output tokens (accounting for 23%), cache writes (43%), and cache reads (34%). The vast majority of the budget was being consumed by context the language model had either already reviewed or was about to process repeatedly.
A closer look at individual review sessions exposed the underlying mechanics. On average, the code quality reviewer executed 16.6 large language model calls per review. Across those iterations, it wrote 72,000 tokens to the cache and read back 1.08 million tokens, because each successive API call resubmitted the entire conversation history as context. As both the call count and context size slowly expanded over the summer months, session costs escalated in tandem.
Understanding Anthropic’s prompt caching mechanism proved vital to solving the problem. The system caches computational prefixes located at the very start of a conversation, allowing subsequent requests to reuse pre-computed data up to the point where the prompt diverges. Because the harness tools, system prompts, and initial instructions for Cloudflare’s reviewers rarely change across routine operations, they were ideal candidates for prefix caching. Writing to the cache costs 1.25 times standard input pricing, while reading costs just 0.1 times, with entries expiring after five minutes unless accessed. The engineering challenge became twofold: ensuring the initial request of each session read from the cache rather than writing to it, and engineering workflows to utilize fewer total calls.
Sharing Cached Prefixes Across Every Repository
Beginning September 14, Cloudflare’s telemetry began recording the token usage of each reviewer’s first LLM call alongside a hash of the initial system block. The resulting data revealed a glaring inefficiency: the opening request of a session found a valid cached prefix only 14% to 26% of the time. Because every review session initializes with identical tool definitions and agent prompts—totalling between 11,000 and 21,000 identical tokens—virtually every run should have been a cache hit.
The root cause lay in how the legacy OpenCode v1 framework constructed system prompts. It bundled the agent prompt together with an environment section containing the working directory and current date, placing a single cache breakpoint at the very conclusion of that combined block. Because the shared core instructions were never isolated, a review conducted in one repository could never leverage the cached data generated by a review in a different repository.
On September 16, engineers introduced a distinct marker at the end of every agent prompt and implemented a plugin hook that split the system block into two separate parts at that precise boundary. The first half now terminated on generic bytes shared universally across all jobs, earning its own dedicated cache breakpoint. To preserve this cache integrity, any repository-specific variables—such as dynamic lists of available skills or fluctuating tool descriptions—were stripped out from the preceding section.
On that same day, Cloudflare integrated the Codex standards reviewer’s reference index directly into its prompt. Previously, every session spent its first three turns executing tool calls to read internal standards and reference files. Embedding these materials directly into the prompt reduced Codex’s average LLM calls per run from 15.9 to 9.6, while dropping its cost per run from $1.12 to $0.72. Combined with the prompt restructuring, these changes immediately lowered the average cost per review from $2.47 to $2.02.
Refining Operations to Trim Additional Calls and Dollars
Subsequent optimizations rolled out on September 21 and 23 focused on eliminating redundant computational steps. For instance, sub-reviewers previously had visibility into the tool responsible for posting final review comments. In over half of all sessions, these sub-reviewers attempted to invoke the tool directly, received a server-side rejection, and spent an additional API call recovering. Removing the unauthorized tool definitions from sub-reviewer prompts eliminated this wasted interaction and pruned roughly 2,900 tokens of unnecessary definitions from every request.
Engineers also optimized file-reading behavior. Recognizing that the code quality reviewer made single tool calls to read individual files on the vast majority of turns, the system was adjusted to read the shared context and all associated patches—up to 150 KB—within its very first response. Furthermore, the system began intelligently skipping redundant checks on re-reviews. Because approximately two-thirds of all jobs represent subsequent reviews of existing merge requests where several secondary reviewers have no new changes to analyze, routine reviews for documentation, performance, and formatting are now skipped up to 56% of the time.
Additional savings were captured by terminating sessions immediately after reviews were posted, preventing unnecessary summary-generation turns, and updating timeout and cancellation logic to prevent abandoned jobs from continuing to consume compute resources silently.
Evaluating Model Shifts and Effort Mixes
While code modifications yielded meaningful savings, experimentation with underlying models and computational effort tiers generated the most substantial financial impact. Transitioning the coordinator and reviewer configurations to newer model iterations, such as moving from Sonnet 5 to Sonnet 5.5, yielded dramatic efficiency gains. Although Sonnet 5 and 5.5 share identical per-token pricing, Sonnet 5.5 required significantly fewer tokens to complete the exact same workloads. Within days of deployment, Sonnet 5.5 proved 62% cheaper per run for code quality, 58% for performance, and 47% for security.
By balancing medium and high-effort configurations, Cloudflare ensured that cost reductions did not come at the expense of engineering rigor. Rigorous blind testing evaluated by external models confirmed that the drop in flagged findings primarily eliminated low-value noise, minor style nitpicks, and inflated severities. Crucially, the rate of important, actionable findings remained steady, while the percentage of serious findings successfully acted upon by human authors rose from 34% to 59%.
Ultimately, Cloudflare’s optimization journey reduced the average review cost to $0.77 while cutting median latency down to 92 seconds. By rigorously auditing token pathways, perfecting cross-repository prompt caching, and matching tasks to optimal model effort tiers, the engineering team transformed an escalating financial liability into a highly efficient, scalable internal service.
