Engineering teams deploying large language models at scale frequently encounter a quiet financial trap: as underlying models grow more sophisticated, the operational costs of automated tasks can balloon rapidly without any major architectural changes. Cloudflare recently confronted this exact challenge with its internal AI code reviewer. Following a dramatic escalation in expenses over the summer, the engineering team launched an aggressive cost-cutting and optimization initiative that slashed the average price of a code review by 68% while simultaneously cutting execution latency in half.
When Cloudflare first detailed its automated AI code reviewer in April, a typical review carried a palatable cost of $1.19. By the end of August, however, that figure had more than doubled to an average of $2.46 per review. This financial creep occurred even though the system’s architecture remained completely unchanged; the coordinator was still routing work to the exact same specialist reviewers, and established risk tiers continued to dictate the intensity of each evaluation.
Midway through September, the team tasked its autonomous agents with tracking down token usage and aggressively reducing consumption. Today, operating at high effort settings, an average review costs a mere $0.77—down 68% from the peak average of $2.43 recorded during the system’s two most expensive weeks. Alongside these financial savings, the optimization work drastically improved performance, reducing the median review time from four minutes and nine seconds down to just one minute and thirty-two seconds. Roughly 40% of these total savings stemmed from direct code modifications and optimizations to the reviewer itself, while the remaining majority was achieved by experimenting with different model selections and adjusting execution effort levels.
How a Review Doubled in Cost
An analysis of weekday review costs reveals a steady upward trajectory throughout the year. The average cost per review rose from $1.22 in March to $1.44 in June, climbed to $1.74 in July, and peaked at $2.32 in August. A minor portion of this increase was driven by a shift toward more intensive review tiers. In March, 59% of reviews landed in the most expensive tier, compared to 65% by August. Engineers naturally produce larger, more complex changes as underlying AI models become more capable, leading to a higher volume of code falling into the comprehensive review category.
However, the primary driver of the cost increase was model migration rather than workload shifts alone. At the beginning of August, Cloudflare transitioned all its reviewers entirely to Anthropic models, a move that elevated the cost per job from $1.38 to $2.03 because evaluations demonstrated that Anthropic models performed better on this specific workload than competing alternatives. Internal standards reviewer Codex ultimately standardized on Claude Opus 4.7 due to performance and reliability considerations, pushing its individual review cost from approximately $0.25 to $0.88 in August—making it more expensive than the coordinator agent itself.
Simultaneously, token volume expanded dramatically. The cache hit rate increased from 85.7% in April to 97% by September, but the total number of tokens per review more than tripled from 0.91 million to 2.99 million. Even with low cache read costs, maintaining a high hit rate on triple the volume naturally escalated total expenses.
What Else Has Changed Since Our Last Post?
Significant changes accompanied the cost-reduction initiative, most notably the transition of the reviewer platform to OpenCode v2, a ground-up rewrite of the original software. Before cutting over entirely, the team ran comparative shadow mode tests across every canary repository on both v1 and v2. Because both versions agreed on approval decisions in 98% of tested merge requests, the team migrated all traffic over in early October and deprecated the v1 codebase. Comparative tests on identical models showed that full-tier reviews on v2 finish approximately 13% faster. Engineers are also experimenting with alternative execution environments, including running the internal AI reviewer solely on Durable Objects using pi-durable.
Where Were the Tokens Going?
Prior to the optimization push, the AI review budget was split among output tokens at roughly 23%, cache writes at 43%, and cache reads at 34%. This distribution revealed that the majority of capital was being consumed by context that the model had already encountered or was scheduled to process again. A single code quality review session averaged 16.6 large language model calls, writing 72,000 tokens to the cache while reading back 1.08 million tokens because each subsequent call transmitted the entire conversation history as context.
Understanding Anthropic’s prompt caching mechanism proved vital to solving this problem. The architecture caches computed prompt prefixes located at the start of a conversation rather than midway through or at the end. Because the harness tools, system prompts, and opening messages of the reviewer sessions rarely changed, they were prime candidates for prefix caching. Because writing to the cache costs 1.25 times the normal input price while reading costs a mere tenth, and entries expire after five minutes unless accessed, the engineering targets became clear: ensure the first request of every session read from the cache rather than writing to it, and minimize the total number of calls per session.
Sharing a Cached Prefix Between Every Repository
On September 14, monitoring revealed that the first request of an agent session found a cached prefix only 14% to 26% of the time, despite every session starting with identical tool definitions and agent prompts ranging from 11k to 21k tokens. The root cause lay in how OpenCode v1 constructed system prompts by combining the agent prompt with an environment section containing working directories and dates, placing a single cache breakpoint only at the end of that combined block. Because the shared components were never stored independently, a review in one repository could never leverage cached data generated by a review in another.
On September 16, engineers introduced a marker and plugin hook that split the system prompt block in two, ensuring the first half shared identical bytes across every job and received its own distinct breakpoint. All repository-specific variables, such as available skills and dynamic tool descriptions, were removed from the preamble. Simultaneously, the reference index for the Codex standards reviewer was integrated directly into its prompt, eliminating the need for every session to spend its initial turns reading reference files via tool calls. Together, these modifications dropped the average cost per review from $2.47 to $2.02.
Fewer Calls, Fewer Dollars
Subsequent optimizations deployed between September 21 and 23 targeted wasteful tool usage and redundant processing. Sub-reviewers previously had visibility into the tool that posts final review comments, prompting them to attempt calling it in up to 76% of sessions, where server-side checks would reject the call and force a recovery turn. Removing inaccessible tools eliminated wasted execution loops and cut thousands of tokens of tool definitions from requests.
Other improvements included enabling parallel file reads, skipping redundant sub-reviewer checks on repeated reviews of the same merge request, and terminating sessions immediately after posting the review rather than passing results back for an unread summary generation. Furthermore, the introduction of status self-checks every 15 seconds ensured that canceled GitLab jobs terminated immediately rather than continuing to consume computational resources silently.
The most complex architectural adjustment involved sharing a single cached prefix across the code quality, security, and performance reviewers. By ensuring these three specialist reviewers send identical initial requests containing the same tools, shared rules files, and merge request patches, the system allows the first reviewer to write to the cache while the subsequent two read from it. Because cache entries cannot be read until the writing response begins streaming, the second and third reviewers pause for up to 20 seconds to synchronize, a minor latency trade-off that yields significant cost reductions.
Changing Models and Effort
Further financial efficiency was unlocked by adjusting model variants and execution effort levels. Transitioning full-tier jobs to medium effort on the same model reduced run times by 20%, prompting the retirement of redundant configurations. Later evaluations showed that migrating to Opus 5.5 at medium effort maintained identical review outcomes and critical finding counts compared to Opus 5, while operating at a lower cost per token.
Similarly, splitting Sonnet reviewers between Sonnet 5 and Sonnet 5.5 demonstrated that the newer model version delivered identical critical findings while utilizing significantly fewer tokens, resulting in a swift migration of all Sonnet workloads to version 5.5. Additional testing proved that running code quality reviews on high effort rather than medium effort increased critical and warning findings sufficiently to justify the modest cost increase, guiding the final production configuration.
Latency
While optimization efforts focused primarily on financial expenditures, the reduction in unnecessary model calls delivered a dramatic improvement in review latency. Prior to the updates, over 37% of reviews exceeded five minutes in duration. Following the optimizations, that figure dropped to just 0.7%, with the 99th percentile of review speeds outperforming three-quarters of all reviews measured in April. Across every tier, processing times compressed significantly, improving the developer experience across the organization.
Review Quality
A sharp reduction in operational costs raises valid concerns regarding evaluation thoroughness. While raw counts of critical and warning findings declined across code quality, security, and performance reviewers, a deeper investigation revealed that the drop was largely driven by the elimination of minor observations, duplicated warnings, and artificially inflated severity ratings.
Comparative tracking of author behavior demonstrated that engineers acted on 59% of the code quality reviewer’s serious findings, up from 34% prior to the changes. Consequently, despite flagging fewer total items, the reviewer generated a comparable volume of actionable code modifications per 1,000 reviews. Blinded evaluations conducted by independent AI models confirmed that important findings per review remained steady, while instances of minor issues being mislabeled as critical dropped by 70%.
System reliability also improved noticeably. Reviews terminating without a verdict fell from nearly 0.8% down to 0.27%, and security reviewer retry rates dropped close to zero.
Where This Leaves Us
Cloudflare’s optimization initiative successfully transformed its automated code review pipeline, reducing average costs from $2.43 down to $0.77 and cutting median review durations to under two minutes. The engineering effort underscores the importance of scrutinizing prompt caching boundaries, ensuring cross-repository prefix reuse, and continuously evaluating model efficiency against output quality. For organizations operating autonomous AI agents at scale, ensuring that initial requests properly leverage prompt caching and verifying that review quality remains robust at optimized effort levels remain critical operational imperatives.
Leave a Reply