Cloudflare has successfully driven down the operational costs and latency of its internal AI code review systems, achieving a 68% reduction in the average cost per review through rigorous token optimization, prompt restructuring, and careful model adjustments.
When the cloud infrastructure and security company initially detailed its automated AI code reviewer in April, an average review came with a palatable price tag of $1.19. However, by the end of August, that figure had more than doubled to roughly $2.46. This upward creep occurred despite a lack of significant architectural changes; the central coordinator was still routing tasks to the same specialist reviewers, and identical risk tiers continued to govern the intensity of each review.
Recognizing the need to rein in expenses, engineering teams tasked their internal agents in mid-September with tracking down where those tokens were being consumed and cutting them aggressively. Today, operating at high effort for code quality, an average review costs just $0.77—down 68% from the $2.43 average recorded across the company’s two most expensive weeks prior to the initiative. Beyond financial savings, the optimization efforts also accelerated the review process. The median review now takes just 1 minute and 32 seconds, a stark improvement over the 4 minutes and 9 seconds it required previously. Roughly 40% of these savings stemmed from direct code optimizations within the reviewer itself, while the remaining majority was unlocked through experimentation with different model selections and effort configurations.
How a Review Doubled in Cost
An analysis of weekday review costs since March illustrates a steady climb: $1.22 in March, $1.44 in June, $1.74 in July, and finally $2.32 in August. A minor factor in this escalation was a shift toward more complex changes landing in the most expensive review tier, rising from 59% of reviews in March to 65% in August. Because underlying large language models have grown increasingly capable, engineers naturally produced larger, more intricate code changes, pushing a greater volume of work into the comprehensive review tier.
However, the primary driver of the cost increase was model evolution. At the beginning of August, Cloudflare migrated all reviewers onto Anthropic models, which immediately elevated the cost per job from $1.38 to $2.03 because evaluations demonstrated superior performance on this specific workload. The internal standards reviewer, Codex, was moved to Claude Opus 4.7 for its reliability and performance, causing its individual review cost to jump from about $0.25 to $0.88 in August—making it more expensive than the coordinator agent itself.
Simultaneously, cache hit rates improved from 85.7% in April to 97% by September, but the total number of tokens processed per review more than tripled, surging from 0.91 million to 2.99 million. Even with low cache read costs, maintaining a high hit rate on a volume three times larger drove up overall expenditures.
What Else Has Changed Since the Last Post?
A major structural upgrade involved migrating the reviewer to OpenCode v2, a ground-up rewrite of its predecessor. Before transitioning all operations, the team ran every review in designated canary repositories on both versions in a comparative shadow mode. Because both versions reached the same approval decision on 98% of tested merge requests, the team completed a full migration in early October and retired the v1 codebase. When evaluated under identical models, full-tier reviews on v2 finish approximately 13% faster. Engineers are also experimenting with alternative execution environments, such as running the internal AI reviewer solely on Durable Objects utilizing pi-durable.
Where Were the Tokens Going?
Prior to the cost-reduction push, AI review expenditures were split roughly between output tokens (23%), cache writes (43%), and cache reads (34%). This breakdown revealed that the vast majority of capital was being consumed by context the model had already processed or was scheduled to read again.
A single session exposed the underlying mechanics: the code quality reviewer averaged 16.6 large language model calls per review, writing 72,000 tokens to the cache and reading back 1.08 million because each subsequent call transmitted the entire conversation as context. As both the number of calls and the size of the context gradually expanded over the preceding months, session costs scaled accordingly.
Anthropic’s prompt caching operates by preserving prompt prefixes computed by the model at the start of a conversation. Because these prefixes only function for initial messages rather than mid-conversation or concluding exchanges, static elements like harness tools, system prompts, and opening messages are prime candidates for caching. Writing to the cache incurs a cost 1.25 times the normal input price, whereas reading costs just 0.1 times, with entries expiring after five minutes unless accessed. To curb spending, the engineering team needed to ensure the first request of every session read from the cache instead of writing to it, while simultaneously reducing the total number of calls per session.
Sharing a Cached Prefix Between Every Repository
On September 14, the reviewer began logging the token usage of each reviewer’s first LLM call alongside a hash of the initial system block. The data revealed that the opening request of a session found a cached prefix only 14% to 26% of the time. Because every session initiated with identical tool definitions and agent prompts—consisting of 11,000 to 21,000 identical tokens across every repository—virtually all of these requests should have registered as cache hits.
The root cause lay in how OpenCode v1 constructed system prompts, combining the agent prompt with an environment section containing the working directory and date, and placing a single cache breakpoint at the very end. Because the shared components were never stored independently, a review in one repository could never leverage the cached work executed by a reviewer in another.
On September 16, developers introduced a marker at the end of every agent prompt alongside a plugin hook that split the block in two at that precise point. The first half now terminated on shared byte sequences common to every job, securing its own isolated breakpoint. To preserve this cacheability, repository-specific variables were stripped from preceding sections, temporarily disabling the list of available skills and locking tool descriptions against runtime modifications.
Simultaneously, the reference index for the Codex internal standards reviewer was integrated directly into its prompt. Previously, every session spent its opening three turns fetching reference files via tool calls. Embedding these files reduced LLM calls per run from 15.9 to 9.6 and dropped the cost per run from $1.12 to $0.72, driving down the overall cost per review from $2.47 to $2.02.
Fewer Calls, Fewer Dollars
A subsequent series of refinements deployed between September 21 and 23 collectively shaved roughly $0.20 off every review. These included removing unused tools—such as the comment-posting tool visible to sub-reviewers, which previously triggered wasted recovery calls in up to 76% of sessions—and enabling parallel file reading so reviewers could ingest shared context and patches up to 150 KB in their initial response.
Additional efficiencies involved skipping redundant reviewers on secondary or later reviews of the same merge request, halting execution immediately after a review is posted rather than forcing an unnecessary summary turn, preventing slow reviewers from being misidentified as provider failures, and implementing self-monitoring logic in GitLab jobs to terminate execution if a container is canceled.
Furthermore, the code quality, security, and performance reviewers were synchronized to share an identical first request containing shared tools, rules files, and all merge request patches. Because cache entries cannot be read until the generating response begins streaming, secondary and tertiary reviewers pause for up to 20 seconds to wait for the initial response to begin before dispatching their own requests, saving approximately nine cents per reviewer.
Changing Models and Effort
Configuration adjustments surrounding models and effort levels played a substantial role in the overall savings. Standardizing full-tier jobs to medium effort on the coordinator level accelerated completion times by 20%. Subsequent evaluations showed that transitioning to Opus 5.5 at medium effort produced no measurable difference in critical findings compared to older configurations while lowering costs.
Further savings materialized after splitting Sonnet reviewers between Sonnet 5 and Sonnet 5.5 in early October. Sonnet 5.5 proved significantly cheaper per run across code quality, performance, and security workloads purely through superior token efficiency. Subsequent testing on code quality revealed that running Sonnet 5.5 at high effort increased costs by 61% but yielded 58% more critical and warning findings, confirming that the elevated effort tier was well worth the investment.
Latency and Review Quality
While latency reduction was not an explicit goal, fewer sequential calls naturally translated to faster reviews. The 99th percentile of review times improved drastically, with slow reviews dropping from 37.9% exceeding five minutes down to just 0.7%. Median full-tier reviews dropped from 5 minutes and 16 seconds to 1 minute and 54 seconds.
To verify that cost-cutting measures did not compromise review integrity, Cloudflare analyzed historical author behavior and subjected a blinded sample of findings to independent model evaluation. The data revealed that authors acted on 59% of the code quality reviewer’s serious findings, up from 34% previously. Although fewer total issues were flagged, the proportion of actionable insights remained consistent. Independent judges confirmed that the reduction in findings primarily eliminated minor observations, duplicates, and inflated severity ratings, while the frequency of genuinely important findings held steady.
Overall reliability also improved, with reviews ending without a verdict dropping from 0.79% to 0.27%, and security reviewer retries falling from 3.5% to 0.04%.
Leave a Reply