A driver’s license tells you who someone is and that they have passed a driving test, but it does not guarantee that they will drive responsibly on the road. The exact same principle applies to the growing ecosystem of automated bots populating the modern internet: verifying who operates a bot creates a baseline of accountability, but true trust depends entirely on how that bot behaves.
Even a known, authenticated bot can still ignore website preferences, overwhelm origin servers, or attempt to evade security controls. Often, this disruptive behavior is not born out of malicious intent; rather, it is frequently the result of a misconfiguration, an algorithmic bug, or a faulty script.
To address these ongoing challenges surrounding automated traffic, Cloudflare has introduced a new evaluation and reporting system designed to monitor when Verified bot behavior deviates from basic standards of digital citizenship. The initiative aims to ensure that when website administrators ask the critical question, “Can we trust you?”, the answer from automated agents remains a definitive yes.
The Evolution of Bot Identity and Trust
In previous examinations of agentic behavior, Cloudflare outlined two foundational questions that form the backbone of modern web security: “Are you who you say you are?” and “Can we trust you?” For an entity to be recognized as a Cloudflare Verified Bot, it must successfully meet specific criteria, relying on cryptographic mechanisms to establish its identity.
Bots use Web Bot Auth to cryptographically sign requests, confirming their origin and identity. These identities are then cataloged by BotBase in an open format. Once the challenge of identification is successfully addressed, the operational focus shifts directly to the second cardinal rule of verified automation: thou shalt do no harm.
The introduction of Cloudflare’s new evaluation system marks a crucial shift in how the industry handles the distinction between risk and trust. A prime example of this dynamic played out in 2025, when Cloudflare officially de-listed Perplexity’s bot from its Verified bot list. The action was taken after the bot attempted to access restricted resources on websites that had explicitly opted out of data scraping, and subsequently tried to circumvent defensive controls when those requests were blocked.
Do Not Engage in Abusive Behavior
While the requirement that automated crawlers should not engage in abusive behavior sounds straightforward, enforcing it involves considerable nuance. When automated crawlers fail to respect established site preferences, they fundamentally undermine trust. If a website explicitly requests a crawler to halt its activities or slow down its request rate, and the crawler ignores these instructions, the relationship breaks down.
To measure adherence objectively, Cloudflare has developed a system that tracks how consistently a crawler respects site preferences over time. These observations translate directly into an explainable trust band. Rather than issuing a permanent, static judgment about a specific bot or its corporate operator, this trust band provides a dynamic range showing how consistently a bot respects site preferences, alongside the specific observed behaviors that informed the assessment.
These initial signals act as common-sense rules and a baseline courtesy toward the human and organizational operators who provide the information bots seek. In the future, Cloudflare plans to share these trust scores publicly on its Radar platform, as well as expand the methodologies used for measurement. This scoring mechanism provides responsible bot operators with a tangible way to demonstrate that trust is being continuously upheld over time, proving that digital verification requires continuous re-verification.
When in Rome, Do as the Romans Do
For decades, the standard mechanism for communicating site owner preferences has been the humble robots.txt file. Cloudflare’s system evaluates whether a bot’s observed traffic patterns remain consistent with these published preferences over time.
A robots.txt file instructs all bots—or specific designated crawlers—to restrict their activities to approved parts of a website. Isolated violations and minor one-off anomalies are often not statistically significant. Instead, the evaluation system looks for repeated patterns of behavior that systematically disregard site preferences. The baseline expectation is simply that responsible bots will consult and honor the preferences published by website owners, in strict accordance with internet standards such as RFC 9309.
Cloudflare emphasizes that robots.txt establishes the minimum set of rules regarding what automated bots and agents should avoid doing. However, it does not imply that any behavior omitted from the file is automatically permitted. For instance, a robots.txt file does not need to explicitly state that a bot must stop crawling when a website communicates through other technical channels that continued access is unwelcome.
What You Do Not Wish for Yourself, Do Not Impose on Others
Even when automated bots remain entirely within allowed directory paths, they can still place substantial server load on the websites they visit. One of the most common and effective ways for a website to push back against excessive load is by issuing an HTTP 429 status code, which translates to "Too Many Requests."
An HTTP 429 response serves as a direct instruction for a visitor to reduce its request rate, frequently accompanied by guidance indicating how long the visitor should wait before attempting to make another request. Cloudflare’s new system carefully considers whether a bot’s traffic successfully adapts after receiving this congestion signal. The system evaluates whether the bot slows down or pauses its requests, or if it brazenly continues sending requests at the exact same high-speed pace.
This behavioral adaptation is then incorporated into the bot’s trust band over time. Consistently slowing down or pausing upon receiving an HTTP 429 signal increases the bot’s trust score, while repeatedly ignoring the signal penalizes the operator and lowers the trust band.
Building Trust Over Time
Observing these operational signals represents an essential first step toward making automated traffic genuinely accountable over time. By pairing verified bot identity with empirical evidence of ongoing behavioral patterns, Cloudflare can provide website owners with greater confidence that the automated agents they choose to allow are actually respecting their preferences. Simultaneously, it gives responsible bot operators a reliable method to prove they deserve trust, while supplying website owners with critical context about the automated traffic arriving at their digital doorsteps.
This new monitoring framework operates alongside existing infrastructure tools like Bot Preference Sync and Bot Management to form a comprehensive accountability loop for automated web traffic. Bot Preference Sync assists website owners in publishing their preferences clearly, while observing how bots interact with those preferences reveals whether Verified bots honor them consistently. Bot Management then empowers website owners to decide how that specific traffic is handled. Together, these tools ensure that trust is earned through transparent, accountable behavior rather than assumed solely on the basis of a verified identity.
Cloudflare is already utilizing this system internally to evaluate how consistently Verified bots honor website preferences across the network. Because trust is established gradually, the evaluation assesses sustained behavioral patterns rather than reacting to isolated events. During this initial phase, the company is actively building historical data and refining how these observations translate into actionable guidance for the broader web ecosystem.
Building upon this foundational work, Cloudflare plans to make the system available to all of its customers. Users will soon have the ability to select the specific trust threshold used to determine how Verified bot traffic is handled on their properties. The overarching goal remains clear: keeping website preferences meaningful and ensuring that digital trust is continuously earned through sustained, responsible behavior.
