Just as a driver’s license proves a person’s identity and confirms they have passed a driving test, it offers no absolute guarantee that they will drive responsibly on the road. The same fundamental principle applies to the automated bots that roam the modern internet. While verifying who operates a bot establishes accountability, true trust ultimately depends on how that bot behaves. Even a known, verified bot can still ignore a website’s preferences, overwhelm origin servers, or attempt to evade security controls. Often, these violations are not driven by malicious intent, but rather stem from misconfigurations or underlying software bugs.
Addressing this ongoing challenge in internet infrastructure, Cloudflare has introduced a new system designed to evaluate and report when the behavior of a verified bot begins to deviate from basic rules of digital conduct. The initiative aims to ensure that the fundamental question of whether a bot can be trusted always receives a definitive answer.
In previous discussions surrounding bot behavior, infrastructure and security companies have highlighted the necessity of verifying two core questions: whether a bot is truly who it claims to be, and whether it can be trusted. Traditionally, bots establish their identity using tools like Web Bot Auth to sign incoming requests, which are then cataloged in an open format through systems like BotBase. Once identification is successfully addressed, the focus shifts naturally to the cardinal rule of verified automation: doing no harm.
The distinction between risk and trust plays a critical role in this framework. A prominent example occurred when Perplexity’s bot was de-listed from Cloudflare’s Verified bot list after it attempted to access resources on websites that had explicitly opted out of data scraping. When those initial requests were blocked, the bot subsequently attempted to circumvent the established controls. Incidents like these highlight why identity alone is no longer sufficient for managing automated web traffic.
Do Not Engage in Abusive Behavior
While the requirement that automated crawlers avoid abusive behavior may sound straightforward, implementing and enforcing it involves considerable nuance. When automated crawlers fail to respect site preferences, it directly undermines trust between website operators and bot creators. When individual publishers request that a crawler either stop harvesting data or reduce its request rate, and the crawler ignores those directives, the breach of trust deepens.
To counter this, Cloudflare has developed a measurement system to track how consistently a crawler honors site preferences over time. Rather than issuing a permanent, static judgment against a bot or its operator, these observations translate into explainable trust bands. These bands provide a dynamic range that shows how consistently a bot respects site preferences alongside the specific observed behaviors that informed the assessment.
Cloudflare treats these operational standards as common-sense rules and basic courtesies extended to the publishers who provide the information bots seek. Moving forward, the company plans to publish these scores openly on Cloudflare Radar while simultaneously expanding its measurement capabilities. This scoring mechanism provides responsible bot operators with a tangible way to demonstrate that trust is being actively maintained over time, reinforcing the security philosophy of verifying and continuously re-verifying automated actors.
When in Rome, Do as the Romans Do
The traditional standard for communicating site owner preferences has long been the robots.txt file. Cloudflare’s new evaluation system carefully considers whether a bot’s observed traffic patterns remain consistent with these published preferences over extended periods.
A robots.txt file instructs all or specific bots to limit their crawling activities to designated parts of a website. Because isolated violations and minor one-off errors can happen during large-scale operations, Cloudflare’s system looks instead for repeated patterns of behavior that deliberately disregard site preferences. The baseline expectation is simply for responsible bots to consult and honor the preferences published by websites, aligning with established internet engineering standards such as RFC 9309.
Importantly, a robots.txt file establishes only the minimum rules for what bots and automated agents should avoid doing. It does not carry any implicit assumption that actions not explicitly forbidden are automatically permitted. For instance, a robots.txt file does not need to explicitly state that a bot must halt its activities when a website communicates through other technical signals that continued access is unwelcome.
What You Do Not Wish for Yourself, Do Not Impose on Others
Even when automated bots remain strictly within allowed paths, they can still place substantial server loads on the websites they visit. One of the most common and direct ways a website can push back against heavy automated traffic is by issuing an HTTP 429 response, commonly known as a "Too Many Requests" error. This status code serves as a clear directive for a visitor to reduce its request rate, frequently accompanied by specific guidance on how long to wait before attempting another connection.
Cloudflare’s evaluation system examines whether a bot’s traffic successfully adapts after receiving this congestion signal. Analysts evaluate whether the bot slows down or pauses its requests, or whether it stubbornly continues sending data at the exact same pace. This behavioral response is then incorporated into the bot’s evolving trust band. Consistently slowing down or pausing increases the bot’s trust score over time, whereas repeatedly ignoring the signal lowers it.
Building Trust Over Time
Observing these behavioral signals represents just the first step toward making verified automation fully accountable across the digital ecosystem. By pairing bot identity with verifiable evidence of ongoing behavior, platforms can give site owners greater confidence that allowed bots are respecting their boundaries. At the same time, this approach gives legitimate bot operators a reliable method to prove they deserve trust, while providing website owners with crucial context regarding the automated traffic arriving at their servers.
This evaluation system integrates with existing tools like Bot Preference Sync and Bot Management to form a comprehensive accountability loop for automated traffic. Bot Preference Sync assists website owners in publishing their preferences clearly, while traffic observation mechanisms reveal whether Verified bots consistently honor those choices over time. Meanwhile, Bot Management empowers website owners to decide precisely how that traffic should be handled. Together, these tools help ensure that trust is earned through transparent, accountable behavior rather than assumed solely on the basis of verified identity.
Cloudflare is already utilizing this system internally to evaluate how consistently Verified bots honor website preferences. Because trust is inherently built over time, the system assesses sustained behavioral patterns rather than isolated events. The company is leveraging this initial phase to build historical data and refine how observations translate into actionable guidance. Building upon this foundation, Cloudflare plans to extend the system to all customers, allowing them to choose the specific trust thresholds used to determine the handling of Verified bot traffic.
For customers looking to manage automated traffic immediately, Cloudflare’s existing Bot Management and Bot Preference Sync tools remain available to align published preferences with configured security policies. Trends regarding the growing population of Verified bots can also be tracked publicly on Cloudflare Radar.
Leave a Reply