Just as possessing a valid driver’s license proves identity and confirms that an individual has passed a driving test but says nothing about whether they will drive responsibly on the road, knowing the operator of a web bot provides accountability without guaranteeing safe behavior. A recognized bot can still ignore a website’s published preferences, flood origin servers with excessive requests, or attempt to evade security controls entirely. While these infractions are not always malicious—frequently stemming from software bugs or misconfigurations—they underscore a growing challenge in web management: verifying identity is only the first step, while true trust depends entirely on subsequent actions.
Cloudflare’s approach to bot management centers on answering two fundamental questions: "Are you who you say you are?" and "Can we trust you?" To qualify as a Cloudflare Verified Bot, an automated agent must generally meet three core criteria, utilizing Web Bot Auth to cryptographically sign requests that establish its identity. These verified identities are then cataloged in an open format via BotBase. Once identification is addressed, the focus shifts to the second cardinal rule of verified automation: doing no harm.
To ensure that the answer to the trust question remains affirmative, Cloudflare is rolling out a new system designed to evaluate and report when verified bot behavior deviates from established norms. This initiative highlights the critical distinction between risk and trust in automated web traffic. A notable example of this dynamic occurred in 2025, when Cloudflare de-listed Perplexity’s bot from its Verified bot list after the crawler sought to access resources on websites that had explicitly opted out of scraping, subsequently attempting to circumvent blocks when those requests were denied.
Do not engage in abusive behavior
While the expectation that automated crawlers should not engage in abusive behavior sounds straightforward, enforcing it involves considerable nuance. When crawlers fail to respect site preferences, they erode trust. Similarly, when website operators explicitly ask a crawler not to index their content or to slow down its request rate, and the crawler ignores those directives, the relationship breaks down.
To address this, Cloudflare has developed a system to measure how consistently a crawler respects site preferences. Rather than issuing a permanent judgment about a bot or its operator, these observations translate into an explainable trust band—a dynamic range showing how consistently a bot heeds site preferences based on observed behaviors. The initial signals evaluated by the system reflect common-sense rules designed as a courtesy to the digital publishers who provide the information bots seek. In the future, Cloudflare plans to publish these scores publicly on its Radar platform while expanding its measurement capabilities, giving responsible bot operators a concrete way to demonstrate that trust is being upheld continuously over time. The underlying philosophy is simple: verify, but continuously re-verify.
When in Rome, do as the Romans do
For decades, the standard mechanism for communicating site owner preferences has been the robots.txt file. Cloudflare’s new evaluation system monitors whether a bot’s observed traffic patterns remain consistent with these published preferences over long periods.
A robots.txt file instructs all or specific bots on which sections of a website they are permitted to crawl. Because isolated violations or one-off anomalies may not always indicate systemic issues, Cloudflare looks instead for repeated patterns of behavior that disregard site preferences. The baseline expectation is that responsible bots will consult and honor the preferences published by websites, in strict alignment with RFC 9309.
Furthermore, website administrators are reminded that a robots.txt file merely sets the minimum baseline for what bots and automated agents should avoid. It does not imply that any behavior not explicitly forbidden is automatically permitted. For instance, a robots.txt file does not need to explicitly state that a bot must cease activity when a website communicates that continued access is unwelcome through other technical channels.
What you do not wish for yourself, do not impose on others
Even when automated bots remain strictly within allowed paths, they can still place substantial server loads on the websites they visit. One of the most common and direct ways a website pushes back against excessive load is by issuing an HTTP 429 "Too Many Requests" response. This status code serves as a clear directive for a visitor to reduce its request rate, sometimes accompanied by specific guidance on how long to wait before attempting another connection.
Cloudflare’s system actively considers whether a bot’s traffic adapts after receiving this congestion signal. The evaluation checks whether the bot slows down or pauses its operations, or if it brazenly continues sending requests at the exact same pace. This behavioral adaptation is then incorporated into the bot’s trust band over time. Consistently slowing down or pausing increases the bot’s trust score, whereas repeatedly ignoring the signal penalizes it.
Building trust over time
Observing these operational signals represents only the initial step toward making verified automation fully accountable. By pairing confirmed bot identity with ongoing evidence of behavioral compliance, website owners gain greater confidence that the automated agents they permit are actively respecting their boundaries. Simultaneously, responsible bot operators are given a transparent mechanism to prove they deserve trust, while administrators receive vital context regarding the automated traffic arriving at their servers.
This evaluation framework operates alongside existing tools such as Bot Preference Sync and Bot Management to form a comprehensive accountability loop for web traffic. Bot Preference Sync assists website owners in publishing their preferences clearly, while the observation layer measures whether Verified bots consistently honor those preferences. Bot Management then empowers website operators to decide precisely how that traffic should be handled. Together, these technologies help ensure that trust is earned through transparent, accountable behavior rather than assumed based on identity credentials alone.
Cloudflare is already utilizing this evaluation system internally to monitor how consistently Verified bots honor website preferences. Because trust is established cumulatively over time, the company assesses sustained behavioral patterns rather than reacting to isolated events. This initial rollout phase allows Cloudflare to build a robust historical baseline and refine how these observations translate into actionable guidance.
Building upon this foundation, Cloudflare plans to extend the system to all customers, giving them the ability to choose the specific trust thresholds used to determine the handling of Verified bot traffic. The overarching goal remains clear: to keep website preferences meaningful and to guarantee that digital trust is continuously earned through sustained, responsible behavior.
Customers looking to manage automated traffic can already utilize Cloudflare’s Bot Management suite alongside Bot Preference Sync to align their published preferences with configured security policies. Trends concerning the expanding population of Verified bots can be explored publicly via Cloudflare Radar, while professionals interested in contributing to the future of digital identity and accountability can review open engineering and product roles on Cloudflare’s careers portal.
Leave a Reply