Just as a driver’s license confirms a person’s identity and proves they passed a driving test rather than guaranteeing they will drive responsibly on the road, verifying the operator of an automated bot only establishes accountability. True trust, however, depends entirely on what the bot does next. A known, authenticated bot can still ignore website preferences, overwhelm origin servers, or evade established controls. While these infractions are not always malicious—frequently stemming from simple misconfigurations, software bugs, or poorly managed scripts—they create significant friction for website operators trying to manage their server loads and protect their content.
To address this challenge in the evolving digital landscape, Cloudflare is rolling out a new system designed to evaluate and report when verified bot behavior deviates from foundational rules of conduct. The initiative aims to ensure that the answer to the critical question—“Can we trust you?”—remains a definitive affirmative.
For years, the web infrastructure and security giant has focused on answering two fundamental questions regarding automated traffic: “Are you who you say you are?” and “Can we trust you?” To achieve the status of a Cloudflare Verified Bot, an entity must satisfy multiple criteria. Bots utilize Web Bot Auth to cryptographically sign requests, establishing a verifiable identity that Cloudflare’s BotBase then catalogues in an open format. Once identification is securely addressed, the focus shifts to the second cardinal rule of verified automation: doing no harm.
The distinction between risk and trust sits at the heart of this new evaluative approach. A vivid example of this dynamic played out in 2025 when Cloudflare de-listed Perplexity’s bot from its Verified bot list. The action was taken after the crawler sought to access resources on websites that had explicitly opted out of data scraping, subsequently attempting to circumvent site controls when those initial requests were blocked. This incident underscored a broader industry problem: mere identification is insufficient if the automated agent operates contrary to the explicit wishes of the website owner.
Do Not Engage in Abusive Behavior
While the requirement for crawlers to avoid abusive behavior sounds straightforward, immense nuance exists in practice. When crawlers fail to respect site preferences, they actively undermine trust. When website operators explicitly request that a crawler refrain from indexing certain directories or slow down its request rate, and the crawler ignores these directives, trust erodes rapidly.
To combat this, Cloudflare has developed a measurement system designed to gauge how consistently a crawler respects site preferences. Rather than issuing a permanent, static judgment about a specific bot or its operator, these observations translate into an explainable trust band. This flexible range illustrates how consistently a bot respects site preferences over time, highlighting the specific observed behaviors that informed the assessment.
Cloudflare considers these operational guidelines to be common-sense rules—a basic professional courtesy extended to the operators who provide the valuable information bots seek. Moving forward, the company plans to share these trust scores publicly on its Radar platform, alongside plans to expand the metrics it evaluates. This scoring mechanism provides responsible bot operators with a tangible way to demonstrate that trust is being actively upheld. The underlying philosophy mirrors the security mantra: verify, but continuously re-verify.
When in Rome, Do as the Romans Do
The robots.txt file has long served as the primary mechanism for website owners to communicate their preferences to automated agents. Cloudflare’s new system evaluates whether a bot’s observed traffic patterns remain consistent with these published preferences over an extended period.
Robots.txt files instruct all or specific bots to limit their crawling activities to designated parts of a website. Isolated violations, single-family errors, and one-off anomalies are rarely significant on their own. Instead, the evaluation system looks for repeated patterns of behavior that systematically disregard site preferences. The baseline expectation is simply for responsible bots to consult and honor the preferences published by websites, aligning closely with international standards such as RFC 9309.
Cloudflare clarifies that a robots.txt file establishes the minimum baseline rules for what bots and agents should not do. However, it carries no implicit assumption that anything not explicitly forbidden is automatically allowed. For instance, a robots.txt file does not need to explicitly state that a bot must halt its operations when a site communicates that continued access is unwelcome through alternative signals.
What You Do Not Wish for Yourself, Do Not Impose on Others
Automated bots can remain entirely within allowed directory paths while still placing a substantial load on the websites they visit. One of the most direct ways a website can push back against excessive traffic is by issuing an HTTP 429 status code, indicating “Too Many Requests.” This response serves as a direct directive for the visitor to reduce its request rate, often accompanied by specific guidance on how long to wait before attempting another connection.
Cloudflare’s system actively monitors whether a bot’s traffic adapts appropriately after receiving this signal. The evaluation checks whether the bot slows down or pauses its requests, or if it stubbornly continues sending traffic at the exact same pace. These reactions are subsequently incorporated into the bot’s evolving trust band. Consistently slowing down or pausing increases the bot’s overall trust score, whereas repeatedly ignoring the signal penalizes it.
Building Trust Over Time
Observing these traffic signals marks only the first step toward making verified automation genuinely accountable. By pairing cryptographic bot identity with continuous evidence of actual behavior, platform operators can give site owners greater confidence that the bots they choose to allow are respecting their boundaries. Simultaneously, it grants bot operators a reliable method to prove their trustworthiness while providing website owners with deep context regarding the automated traffic landing on their servers.
This newly introduced system operates alongside existing tools like Bot Preference Sync and Bot Management to form a comprehensive accountability loop for automated traffic. Bot Preference Sync assists website owners in publishing their preferences clearly, while observing how bots interact with these preferences reveals whether verified entities consistently honor them over time. Bot Management allows website owners to decide how that incoming traffic is handled based on these insights. Together, these tools ensure that trust is earned through transparent, accountable behavior rather than assumed purely from digital credentials.
Cloudflare is already utilizing this system internally to evaluate how consistently verified bots honor website preferences. Because trust is inherently built over time, the platform assesses sustained behavioral patterns rather than isolated events. The current phase is being used to build historical data and refine how these observations translate into actionable guidance for web administrators. Building upon this foundation, Cloudflare plans to roll out the system to all customers, empowering them to choose the specific trust thresholds used to determine how verified bot traffic is handled on their properties.
For organizations looking to manage automated traffic immediately, Cloudflare customers can leverage Bot Management and Bot Preference Sync to align published preferences with configured security policies. Trends concerning the expanding population of verified bots can be explored publicly via Cloudflare Radar, while those interested in contributing to the future of digital identity and accountability can find open technical roles on the company’s careers page.
Leave a Reply