Cloudflare has introduced a major update to its BotBase platform, rolling out a dedicated suite of tools designed specifically for bot operators. The new release, titled BotBase for Operators, addresses long-standing challenges in the automated traffic ecosystem by providing unprecedented visibility, streamlined submission tracking, and automated verification processes. This development follows last month’s introduction of Content Independence Day features, which initially focused on giving website owners granular visibility and control over automated traffic through searchable bot directories and Business Insights tools.
While website owners have increasingly demanded the ability to manage automated traffic sustainably, Cloudflare’s latest initiative acknowledges that the modern web ecosystem relies heavily on cooperation from both sides. For the system to function effectively, bot operators require clear, accessible mechanisms to identify themselves, explain the exact functions of their crawlers, and maintain up-to-date operational data. Historically, however, the operator experience ended abruptly the moment a submission form was completed. Operators had no straightforward way to monitor review statuses, understand the rationale behind rejections, or modify existing records without starting over from scratch. BotBase for Operators aims to dismantle this black-box approach by prioritizing operational transparency.
A New Home for Bot Submissions
To better integrate the submission process into the broader landscape of security and trust tools, Cloudflare has relocated the bot intake form within its dashboard. Previously buried under account configurations, the submission interface now resides alongside other security controls under the application security and bot base sections of the Cloudflare dashboard. This centralized placement acknowledges the direct connection between individual bots and the wider web ecosystem, making it intuitive for operators to manage their entries alongside their security configurations.
By solving the initial challenge of finding the submission portal, Cloudflare has shifted its focus to the more pressing concern voiced by developers: understanding what happens after a submission is made. Feedback from numerous bot operators consistently highlighted the frustration of submitting a crawler profile and receiving no further communication regarding its progress or standing.
See Where Your Submission Stands
The core of the new release is the Submission history tab, a dedicated interface designed to eliminate the uncertainty of the review process. Through this dashboard view, operators can track every bot submitted under their account, complete with clearly defined statuses for each entry. If a submission encounters rejection, the system now displays the specific reason for denial. If a bot is accepted but subjected to classification adjustments, the platform details the exact nature of those modifications.
This transparency effectively replaces the previous necessity of emailing customer support simply to check on a review’s progress or to confirm whether a crawler had been evaluated. Furthermore, a filtering option on the bot directory screen allows operators to easily view all submissions associated with their currently logged-in account, ensuring complete oversight of their digital footprint across the network.
Keep Your Bot’s Information Up to Date
Operational details for automated crawlers frequently change over time. Web infrastructure updates can lead organizations to host IP lists at entirely new endpoints, or operators might transition from traditional IP allowlists to cryptographic traffic signing via Web Bot Auth. Previously, reflecting these modifications required submitting an entirely new form and creating a duplicate entry from scratch.
The updated platform introduces the ability to directly edit existing submissions, as well as cancel pending reviews that are still awaiting evaluation. Cloudflare emphasizes that maintaining accurate and current information is vital for crawlers seeking and retaining Verified status. Because Verified status increasingly dictates whether website owners across the network choose to permit specific automated traffic, keeping operational data precise serves the interests of both operators and publishers, even though the final authority to allow or block traffic always rests with the individual site owner.
A Submission Form Built on an Updated, Pragmatic Taxonomy
Modern automated traffic is diverse, with crawlers performing a wide spectrum of tasks ranging from basic search engine indexing to complex AI model training and user-assisted data retrieval. To accurately capture these nuanced behaviors, Cloudflare has integrated the pragmatic taxonomy and content use model initially introduced on July 1. Instead of forcing operators to squeeze their technologies into a single, restrictive label, the intake form requires a multi-faceted description based on behavior, content usage, and operational structure.
First, operators must define the specific actions their bot performs, such as indexing pages for search, acting as an agent on behalf of a user, collecting data, training models, or supporting search engine optimization tools. Multiple applicable behaviors can be selected simultaneously to reflect the true scope of the crawler’s activity.
Second, the form addresses how the bot utilizes the information it reads, recognizing a fundamental difference between a crawler skimming a page to generate a search snippet and one storing content for machine learning model training. By aligning with the Content Signals model—the same framework website owners utilize to establish their own access rules via robots.txt declarations—operators declare the specific level of content use their systems require. This ensures that a bot’s stated intentions can be automatically checked against the explicit preferences of individual publishers.
Third, the taxonomy clarifies the underlying infrastructure and operational responsibility. The platform distinguishes between direct operators—such as search engines crawling the web to build their own proprietary indices—and intermediaries, which are platforms that carry traffic generated by third parties without originating the request themselves. An example of an intermediary includes a general-purpose artificial intelligence assistant that fetches web pages because a user typed a prompt into a third-party application built on that assistant’s API. Providing this comprehensive picture allows website owners to make informed decisions regarding which traffic to welcome and which to restrict.
Faster, More Consistent Review
The growth of the automated web has placed unprecedented strain on traditional review workflows. The volume of new bot submissions has surged dramatically, multiplying approximately sevenfold since 2023. Continuing to evaluate every single submission through manual human inspection was no longer sustainable. Previously, each entry followed a fully manual path where team members checked details against internal rubrics, resulting in thorough assessments that struggled to keep pace with industry demand.
To resolve this bottleneck, Cloudflare has overhauled the review pipeline with automated checks. Submitted bots are now evaluated programmatically to determine whether they represent duplicate entries, whether user-agent patterns are sufficiently distinct to avoid overlapping with existing registrations, and whether claimed verification methods hold validity. The system automatically fetches IP lists, confirms reverse DNS records, and validates Web Bot Auth signatures without requiring manual intervention. Submissions that pass these automated protocols are tracked immediately, while complex cases requiring human judgment are routed to staff with specific flags already attached, significantly accelerating turnaround times for operators.
What’s Next
The launch of BotBase for Operators marks an important step toward establishing a more transparent and collaborative automated traffic ecosystem. By replacing traditional opacity with real-time tracking, editable submissions, and modernized taxonomies, Cloudflare aims to transform a historically one-way application process into an ongoing, manageable relationship. As development continues, the platform will look to introduce further guidance helping operators understand how their crawlers are perceived across the network, ultimately fostering a more sustainable balance between website owners and automated traffic operators.
Leave a Reply