Cloudflare has announced a major enhancement to its observability infrastructure, rolling out Logpush Transformers to general availability and extending the feature across all self-serve plans. The update marks a significant step in the company’s broader initiative to bring enterprise-grade observability features to every customer tier, ensuring that organizations of all sizes can efficiently route, filter, and reshape their telemetry data. The rollout was part of a broader package of eight major updates to Cloudflare Observability unveiled during the company’s annual Birthday Week celebration.
Logpush is designed to deliver logs in batches as quickly as possible, enabling Cloudflare to provide crucial network and security information to users almost in real time. Observability teams traditionally rely on Logpush to export Cloudflare logs directly to security information and event management systems, commonly known as SIEMs, as well as to third-party analytics platforms and object storage buckets. However, before those logs can safely and effectively reach their ultimate destinations, engineering teams frequently encounter the need to perform intermediate data massaging. This often involves stripping out routine health checks, omitting sensitive Personally Identifiable Information, or completely reshaping log records so they seamlessly fit into existing corporate dashboards and investigative workflows.
Historically, meeting these requirements has forced organizations to deploy, scale, monitor, and maintain a separate data-processing service specifically designed to handle log transformations before ingestion. For example, a modern enterprise collecting logs from multiple Content Delivery Networks might need to inject a static provider field into every incoming record to properly normalize data before it hits a central analytics platform. Furthermore, teams often incur unnecessary financial costs by paying to export, ingest, and store massive volumes of log records that they intend to discard immediately upon arrival. Filtering out unwanted traffic before delivery significantly reduces downstream data volume and, consequently, lowers overall cloud storage and ingestion expenses.
With the general availability of Transformers, that intermediate processing layer is now integrated directly into the core Logpush export workflow. Instead of maintaining external services, teams can define a custom SQL query to filter incoming records, redact sensitive data fields, and reshape the final output by adding, renaming, removing, or nesting attributes. Because Logpush evaluates and applies this query prior to export, each designated destination receives data precisely in the required schema. Cloudflare achieved this capability by embedding an advanced SQL transformation engine directly inside Logpush’s high-throughput export pipeline, maintaining predictable performance and resource utilization without sacrificing delivery speed.
How Logpush Transformers Work
At its core, a Transformer is simply a SQL query written specifically for an individual Logpush dataset. The query references native fields from that specific dataset and dictates exactly which records and fields Logpush should process and export. When a job runs, Logpush applies the defined query to every batch of logs prior to delivery. Any records that fail to match the criteria outlined in the query’s WHERE clause are immediately dropped. For the records that do survive, the output generated by the SELECT statement becomes the final exported record.
Because these Transformers can fundamentally alter the structure and internal contents of each record, the resulting transformed data is exported in newline-delimited JSON format, widely known as NDJSON. Customers are given full control over this process and can create, preview, and manage their Transformers directly within the Cloudflare dashboard using an interface called Transformer Studio, or they can handle the configuration programmatically through the platform’s API. Once a Transformer is saved, administrators can attach it to any compatible NDJSON Logpush job utilizing the same underlying dataset. Every time a Transformer is updated, the system automatically generates a new version, allowing engineering teams to easily track and review how their data transformation logic has evolved over time.
To illustrate the practical application of this workflow, consider a standard Cloudflare HTTP request log. Left unmodified, the raw log contains a wealth of operational data that might not align with a custom schema enforced by a team monitoring multiple CDN vendors simultaneously. By deploying a Transformer, an engineer can construct a SQL query that extracts the epoch timestamp, uses regular expressions to redact portions of the client IP address for privacy compliance, standardizes HTTP request methods to uppercase, isolates edge response status codes, injects a static provider identifier, and groups host and path attributes into a nested request object.
At the same time, the query’s filter logic can be configured to completely drop internal traffic, monitoring probes, and health check endpoints such as specific paths and subdomains. The resulting NDJSON record emitted by the pipeline emerges clean, properly nested, anonymized, and stripped of unnecessary noise, matching the exact expectations of downstream analytics platforms without requiring external intervention.
How Transformers Executes SQL with Apache DataFusion and Arrow
Cloudflare’s massive global infrastructure ingests more than one billion events every second. Its dedicated Logpush delivery service, known internally as Bufstream, is responsible for pushing more than 110 million Logpush records every second to customer-configured destinations. This delivery engine operates across hundreds of Kubernetes pods, with each pod claiming a specific time slice of incoming data originating from Buftee, Cloudflare’s distributed, log-based in-house message broker. This architecture naturally distributes computational work horizontally across various Logpush jobs.
Because job volumes follow a Pareto-like distribution—where the vast majority of jobs operate at relatively low volumes while a small cohort of high-volume enterprise jobs processes millions of logs every single second—any integrated transformation feature had to adhere to strict operational standards. Transformers needed to match the reliability, high throughput, and predictable resource utilization characteristic of the rest of the Logpush platform, all without introducing new single points of failure into the critical export hot path.
Initial engineering prototypes experimented with running customer-defined WebAssembly Transformers inside Cloudflare Workers, which provides globally distributed and isolated compute environments. However, engineers quickly realized that invoking a Worker would introduce an external network call directly into the Logpush export path. That remote network dependency carried the inherent risk of introducing transient timeouts and retries, which in turn could degrade delivery throughput and increase overall latency.
To prioritize absolute reliability and maximum delivery throughput, the development team ultimately decided to keep Transformer execution entirely in-process within the core delivery service. After evaluating multiple candidates, Cloudflare selected Apache DataFusion as the underlying SQL execution engine, a technology already proven in production within other parts of the company’s ecosystem, such as Cloudflare Basin.
Apache DataFusion is a modern, high-performance query engine written in Rust and built directly upon the Apache Arrow ecosystem. It features a robust SQL planner alongside a vectorized, columnar execution engine designed to maximize modern CPU capabilities. Internal benchmarking revealed impressive performance metrics: executing a prepared Transformer against a synthetic batch of five thousand HTTP request rows originating from a single Go goroutine took approximately three milliseconds, translating to a processing rate of roughly 1.6 million rows per second.
Attaching a Transformer to a job naturally introduces additional CPU overhead along the processing path. To mitigate this, Bufstream incorporates per-job transform parallelism, allowing high-volume jobs to execute transformation batches concurrently. By embedding DataFusion directly into Bufstream, these transformation batches can leverage the service’s existing delivery infrastructure, including its sophisticated scheduling, batching, and destination controls, entirely avoiding the need for a separate transformation tier or additional network hops.
However, selecting DataFusion meant bridging Bufstream—which is written in the Go programming language—with an execution engine written in Rust. The primary architectural challenge lay in moving strongly typed data batches efficiently across that strict language boundary.
From Records to Columns
Bufstream receives compressed Logpush batches containing Cap’n Proto records directly from the internal message broker. When a job has an active Transformer attached, Bufstream decompresses the batch, parses the eligible logs, and constructs a typed Apache Arrow RecordBatch before the execution of the transform begins.
Apache Arrow serves as a language-independent, columnar in-memory data format specifically engineered to facilitate hyper-efficient analytics on modern hardware. It represents both flat and nested data structures as contiguous, strongly typed columns. By utilizing Apache Arrow Interprocess Communication, or IPC, Bufstream and DataFusion share a standardized mechanism to exchange data batches smoothly across the foreign function interface boundary between Go and Rust. This boundary is exposed via a standard C application binary interface and invoked cleanly from Go using cgo, the language’s native mechanism for calling C-language code.
Crucially, the Go and Rust runtimes do not share ownership of the transform data in memory. Go passes a direct pointer pointing to its input Arrow IPC buffer to Rust strictly for the duration of the cgo invocation, and Rust is strictly prohibited from retaining that pointer after returning control. Conversely, Rust assumes full ownership of the transformed output buffer, which Go then copies into Go-managed memory before explicitly releasing the Rust-side allocation. This strict discipline makes memory ownership and object lifetimes explicit across the language boundary, effectively preventing cross-runtime memory management bugs and leaks.
Using Arrow IPC allows the embedded Rust DataFusion wrapper to utilize an Arrow StreamReader to read typed RecordBatch values directly, bypassing the need to parse individual rows from an intermediate text format like JSON and rebuild typed Arrow columns from scratch. Because field types, nullability constraints, and nested values are represented consistently on both sides of the language divide, DataFusion receives data in its native columnar model.
While crossing the Go and Rust boundary does carry minor overhead—as batches still require serialization, decoding, and memory management—the adoption of Arrow IPC keeps that work strictly batch-oriented, successfully avoiding costly per-record parsing and conversion cycles. This design makes the integration well-suited for Logpush’s high-volume delivery pipeline while simultaneously enforcing strict schema validation against the official Logpush dataset definitions.
Every Arrow batch requires an accompanying schema that clearly defines its constituent fields and data types. Rather than attempting to dynamically infer schemas from the logs found within an isolated batch, Bufstream supplies a pre-generated schema derived directly from each Logpush dataset’s formal specification. This provides DataFusion with a stable, predictable view of all available fields and data types, enabling comprehensive query validation prior to execution. The specific table named within a Transformer’s FROM clause dictates which dataset schema the query will utilize.
Executing the Transform and Handling Failures
When a customer saves a new or updated Transformer through the API or the dashboard interface, Cloudflare stores the associated SQL query securely in R2 storage. Bufstream retrieves this query via a multi-layer caching mechanism and translates it into a reusable DataFusion logical plan, which formally defines the required scan operations, filters, projections, and expressions. Bufstream prepares this execution plan precisely once per time slice of a job’s data and efficiently reuses it across every incoming Arrow batch within that same slice. As subsequent Arrow batches arrive, the Rust wrapper binds the batch as the logical plan’s table input, executes the compiled query, and returns fresh Arrow batches containing only the filtered records and projected fields.
A critical design principle of Logpush Transformers is that they are built to fail closed. If a particular data transformation fails to execute correctly for any reason, Bufstream will neither advance the delivery offset of the job nor transmit the untransformed, raw data batch downstream. Delivering raw data in place of transformed records could easily violate the strict schema, filtering, or redaction expectations of downstream security and analytics systems, making failure isolation a top priority. Instead, Bufstream handles failures by re-attempting the transformation process.
To prevent a persistently failing job from monopolizing system capacity and starving healthy workloads, Bufstream implements progressive backoff delays for retries and surfaces explicit error diagnostics directly on the affected job. Jobs that remain trapped in a failing state for an extended duration are automatically disabled to protect overall platform stability. Customers can actively monitor successful and failed transformation counts, data volumes measured before and after transformation, and overall transformation durations through the Transformer Health section located within the main Health tab of the Logpush dashboard.
After successful execution, Bufstream converts the resulting Arrow output back into standard NDJSON logs. It subsequently batches, gzip-compresses, and uploads those logs to the customer’s configured destination. Because external destinations naturally accept data at varying rates and can experience temporary slowdowns, Bufstream dynamically adjusts its internal upload concurrency while continuously respecting strict destination limits. By embedding SQL execution directly into Bufstream, Transformers seamlessly leverage Logpush’s existing distributed infrastructure, robust batching logic, and destination controls, abstracting away complex data engineering challenges while allowing users to apply sophisticated modifications with minimal effort.
The feature is currently available across Free, Pro, Business, and Enterprise plans, with comprehensive configuration details and pricing information documented in Cloudflare’s official developer resources.
Leave a Reply