Cloudflare has announced a major expansion of its observability tooling, making its powerful Logpush service available across all self-serve plans while bringing its advanced Logpush Transformers feature to general availability. The updates, which formed part of the company’s recent product cycle, reflect a broader institutional commitment to democratizing enterprise-grade features and making complex infrastructure capabilities accessible to every user, regardless of organization size.
For observability teams operating modern web applications, the ability to rapidly export logs to security information and event management systems, analytics platforms, and object storage is critical. However, before those high-volume logs reach their final destination, organizations frequently encounter the need to filter out health checks, redact sensitive user data, or completely reshape records to fit specific dashboard schemas and investigative workflows. Historically, accomplishing this required deploying, scaling, monitoring, and maintaining an entirely separate, intermediate data processing service.
These secondary processing layers add operational overhead and financial costs. For instance, teams collecting telemetry from multiple content delivery networks often need to inject static provider fields to normalize incoming data before it hits their central analytics platform. Furthermore, organizations routinely pay for exporting, ingesting, and storing raw records that they intend to discard immediately upon arrival. By filtering out unneeded traffic at the source, engineering teams can significantly reduce downstream data volume and suppress unnecessary storage bills.
With the general availability of Transformers, Cloudflare has embedded this necessary data manipulation directly into the Logpush export workflow. Instead of maintaining external infrastructure, teams can define concise SQL queries to filter unwanted records, redact sensitive information, and reshape outputs by adding, renaming, removing, or nesting fields. Because Logpush applies these queries prior to export, every destination receives data pre-formatted and ready for analysis. By embedding a high-throughput SQL transformation engine directly into the delivery pipeline, Cloudflare has automated data hygiene while maintaining predictable performance and resource utilization.
How Logpush Transformers Work
At its core, a Transformer is a custom SQL query written specifically for a chosen Logpush dataset. The query references fields native to that dataset, establishing clear rules for which records and fields should be exported. When a job runs, Logpush applies the query to each batch of logs before transmission. Any records failing to satisfy the conditions specified in the WHERE clause are instantly dropped. For the records that survive, the output defined by the SELECT statement dictates the final exported structure.
The resulting transformed records are exported in newline-delimited JSON format, commonly known as NDJSON. Users can build, preview, and govern these Transformers directly within the Cloudflare dashboard using a dedicated interface called Transformer Studio, or they can manage them programmatically through the platform’s API. Once a Transformer is saved, it can be attached to any compatible NDJSON Logpush job sharing the same underlying dataset. The platform automatically tracks updates by generating a new version with each modification, enabling engineers to audit how a Transformer has evolved over time.
To illustrate the capability, consider a standard Cloudflare HTTP request log. Left unmodified, the raw JSON record contains a wealth of operational telemetry. By applying a custom Transformer, an engineering team can effectively normalize this log to match a unified schema shared across multiple CDN providers. The query can perform several transformations simultaneously: it can extract epoch timestamps, selectively truncate and redact segments of client IP addresses to protect user privacy, convert HTTP methods to uppercase, filter out internal traffic and health-check endpoints, and nest host and path attributes into a cleaner sub-object while flagging server errors.
This flexibility allows organizations to enforce strict privacy compliance, eliminate noise from automated monitoring tools, and standardize schemas across heterogeneous logging pipelines without writing custom middleware.
How Transformers Executes SQL with Apache DataFusion and Arrow
Cloudflare’s massive data pipeline ingests more than one billion events every second. Within this architecture, the Logpush delivery service—known internally as Bufstream—dispatches more than 110 million records every second directly to customer destinations. Bufstream operates across hundreds of Kubernetes pods, each claiming designated time slices of data from Buftee, Cloudflare’s distributed, log-based in-house message broker. This decentralized model distributes workload horizontally across individual Logpush jobs.
Job volumes follow a Pareto-like distribution, where the vast majority of jobs operate at modest volumes while a small cohort of high-volume enterprise jobs processes millions of logs every second. For Transformers to succeed, they had to match the strict operational requirements of the existing Logpush architecture: they needed to guarantee reliable delivery, sustain high throughput, and ensure predictable resource consumption without introducing vulnerable network dependencies into the export hot path.
Initial engineering prototypes explored running customer-defined WebAssembly Transformers inside Cloudflare Workers, leveraging globally distributed and isolated compute nodes. However, invoking a Worker introduced an external network call directly into the Logpush export path. That remote dependency risked introducing transient timeouts, retry loops, delivery latency, and reduced overall throughput.
To prioritize absolute reliability and delivery speed, engineers decided to keep Transformer execution strictly in-process within the delivery service itself. After evaluating various SQL execution engines, the team selected Apache DataFusion, a technology already trusted to power other internal infrastructure components like Cloudflare Basin. Apache DataFusion is a high-performance Rust-based query engine built on top of Apache Arrow, featuring an advanced SQL planner and a vectorized, columnar execution engine. Benchmarks demonstrated that executing a prepared Transformer against a synthetic batch of five thousand HTTP request rows from a single Go goroutine took approximately three milliseconds, equating to a processing speed of roughly 1.6 million rows per second.
Attaching a Transformer to a job inevitably introduces additional CPU computation. To mitigate this, Bufstream incorporates per-job transform parallelism, allowing high-volume jobs to execute transformation batches concurrently. Embedding DataFusion directly into Bufstream enables these batches to leverage the service’s native scheduling, batching, and destination controls, entirely bypassing the need for an external transformation tier or additional network hops.
Choosing DataFusion did introduce a technical challenge: bridging Bufstream, which is written in Go, with an execution engine written in Rust. Solving this required designing a mechanism capable of moving typed batches efficiently across the foreign function interface boundary.
From Records to Columns
Bufstream receives compressed Logpush batches containing Cap’n Proto records directly from the internal message broker. When a job has an active Transformer attached, Bufstream decompresses the batch, parses the eligible logs, and constructs a typed Apache Arrow RecordBatch before initiating the transformation.
Apache Arrow provides a language-independent, columnar in-memory format engineered specifically for high-efficiency analytics on modern computing hardware. By representing flat and nested data as continuous, typed columns, Arrow enables seamless data exchange. The Apache Arrow Interprocess Communication mechanism provides Bufstream and DataFusion with a standardized method to pass batches across the boundary between Go and Rust, exposed through a standard C application binary interface and invoked from Go using standard interop mechanisms.
Crucially, the Go and Rust runtimes do not share ownership of the transformation data. Go passes a pointer referencing its input Arrow IPC buffer to Rust strictly for the duration of the execution call, and Rust relinquishes that reference immediately upon return. Conversely, Rust assumes full ownership of the transformed output buffer, which Go then copies into Go-managed memory before explicitly releasing the Rust-allocated memory. This strict management strategy keeps memory lifetimes explicit across language runtimes, preventing cross-runtime memory management vulnerabilities.
Utilizing Arrow IPC allows the Rust DataFusion wrapper to consume typed record batches directly via a streaming reader rather than parsing individual rows from an intermediate text format like JSON and manually rebuilding typed columns. Because field types, nullability rules, and nested values remain consistent across both sides of the boundary, DataFusion receives data in its native columnar model. Although crossing the language boundary requires serialization, decoding, and memory management, the batch-oriented nature of Arrow IPC avoids the immense overhead of per-record parsing. This preserves the high-throughput capabilities of the delivery pipeline while enforcing strict schema validation against official dataset specifications.
Every Arrow batch relies on a predefined schema detailing its fields and types. Rather than attempting to infer schemas dynamically from individual log batches, Bufstream supplies a generated schema derived directly from each Logpush dataset’s formal specification. This provides DataFusion with a stable, predictable view of available fields, facilitating robust query validation.
Executing the Transform and Managing Failures
When customers save or update a Transformer through the dashboard or API, Cloudflare securely stores the SQL query in object storage. Bufstream retrieves the query through a multi-layer caching system and translates it into a reusable DataFusion logical plan encompassing the necessary data scans, filters, projections, and expressions. Bufstream prepares this execution plan once per time slice of a job’s data, reusing it across every Arrow batch within that slice. For each incoming batch, the Rust wrapper binds the data as the plan’s table input, executes the query, and returns Arrow batches containing only the selected records and fields.
Transformers are engineered with a strict fail-closed security posture. If a transformation encounters an error and cannot be executed properly, Bufstream will neither advance the delivery offset of the job nor transmit the untransformed batch. Delivering raw data in place of transformed records could violate downstream schema assumptions, data filtering rules, or privacy redaction mandates, making retries the preferred path over data corruption.
To prevent a persistently failing job from exhausting system capacity needed by healthy workloads, Bufstream progressively backs off retry intervals while surfacing descriptive error states on the affected job. Jobs that remain in a failing state over an extended duration are automatically disabled to protect platform stability. Organizations can monitor successful and failed transformation counts, data volumes measured before and after processing, and transformation durations directly within the dedicated health interface inside the Logpush dashboard.
Following successful execution, Bufstream converts the resulting Arrow output back into NDJSON logs, which are then batched, compressed, and uploaded to the customer’s configured destination. Because destination endpoints ingest data at varying rates and may occasionally experience slowdowns, Bufstream dynamically adjusts upload concurrency while respecting established destination limits. By embedding SQL execution natively into the delivery architecture, Cloudflare abstracts away complex distributed systems engineering, allowing customers to deploy advanced log processing with minimal effort.
Logpush Transformers are now available across Free, Pro, Business, and Enterprise plans, marking a significant step forward in making enterprise-grade observability accessible to developers worldwide.
Leave a Reply