Skip to content
INTERNET INFRASTRUCTURE & NETWORKS

Revolutionizing Network Forensics: PCAP-LM Bridges the Gap Between Packet Captures and Large Language Models

Packet captures have long stood as the indisputable ground truth of network operations. When a customer reports agonizingly slow downloads, or a dedicated security team races to investigate a sudden network anomaly, engineers inevitably resort to the same ritual: they open a capture file and painstakingly decode the raw data that crossed the physical wire. This foundational work, while essential, is notoriously slow, demands a rare and specialized set of human expertise, and ultimately fails to scale in modern, high-speed networking environments.

Large language models (LLMs) appear to be the natural remedy for this operational bottleneck. Already proven adept at summarizing sprawling system logs, generating complex incident reports, and drafting step-by-step troubleshooting guides, AI seems tailor-made to assist with network diagnostics. Yet, when an engineer hands a standard packet capture to an LLM, the system invariably chokes. When converted into human-readable text using conventional command-line tools, even a modest few megabytes of network traffic can effortlessly balloon to nearly 20 million tokens—vastly exceeding the context window limits of any commercially available language model.

To solve this critical mismatch, researchers have developed PCAP-LM, an innovative translation framework designed to rewrite packet captures so that an AI assistant can read an entire network conversation in a single, cohesive pass. The complete methodology, technical architecture, and empirical findings are detailed in a research paper spearheaded by Xavier Marjou, a telecommunications and deep learning researcher at Orange in France with three decades of industry experience.

Translate, Don’t Just Shrink

At first glance, standard data compression might seem like the obvious solution to the token explosion problem. However, tools like gzip achieve a compression factor of only about two on raw capture files. This limitation exists because the vast majority of modern internet traffic is heavily encrypted, and encrypted data is already compressed and structured as densely as information can theoretically get.

What a network analyst actually needs during an investigation is not a microscopic view of every encrypted byte, but rather the overarching narrative of each conversation: who initiated communication, who they spoke with, the exact timing and duration of the exchange, the volume of data flowing in each direction, whether the underlying connection successfully maintained encryption, and whether any abnormal disruptions occurred along the way.

PCAP-LM preserves this high-level narrative while deliberately discarding the superfluous details. The system rewrites each individual network conversation as a compact, elegant line of symbols, bearing a distinct resemblance to musical notation. In this framework, every packet is reduced to a concise mark denoting its direction, protocol type, payload size class, and relative timing. These atomic indicators are known as "PacketGlyphs."

To ensure general-purpose models can interpret the notation without requiring specialized training or fine-tuning, a short legend is included at the top of the document. Highly repetitive stretches of traffic—such as the thousands of near-identical data packets generated during a large file download—are intelligently folded into a single run-length encoding statement indicating a specific repeating pattern. Furthermore, frequent symbol combinations are merged into composite tokens using a constrained variant of Byte-Pair Encoding. In optimal scenarios, a massive download spanning 14,000 distinct packets collapses neatly into a single line of text. Across real-world captures featuring variable sizes and irregular timing, a typical PCAP-LM document stabilizes at around 100 kilobytes of text.

Putting PCAP-LM to the Test

The research team rigorously evaluated PCAP-LM using real-world traffic captures gathered during a comprehensive 5G and 4G measurement campaign. The results demonstrated a dramatic reduction in data footprint. On average, the rewritten files were approximately 800 times smaller than the standard, highly detailed text output generated by Wireshark’s command-line utility, allowing every single capture to fit comfortably within the context window of a modern AI model.

However, a drastically smaller file size is only valuable if the underlying information remains thoroughly accurate. To test data fidelity, the researchers presented a state-of-the-art AI model with a battery of ten foundational forensic questions across 30 previously unseen captures. The queries probed key network characteristics, including the total number of distinct conversations, session durations, packet counts, the presence of connection resets or retransmissions, encryption status, and the specific services contacted.

PCAP-LM: Helping AI assistants read network traffic | APNIC Blog

Equipped with the PCAP-LM representation, the AI model achieved an impressive 99.3 percent accuracy rate. Conversely, when fed conventional text output that had been aggressively truncated to match the exact length of the PCAP-LM document, the model managed a mere 51 percent accuracy.

This stark performance gap highlights a crucial operational lesson. Because the conventional input was heavily truncated due to strict token limits, the model could only observe the first few dozen packets of each capture, rendering systemic questions regarding total duration or aggregate packet counts entirely unanswerable. The experiment demonstrates that a holistic summary covering an entire capture is significantly more effective for macro-level diagnostics than a hyper-detailed view restricted to a tiny fraction of the data.

Understanding System Limitations

Despite its high efficiency, the PCAP-LM approach involves deliberate trade-offs. The framework purposefully omits raw payload contents, precise TCP sequence numbers, and exact microsecond timings. Furthermore, basic heuristic checks within the current model failed to catch retransmissions in roughly a quarter of the affected test captures, all of which originated from traffic routed behind an intermediate proxy.

To mitigate these inherent limitations without sacrificing performance, each generated PCAP-LM document retains a specialized index pointer linking back to the original source packets. This structural design allows either an autonomous LLM or a human network engineer to seamlessly retrieve full packet details whenever an anomaly or suspicious pattern is detected in the summary view.

Current testing has also been primarily confined to a specific category of traffic: large, encrypted downloads originating from a single test server. Expanding the framework to encompass diverse protocols, complex enterprise applications, and chaotic mixed environments will likely require the introduction of additional symbolic vocabulary and potential retraining of the compression parameters. Crucially, PCAP-LM never attempts to decrypt traffic; encrypted payloads remain securely opaque by design, preserving user privacy and regulatory compliance.

Practical Implications for Network Operators

The broader significance of PCAP-LM extends far beyond academic experimentation, touching directly on how modern network operators can safely and practically deploy artificial intelligence within production environments. Integrating AI into network monitoring has historically been fraught with friction, balancing the prohibitive costs of processing massive data volumes against the strict security constraints required to protect sensitive infrastructure.

By shrinking multi-megabyte packet captures down to human- and machine-readable text documents that consume a fraction of the computational overhead, PCAP-LM dramatically lowers the financial and technical barriers to AI-driven network analysis. Operators can run continuous diagnostic workflows without overwhelming local hardware resources or exceeding the token processing limits of commercial language models.

This development opens up a broader philosophical discussion within the telecommunications engineering community regarding how artificial intelligence should interface with raw network data. Standard command-line-driven autonomous agents frequently excel at executing targeted, narrow queries, yet they routinely suffer from tunnel vision, missing broader systemic issues unfolding across the wider network topology. PCAP-LM directly counters this limitation by exposing the holistic behavioral patterns of entire conversations at a glance, proving that a well-structured macro summary can often reveal insights that micro-level inspection misses entirely.

As the industry continues to explore the convergence of artificial intelligence and network architecture, researchers are encouraging network operators to test these hybrid methodologies against their own operational challenges. By combining holistic AI summaries with side-indices that permit targeted raw packet retrieval when anomalies arise, engineering teams can build resilient, efficient diagnostic systems capable of catching unknown network anomalies faster than ever before.