Chamelio: A chameleonic network stack that adapts to every tenant

Cloud providers strive to maximize resource usage by sharing infrastructure across tenants, while applications increasingly demand transport protocols tailored to their workloads. Whether deploying domain-specific congestion control, QUIC variants, or Remote Direct Memory Access (RDMA)-over-Ethernet, tenants need the flexibility to customize network behaviour to meet stringent performance requirements. Yet today's deployment model enables such customization only at the expense of resource efficiency.

The status quo: Siloed network stacks

The most popular deployment model for cloud applications is to rent virtual machines (VMs). VMs enable cloud providers to run workloads from multiple tenants on the same physical machine without compromising isolation. Tenants can specialize transport protocols either by modifying the guest operating system or by building tailored userspace network stacks, all while the virtualized environment provides strong isolation boundaries between tenants.

VMs are typically statically provisioned with dedicated cores to prevent performance interference between tenants, creating silos of stranded resources. For example, a VM provisioned to sustain peak network activity leaves much of its allocated CPU capacity idle during periods of low network demand. This example illustrates a trade-off. Siloed per-VM network stacks enable safe per-tenant customization and isolation, but sacrifice resource efficiency.

On the other hand, a shared datapath between VMs consolidates VM networking into a single datapath instance, offering efficiency benefits, but opens the possibility of noisy neighbour interference between tenants and limits customizability, because tenants cannot freely upload custom implementations into the shared datapath instance on the host.

Reconciling customization, isolation, and resource efficiency

Chamelio is a provider-managed network stack for cloud VMs that runs in userspace on the host and allows tenants to upload custom transport protocols to a shared datapath. It achieves this using Extended Berkeley Packet Filter (eBPF), which provides a safe, verifiable execution environment for extending the host datapath without compromising provider control.

Each uploaded protocol includes eBPF bytecode for three fast-path handlers that capture the common-case processing for packet reception, scheduler-triggered send, and dequeue events (, , and ). A control plane uses userspace eBPF to verify the safety of uploaded protocols, allowing the handlers to execute safely on the host. Complex or state-heavy logic remains in a VM-resident slow path. Fine-grained runtime cycle accounting enforces fair sharing of the datapath to prevent performance interference between co-located tenants.

The Chamelio shared datapath improves per-core throughput by 1.96× relative to siloed network stacks and reaches 9.2 mega-requests per second(Mreq/s) with a custom Transport Control Protocol (TCP) implementation. It matches the performance of highly optimized hand-tuned Data Plane Development Kit (DPDK) stacks, while maintaining a 99th-percentile round-trip-time (RTT) of 26 µs.

Enabling full transport protocol customization with eBPF

Users write eBPF code for , , and handlers. The handler parses incoming packets and maps them to protocol state, constructs outgoing packets using state retrieved from shared memory, and processes messages from the slow path and applications. These handlers capture the latency-critical common case across network protocols because they align with the three events that drive packet processing: Packet arrival, the scheduler releasing staged work, and the delivery of work from the application or slow path.

Reducing customization overheads

eBPF introduces overheads such as dispatch, validation, and boundary crossings, which can quickly dominate per-packet costs. Furthermore, eBPF programs are compiled independently from the underlying network stack. This separation makes the eBPF program opaque to the compiler and forgoes optimization opportunities across boundaries in the eBPF program and fixed network datapath that are typically applied to conventionally compiled code.

Chamelio therefore converts the uploaded handler bytecode to LLVM Intermediate Representation (IR) instead of immediately lowering it to machine code. The control plane links the IR with precompiled infrastructure modules (like Address Resolution Protocol handling and network virtualization), helper implementations, and the IR of active guest protocols from co-resident VMs in the same shared fast path.

The result is one aggregate module representing the full fast path rather than opaque callbacks. Chamelio also internalizes infrastructure, helper, and eBPF definitions across the aggregate module before machine code generation. This makes those functions local to the module, allowing the compiler to inline across tenant/provider and tenant/tenant boundaries and eliminate now-redundant definitions.

Together, these optimizations collapse callback layers, propagate constants, and eliminate unnecessary stores to shared state and subsequent reloads across the combined fast path. This joint compilation of infrastructure code and custom-defined eBPF handlers cuts the eBPF programmability tax from 23.9% to 3.8%.

Enforcing performance isolation

The eBPF verifier enforces the safety of tenant-supplied protocol code, but safe code can still become a noisy neighbour by consuming excessive CPU cycles or triggering behaviour that degrades the performance of other tenants. Existing mechanisms, such as loop termination checks and instruction limits, do not provide sufficient isolation. Programs invoke helper functions whose costs are not accounted for during verification, and worst-case execution time estimates derived from control-flow analysis remain imprecise. Admission enforcement alone cannot guarantee performance isolation and must therefore be complemented by runtime enforcement.

Chamelio enforces performance isolation by allocating each guest a token budget, where each token represents one timestamp counter (TSC) cycle in the fast path. The fast path measures cycles spent executing each protocol's handlers by reading the TSC before and after execution and charges the elapsed time to the protocol's budget. When a protocol exhausts its budget, the datapath skips it in subsequent polling rounds. The control plane later refills its credits, after which processing resumes. This mechanism bounds the work each tenant can perform in the datapath and prevents any single protocol from causing unbounded interference.

Conclusion

Chamelio shows that cloud operators no longer need to choose between the efficiency of shared datapaths and tenant-defined transport protocols. It is time to rethink the assumption that customization requires siloed network stacks. eBPF-based extensibility and fine-grained runtime isolation make shared datapaths a practical path towards more efficient cloud networks.

Matheus Stolet is a PhD student at the Max Planck Institute for Software Systems, focused on designing datacentre network stacks that deliver better performance, efficiency, and isolation.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.