Enabling Fault-Tolerant Multicast in Cloud-Native Architectures: Bridging On-Prem and Multi-Cloud Environments

Enabling Fault-Tolerant Multicast in Cloud-Native Architectures: Bridging On-Prem and Multi-Cloud Environments Date : July 14th , 2025 Tejas Gajjar Technical Lead & Principal Architect Macy's Inc. | Cloud | Infrastructure and Platform Engineering IEEE Senior Member | BCS Fellow Collaborators swXtch.io, Oracle Cloud Infrastructure (OCI), Azure Networking Team Abstract Cloud providers lack native support for multicast, limiting deployment of high-availability applications that depend on real-time coordination and consistent data propagation. This white paper presents a solution for enabling software-defined, fault-tolerant multicast communication in Oracle Cloud Infrastructure (OCI) and multi-cloud environments using swXtch.io's cloudSwXtch platform. It covers technical challenges, overlay architecture, license management, NSG troubleshooting, and validation in real-world use cases. The solution supports sub-millisecond synchronization, seamless failover, and elastic scalability across Availability Domains (ADs), enabling enterprises to modernize legacy data center multicast patterns in the cloud. 1. Introduction Multicast communication is essential for applications like distributed databases, real-time analytics, trading systems, and event coordination tools. However, cloud platforms lack support for foundational multicast protocols [8][9][10]. This prevents businesses from migrating real-time systems to the cloud without significant architecture changes. To solve this, we developed and tested a multicast overlay using swXtch.io's, cloudSwXtch platform [1]. This solution brings resilient, high-performance multicast support to public cloud environments using enhanced UDP, time synchronization, and intelligent routing. 2. Business Need More and more, industries like financial services, logistics, media, and AI need systems that are low-latency and fault-tolerant. Because of the limits of the cloud, alternatives like TCP unicast replication have to be used, which slows down performance and adds extra work. The goal of this project was to make multicast replication possible in OCI's Ashburn region, with automatic failover, flexibility, and compatibility across availability domains. 3. Technical Challenges Cloud-based multicast implementation faces multiple hurdles: • Lack of native IGMP/MLD protocol support in cloud networking [8][9]. • Unreliable UDP transport layer and dynamic packet routing. • Clock synchronization gap across availability zones [3]. • NSG (Network Security Group) misconfiguration [13] and inconsistent logging in OCI. • Complex overlay membership and routing table management. • Ensuring seamless failover with minimal latency. 4. Overlay-Based Multicast Solution 4.1 Architecture The multicast overlay uses swXtch.io nodes with dual virtual NICs, forming a distributed mesh. Enhanced UDP transport supports packet reliability, while PTP ensures sub-millisecond synchronization [3]. IGMP logic is embedded in the overlay, making multicast joins/leaves dynamic. My Role: • Designed and implemented the multi-cloud orchestration framework using Terraform and Helm [6][7]. • Architected deployment across OCI, Azure, and GCP with consistent overlay behavior. • Created self-healing, fault-tolerant routes with < 50ms failover recovery [2]. 4.2 Membership & Routing Overlay nodes natively propagate multicast group joins and leaves, eliminating the need for tunneling. Custom routing logic ensures deterministic delivery across cloud regions and allows dynamic scale-out. 4.3 Fault Tolerance & Synchronization Using SDN Fast Failover techniques and PTP-based synchronization, the system ensures both network resilience and strict event ordering. Contribution: • Developed failover detection logic that reduced node recovery time by 50%. • Authored auto-synchronization routines to dynamically maintain group consistency. • Integrated Prometheus and Grafana for full-stack monitoring and observability [4][5]. 5. Implementation in Oracle Cloud Infrastructure (OCI) 5.1 swXtch.io VM Deployment & Configuration • Deployed swXtch.io overlay nodes with dual vNICs and updated routing using netplan. • Aligned MAC and IP settings to ensure multicast consistency. • Used iptables and static routing to accept multicast from X.X.X.0/21. My Role: • Led the design of overlay deployment strategy in OCI. • Troubleshot complex routing and firewall issues across subnets. 5.2 License Provisioning and Service Restart SOP A validated procedure was created for safely applying licenses without causing disruption: # bash # Validate fingerprint curl -s http:// X.X.X.Y /top/dashboard | grep -m 2 -Eo '"fingerprint"[^,]*' | head -1 # Backup and apply for license sudo cp /swxtch/license.json /swxtch/license.json.$(date +%Y%m%d%H%M%S) sudo cp NEWLICENSEfile /swxtch/license.json # Restart services sudo systemctl restart swxtch-ctrl.service swxtch-repl.service Contribution: • Authored and validated the license update SOP. • Ensured zero-downtime upgrades through controlled validation. 5.3 NSG Troubleshooting and Resolution Issue: A UDP packet drop between overlay nodes occurred due to an incorrect NSG (Network Security Group) source CIDR configuration. Cloud flow logs misleadingly displayed an "ACCEPT" action despite actual packet drops[13]. Original NSG Configuration: X.X.X.X/22 Failing Source IP: X.X.X.Y (outside of the configured CIDR block) Root Cause: The NSG source rule did not fully cover all relevant subnets in the non-production VCN, leading to silent packet drops during overlay node communication. Resolution: Expanded the NSG ingress rule to X.X.X.X/21, which encompassed all required subnets. This update immediately restored connectivity and resolved the UDP drop issue between overlay nodes. Fig:1 NSG troubleshooting Workflow Additional Insight: The inconsistency between NSG flow logs and actual packet routing behavior was identified as a misdirection point in troubleshooting. This issue was escalated as a product-level feedback item to the cloud provider for improved diagnostic transparency. 6. Case Study: Distributed Consensus with Multicast Replication 6.1 Use Case A globally distributed database cluster implementing Paxos required multicast to replicate state rapidly across nodes. 6.2 Integration • Deployed swXtch.io nodes alongside database nodes. • Enabled IGMP/FEC support with application layer validation. 6.3 Results Metric Baseline With Overlay Synchronization latency 18 ms 2.4 ms Failover response time ~120 ms <50 ms Packet loss rate 0.1% <0.01% Cluster scale-out 1x 3x without config change Contribution: • Led the design and live multi-cloud validation of the database overlay integration. • Benchmarked and optimized performance, influencing future database architecture. 7. Evaluation & Results • Performance: Sub-millisecond propagation verified across regions. • Observability: Prometheus/Grafana stack monitored group health and performance. • Resilience: Packet loss under 0.01%; seamless failover in failure simulation. 8. Industry Impact and Innovation 8.1 Bringing Multicast into the Cloud-Native Era Traditionally, multicast communication has been confined to on-premises environments due to its reliance on Layer-2 and Layer-3 protocols like IGMP, MLD, and PIM [8][9][10]. These protocols are not natively supported by most public cloud providers, which has hindered the migration of real-time, event-driven applications to cloud platforms like Oracle Cloud Infrastructure (OCI), Microsoft Azure, and Google Cloud Platform. The multicast overlay solution introduced in this white paper, built on swXtch.io's cloudSwXtch platform [1][13], directly addresses this limitation [1]. It enables lossless, fault-tolerant multicast communication over software-defined networks in cloud and hybrid environments. This effectively extends multicast capabilities into the cloud-native era, making it possible to run latency-sensitive, distributed applications that were once considered cloud-incompatible. 8.2 Broad Industry Applicability The ability to support multicast in cloud environments opens the door to significant benefits across multiple industries: Financial Services Facilitates the distribution of real-time market data and order book synchronization using protocols such as FIX and FAST. Reduces end-to-end data transmission latency from tens of milliseconds to sub-3ms, enhancing competitiveness in high-frequency trading [10][21]. Retail and Logistics Supports distributed inventory synchronization, warehouse automation, and edge-device communication. Real-time updates for promotions and loyalty programs across geographically dispersed point-of-sale systems improve customer experience and operational efficiency [13]. Media and Broadcasting Enables multicast replication for IPTV and live streaming. Integrates with cloud-based ingest, encoding, and distribution pipelines using overlay networking, without the need for physical broadcast infrastructure [10][1][13]. Industrial IoT and Smart Manufacturing Powers event-driven propagation in MES/SCADA systems. Ensures accurate, real-time telemetry sharing across production zones or plants (Grafana, n.d.; Prometheus, n.d.). Cloud AI and Digital Twins Facilitates low-latency model parameter exchange for collaborative AI training or federated learning systems. Provides real-time broadcasting of simulation state for use cases such as metaverse development and smart infrastructure monitoring (Shawish & Salama, 2014). 8.3 Key Technical Contributions This work introduces several original engineering enhancements: Engineering Advancements Implements reliable multicast over UDP using Forward Error Correction (FEC) and redundant overlay paths (swXtch.io, n.d.). Achieves sub-millisecond synchronization using Precision Time Protocol (PTP), critical for maintaining order in event-driven architectures (Berkeley AutoLab, n.d.). Introduces auto-healing group orchestration to dynamically manage joins/leaves across distributed nodes. Utilizes SDN-based fast failover strategies to maintain event propagation with under-50ms recovery (arXiv, 2022). Observability and Monitoring Integrates Prometheus and Grafana for real-time metrics and alerting, ensuring visibility into group health and packet metrics (Prometheus, n.d.; Grafana, n.d.). Enables per-group and per-node telemetry for fault isolation and performance tuning. DevOps and Infrastructure-as-Code Deploys infrastructure seamlessly across OCI, Azure, and GCP using Terraform and Helm charts (Helm, n.d.; GitHub Actions, n.d.). Automates license provisioning and NSG configuration for scalable, reproducible rollouts (Azure, n.d.; Oracle, n.d.). Standards Readiness Supports future collaboration with IETF's MBONED and INTAREA working groups to standardize multicast-over-cloud architectures (IETF MBONED, n.d.). Proposes encapsulation models aligned with RFC 7348 (VXLAN) and RFC 5110 for compatibility across multi-vendor environments (IETF, 2008; IETF, 2007). 8.4 Strategic Impact This multicast overlay solution offers a transformative pathway for enterprises: Accelerates Cloud Adoption: Makes it feasible to lift-and-shift mission-critical multicast-dependent systems. Improves Resilience: Achieves fault tolerance through auto-failover and self-healing overlays. Enhances Performance Predictability: Ensures application responsiveness by maintaining low latency and consistent packet delivery. Fosters Ecosystem Collaboration: Encourages cross-industry standardization and interoperability through active participation in IETF and open-source communities. 8.5 Adoption Readiness and Competitive Advantage Performance benchmarks during real-world OCI deployments confirm the solution's robustness: Latency improved by up to 86% Failover recovery time cut by up to 60% Packet loss reduced to less than 0.01% These gains make the multicast overlay ready for enterprise production environments and position it as a reference model for cloud-native multicast implementations. For vendors and platforms, adopting this architecture offers a strategic advantage in delivering reliable, real-time communication at a scale. 9. Future Work 9.1 Support Layer-2 Ethernet multicast extensions While the current multicast overlay solution enables reliable Layer-3 (IP-based) multicast replication across cloud-native environments, many enterprise and industrial applications require support for Layer-2 Ethernet multicast, which operates at the data link layer using MAC-level addressing (e.g., 01:00:5e:xx:xx:xx) [11]. These Layer-2 multicast flows are essential for: • Real-time industrial control systems • Multimedia broadcast (e.g., IPTV, live video feeds) • High-frequency trading systems with low-latency messaging • Storage area network (SAN) protocols such as FCoE or iSCSI extensions Current Limitation: Public cloud platforms do not provide native support for Layer-2 networking constructs such as Ethernet broadcast domains or L2 multicast propagation. This prevents traditional multicast applications (e.g., those using Ethernet frames without IP encapsulation) from functioning as-is in the cloud. Objective: To extend the current overlay beyond Layer-3 to emulate Layer-2 multicast capabilities in cloud-native environments, thus supporting legacy Ethernet-based applications without redesigning their network stack. Planned Enhancements: 1. Encapsulation Support: Introduce support for Ethernet-over-UDP encapsulation (e.g., using VXLAN or Geneve) to tunnel L2 multicast frames over IP-based cloud networks [11]. 2. MAC Address Grouping: Implement multicast MAC group membership management in swXtch.io nodes, mimicking switch-level MAC learning and IGMP snooping behavior. 3. Broadcast Domain Simulation: Emulate broadcast domains using overlay VLAN tagging or tenant-specific L2 overlays, allowing isolated multicast segments within shared infrastructure [11][12]. 4. Switch-Level Replication Logic: Integrate replication engines within swXtch.io control plane to support one-to-many frame forwarding using software-defined multicast tree construction. 5. Performance Optimization: Optimize packet forwarding paths to avoid kernel-mode bottlenecks in virtual NICs and reduce encapsulation overhead through zero-copy mechanisms or DPDK integration. 6. Standards Alignment: Align implementation with industry standards like IEEE 802.1Q (VLANs) and RFC 7348 (VXLAN) to ensure compatibility with on-prem switches and edge devices [11]. Impact and Value: • Enables seamless migration of on-premises Layer-2 multicast workloads to the cloud without architectural changes. • Expands applicability of swXtch.io's platform to broadcast-heavy and latency-sensitive domains. • Bridges legacy and modern cloud-native systems under a unified overlay infrastructure. 9.2 Publish OCI/Azure integration Helm charts Helm is the de facto package manager for Kubernetes, enabling users to define, install, and upgrade complex applications via reusable charts. While Helm charts are widely available for AWS and GCP ecosystems, support for Oracle Cloud Infrastructure (OCI) and Azure in the context of multicast overlays and swXtch.io integration is sparse or ad hoc. Given that cloudSwXtch-based overlays are critical to enabling multicast in cloud-native systems, publishing official, production-ready Helm charts for OCI and Azure will accelerate adoption, standardize deployments, and ensure repeatability. Objective: To design, validate, and publish open and customizable Helm charts for seamless deployment of swXtch.io multicast overlays in Kubernetes environments running on Oracle Cloud Infrastructure and Microsoft Azure. Fig:2 Helm Deployment pipeline for OCI/Azure swXtch.io Overlay Planned Features of the Helm Charts 1. Multi-AZ/AD Awareness o Support for deploying swXtch.io overlay nodes in multiple Availability Domains (ADs) or Availability Zones (AZs). o Region-specific configuration blocks for OCI and Azure. 2. Custom Networking Support o Define VCN/subnet IDs for OCI and VNET/subnet parameters for Azure. o Enable custom static routes, dual vNIC attachments, and multicast-specific iptables rules. 3. Secret Management & Licensing o Secure handling of swXtch.io license keys via Kubernetes secrets or HashiCorp Vault integration [14][15][16]. o Initialization scripts for license activation and validation. 4. Observability Integration o Out-of-the-box integration with Prometheus and Grafana dashboards for overlay node health, group joins/leaves, and packet metrics. o Optional Fluent Bit/Elastic integration for log aggregation. 5. Security Hardening o Predefined NetworkPolicies and NSG configurations for overlay communication. o Helm templating of OCI NSGs and Azure NSG rules. 6. Horizontal Scaling & Auto-Recovery o Native support for Kubernetes Horizontal Pod Autoscaler (HPA). o Optional support for deployment using StatefulSets for ordered multicast roles. Technical Deliverables • Chart.yaml with OCI/Azure-specific metadata • OCI Values file (values.oci.yaml) with: o tenancy, region, AD, VCN, subnet, NSG o swXtch.io image tags and license config • Azure Values file (values.azure.yaml) with: o subscription, resource group, VNET, subnet o swXtch.io node sizes, storage, and NIC configuration • CI/CD pipelines for chart validation and packaging using GitHub Actions or OCI DevOps [13][15]. Publishing & Community Impact • Publish Helm charts on ArtifactHub or GitHub public repo under a suitable OSS license. • Accompany with detailed README and deployment tutorials (with diagrams). • Include enterprise-ready Helm chart overlays for private deployments via GitOps or ArgoCD [13][16]. • Drive community feedback and collaboration for enhancements, e.g., L2 overlay extension or BGP peering support. Value Proposition • Simplifies multicast overlay deployment across OCI and Azure. • Reduces deployment time from hours to minutes with a single command. • Supports repeatable infrastructure-as-code practices for regulated industries. • Encourages open-source alignment and community engagement with swXtch.io architecture. 9.3 Collaborate with IETF to standardize multicast-over-cloud patterns The Internet Engineering Task Force (IETF) is the premier global standards body responsible for defining protocols that shape the internet and networking technologies. While multicast is well-established in traditional LAN/WAN environments through standards such as: • IGMP (RFC 2236/3376) for IPv4, • MLD (RFC 3810) for IPv6, and • PIM-SM (RFC 7761) for routing, these protocols were designed with on-premises and carrier-grade network infrastructures in mind-not for virtualized, isolated cloud environments offered by providers like AWS, Azure, GCP, or OCI. Currently, cloud-native networks do not support these multicast protocols due to: Security and tenant isolation concerns, Lack of native support for IGMP snooping or L2 multicast forwarding, Absence of a control plane for group management across VPC/VNet boundaries. Objective Establish an IETF working group or participate in an existing one (e.g., MBONED, INTAREA, or V6OPS) to help define a new multicast-over-cloud standard, which may include[18]. Multicast Overlay Transport Protocol (MOTP): A transport-agnostic protocol to support reliable multicast communication over cloud virtual networks using overlay nodes. Cloud Multicast Control Plane (CMCP): Define control mechanisms to simulate group membership, IGMP join/leave, and dynamic routing in environments that do not allow L2 broadcast. Security Model for Cloud Multicast: Formalize tenant isolation, authentication of multicast groups, and end-to-end encryption of one-to-many streams. Service Discovery and Auto-registration: Cloud-native multicast-aware applications should automatically discover groups, publishers, and subscribers within logical boundaries (e.g., Kubernetes namespaces, VNets, or VCNs). Proposed Contributions to the IETF Draft an Informational RFC or Internet-Draft (ID) describing the limitations of current multicast protocols in public cloud. Submit architectural recommendations based on real-world deployment experience with swXtch.io overlay in OCI, Azure, and GCP. Collaborate with cloud providers, SDN vendors, and edge networking researchers to define a reference model for multicast in cloud-native fabrics. Propose integration of this model into IETF multicast architecture documents, as an extension to or companion of RFC 5110 [12]. Strategic Benefits Industry Influence: Early contribution helps shape the direction of multicast standardization in cloud environments. Interoperability: Reduces fragmentation by establishing a unified approach that can work across cloud vendors and SDN overlays. Vendor Agnosticism: Creates standards that can be adopted by commercial solutions e.g., swXtch.io, Cisco Nexus Cloud, Arista CloudVision. Support for Next-Gen Use Cases: Enables scalable distribution for AI model synchronization, blockchain replication, digital twin simulations, and metaverse platforms all requiring real-time state propagation. Suggested Steps Forward Engage with existing IETF mailing lists such as mboned@ietf.org and int-area@ietf.org [18]. Present a draft at IETF meetings [10][12]. From a Birds of a Feather (BoF) session to discuss cross-cloud multicast overlay patterns. Partner with academia, hyperscalers, and enterprise network architects to gather feedback. Iterate on proposed standards and push toward Working Group adoption. 10. Conclusion By engineering and validating a multicast overlay across OCI and multi-cloud settings, this work resolves a long-standing gap in public cloud capabilities. The project demonstrates that real-time, lossless, and fault-tolerant communication is achievable in cloud-native architecture with proper tooling, orchestration, and technical leadership. References 1. swXtch.io Technical Documentation https://www.swxtch.io/documentation/ 2. SDN Fast Failover Research arXiv Preprint, 2022. https://arxiv.org/abs/2204.00212 3. Precision Time Protocol for Distributed Systems (PTP) Berkeley AutoLab Project. https://autolab.berkeley.edu/ 4. Prometheus Monitoring Toolkit https://prometheus.io/ 5. Grafana Observability Platform https://grafana.com/ 6. Kubernetes Helm https://helm.sh/ 7. Kubernetes Horizontal Pod Autoscaler (HPA) https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ 8. IETF RFC 2236 - Internet Group Management Protocol, Version 2 (IGMPv2) https://datatracker.ietf.org/doc/html/rfc2236 9. IETF RFC 3810 - Multicast Listener Discovery Version 2 (MLDv2) https://datatracker.ietf.org/doc/html/rfc3810 10. IETF RFC 7761 - Protocol Independent Multicast - Sparse Mode (PIM-SM) https://datatracker.ietf.org/doc/html/rfc7761 11. IETF RFC 7348 - Virtual eXtensible Local Area Network (VXLAN) https://datatracker.ietf.org/doc/html/rfc7348 12. IETF RFC 5110 - Overview of Multicast in MPLS/BGP IP VPNs https://datatracker.ietf.org/doc/html/rfc5110 13. Oracle Cloud Infrastructure Documentation - Networking, VCN, and NSG configuration https://docs.oracle.com/en-us/iaas/Content/home.htm 14. Azure Virtual Network and NSG Documentation https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview 15. GitHub Actions Documentation - CI/CD pipelines for Helm charts https://docs.github.com/en/actions 16. ArtifactHub - Helm Chart Repository https://artifacthub.io/ 17. Helm Best Practices Guide https://helm.sh/docs/chart_best_practices/ 18. MBONED IETF Working Group Charter https://datatracker.ietf.org/wg/mboned/about/ 19. arXiv. (2022). SDN Fast Failover Research. Retrieved from https://arxiv.org/abs/2204.00212 20. Azure. (n.d.). Network security groups overview. Retrieved from https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview 21. Berkeley AutoLab. (n.d.). Precision Time Protocol for Distributed Systems (PTP). Retrieved from https://autolab.berkeley.edu

Tejas Gajjar is a senior cloud infrastructure and platform engineering leader at Macy's Inc., with over 15 years of experience in enterprise integration, fault-tolerant architecture, and multi-cloud deployment. He is a Fellow of the British Computer Society (BCS) and a Senior Member of IEEE, recognized for his contributions to cloud-native engineering and technical mentorship. Tejas is an invited speaker at multiple U.S. tech conferences and serves as a judge for IEEE and 48in48.org global events. About swXtch.io swXtch.io is a pioneer in overlay networking solutions that enable multicast and high-performance data delivery in cloud-native environments. Their flagship product, cloudSwXtch, brings traditional multicast capabilities to public cloud platforms without requiring changes to application code. Learn more at www.swxtch.io. About Macy's Technology Macy's Inc. is one of the largest omnichannel retailers in the United States, backed by a robust technology division that powers e-commerce, data infrastructure, and real-time supply chain systems across its brands. The company's cloud-first transformation strategy leverages automation, containerization, and resilient platform engineering to deliver exceptional customer experiences.