Efficient Multi-WAN Transport for 5G with OTTER

In the ongoing cloudification of 5G, software network functions (NFs) are replacing fixed-function network hardware, allowing 5G network operators to leverage the benefits of cloud computing. The migration of NFs and their management to the cloud causes 5G traffic to traverse an operator’s wide-area network (WAN) to the cloud WAN that hosts the datacenters (DCs) running 5G NFs and applications. However, achieving end-to-end performance for 5G traffic across two WANs is hard. Placing 5G flows across two WANs with different performance and reliability characteristics, edge and DC resource constraints, and interference from other flows is different and more challenging than single-WAN traffic engineering. We address this challenge and show that orchestrating paths across a multi-WAN overlay 1 allows us to achieve average 13% more throughput, 15% less RTT, 45% less jitter, or reduce average loss from 0.06% to under 0.001%. We implement our multi-WAN 5G flow placement in a scalable optimization prototype that allocates 26%–45% more bytes on the network than greedy baselines while also satisfying the service demands of more flows.

在5G持续“云化”的进程中， 软件化的网络功能 (Network Functions, NFs) 正在取代功能固定的网络硬件， 这使得5G网络运营商能够利用云计算带来的优势。

然而，将网络功能及其管理迁移至云端，会导致5G流量需要穿越运营商的广域网 (WAN)，进入托管着运行5G网络功能和应用的数据中心 (DC) 的云广域网。

要在两个广域网之间为5G流量实现端到端的性能保障是十分困难的。与单广域网的流量工程相比，将5G流（flow）部署在两个具有不同性能和可靠性特征的广域网中，同时还需考虑边缘和数据中心的资源限制以及其他流的干扰，是一个截然不同且更具挑战性的问题。

我们致力于解决这一挑战，并证明通过在一个多广域网覆盖网络（multi-WAN overlay）上协同编排路径，我们能够平均提升13%的吞吐量，降低15%的往返时间 (RTT)，减少45%的抖动，并将平均丢包率从0.06%降低到0.001%以下。我们在一个可扩展的优化原型中实现了我们的多广域网5G流放置方案，该方案比贪心基线算法在网络上多分配了26%–45%的字节，同时满足了更多流量的服务需求。

Introduction

The majority of 5G access networks today use legacy telecom equipment and architecture. The next generation of 5G is designed to deliver superior Quality-of-Service (QoS) through a combination of new frequency bands, new radio technology, and the cloudification of telecom network functions (NFs). These innovations are expected to revolutionize 5G and subsequent generations of access networks. In this work we focus on the implications of improved QoS and cloudification of access on wide-area networking (WAN).

As part of the cloudification of 5G, monolithic and proprietary hardware implementations of mobile wireless NFs are evolving into disaggregated software-based NFs that run on commodity off-the-shelf compute [79]. This disaggregation allows 5G operators to deploy their virtualized radio access networks (vRAN) on edge compute close to end users, and packet cores in datacenters (DCs) [56,61].

Cloudification is driving performance-sensitive interdomain traffic onto cloud WANs. For instance, traffic from user equipment (UEs) traverses a 5G operator’s WAN to a cloud WAN to reach 5G Core NFs and applications hosted in the cloud [56, 61]. The burden of providing QoS to this traffic is now shared between two WANs: the 5G operator’s WAN and the cloud WAN. However, existing traffic management mechanisms lack the ability to orchestrate paths shared between two WANs. In fact, 5G networks are already bottlenecked on WAN performance [91], and this will get worse as the upcoming 5G New Radio (5G NR) unlocks ultra-low latency and high-bandwidth radio modes.

Improving the performance of inter-domain 5G traffic is challenging for three main reasons. First, cloudified access traffic is destined to cloud DCs that host 5G NFs and applications. The location of these 5G services in the cloud is itself dynamic [33,80] and conditional on the availability of compute resources, making it hard to achieve performanceoptimal routing. Second, the primary mechanism to achieve network performance goals in WANs today is intra-domain traffic engineering (TE) [39, 41, 44] which does not capture inter-domain route performance goals or fine-grained service objectives of 5G traffic. Third, the dynamic nature of 5G traffic requires on-demand flow placement, which is not achievable by traditional TE mechanisms that operate on periodic (e.g., 5 minute) allocation intervals.

当今大多数5G接入网络仍在使用传统的电信设备和架构。下一代5G旨在通过结合新的频段、新的无线电技术以及电信网络功能 (NF) 的云化，来提供卓越的服务质量 (QoS)。这些创新有望为5G及后续几代的接入网络带来革命性的变化。在本文中，我们专注于QoS的提升和接入网云化对广域网 (WAN) 产生的影响。

作为5G云化的一部分，原先一体化且专有的移动无线网络功能硬件实现，正在演变为在商用现成品（commodity off-the-shelf）计算设备上运行的解耦式、基于软件的网络功能 [79]。这种解耦使得5G运营商可以将其虚拟化无线接入网络 (vRAN) 部署在靠近终端用户的边缘计算节点上，并将分组核心网部署在数据中心 (DC) 中 [56, 61]。

云化正在将对性能敏感的域间流量推向云广域网。例如，来自用户设备 (UE) 的流量需要穿越5G运营商的广域网，再进入云广域网，才能到达托管在云端的5G核心网功能和应用程序 [56, 61]。 因此，为这些流量提供QoS的责任现在由两个广域网共同承担：5G运营商的广域网和云广域网。

然而，现有的流量管理机制缺乏在两个共享的广域网之间进行路径协同编排的能力。事实上，5G网络已经面临广域网性能的瓶颈 [91]，而随着即将到来的5G新空口 (5G NR) 解锁超低延迟和高带宽的无线模式，这一问题将变得更加严重。

提升域间5G流量的性能之所以充满挑战，主要有三个原因。

首先，云化后的接入流量其 目的地是托管5G网络功能和应用的云数据中心。这些5G服务在云中的位置本身就是动态的 [33, 80]，并且取决于计算资源的可用性，这使得实现性能最优的路由变得困难。

其次，当今在广域网中实现网络性能目标的主要机制是域内流量工程 (TE) [39, 41, 44]，这种机制无法捕捉到域间的路由性能目标或5G流量细粒度的服务目标。

第三，5G流量的动态性需要按需进行流放置，而传统的流量工程机制通常以周期性（例如，5分钟）的分配间隔运行，无法满足这一要求。

We develop OTTER (Overlay Traffic Transport and Efficient Resource allocation), a system that provides efficient, on-demand transport across operator and cloud WANs for 5G traffic with differing QoS needs. It uses a multi-WAN overlay to co-optimize the placement of 5G NFs and applications, and the network paths of 5G traffic. OTTER dynamically computes optimal routes for multiple performance metrics and implements them using scalable forwarding mechanisms involving VMs and cloud routing gateways. OTTER is cloud-agnostic, allowing operators to deploy it on cloud providers of their choice.

Our contributions. OTTER makes two key contributions. First, it develops an efficient algorithm for allocating network and compute resources to 5G demands across operator and cloud WANs. This algorithm forms the core of OTTER ’s multi-WAN SDN (Software Defined Networking) controller. Second, OTTER implements the optimal routing and compute placement strategy of the controller using a novel multi-WAN forwarding mechanism. We develop the multiWAN forwarding mechanism as part of the OTTER Orchestrator, which programmatically creates multi-WAN overlay networks with fine-grained routing policies for operators to deploy their 5G NFs.

We design OTTER with the observation that the operator and cloud have newly aligned economic incentives for 5G, as their business relationship goes beyond peering to also include SaaS and compute. The OTTER Orchestrator scales to country-wide sizes of large operators by leveraging cloudnative support. The OTTER Optimizer dynamically places both compute and network workloads on multi-WAN overlays, for multiple traffic classes simultaneously.

我们开发了 OTTER (Overlay Traffic Transport and Efficient Resource allocation)，这是一个为具有不同QoS需求的5G流量在运营商和云广域网之间提供高效、按需传输的系统。它使用一个多广域网覆盖网络来协同优化5G网络功能和应用的部署位置，以及5G流量的网络路径。OTTER能为多种性能指标动态计算最优路由，并通过涉及虚拟机 (VM) 和云路由网关的可扩展转发机制来实现这些路由。OTTER是云无关的，允许运营商在他们选择的任何云提供商上部署。

我们的贡献。 OTTER做出了两大关键贡献。

首先，它为跨运营商和云广域网分配5G需求的网络和计算资源开发了一种高效算法。该算法构成了OTTER多广域网软件定义网络 (SDN) 控制器的核心。

其次，OTTER通过一种新颖的多广域网转发机制来执行控制器计算出的最优路由和计算资源放置策略。我们开发了这种多广域网转发机制，作为OTTER编排器 (Orchestrator) 的一部分，它能以编程方式创建具有细粒度路由策略的多广域网覆盖网络，供运营商部署其5G网络功能。

我们设计OTTER的观察基础是： 运营商和云服务商在5G领域形成了新的一致性经济激励，因为他们的商业关系已超越对等互联 (peering)，还包括了软件即服务 (SaaS) 和计算服务。

OTTER编排器利用云原生支持，可以扩展到大型运营商的国家级网络规模。

OTTER优化器 (Optimizer) 能够在多广域网覆盖网络上，同时为多种流量类别动态地放置计算和网络工作负载。

Results. OTTER’s path orchestration across two commercial WANs outperforms both public paths as well as topologies designed for enterprise-grade privacy and security. Our deployment topology scaled out to the continental US shows 13% higher throughput on average, with a best case of 136% higher, which can be 6-10 Gbps more. For flows that care about round trip time (RTT), we achieve 15% average reduction, up to 42 ms less. Jitter-sensitive flows can expect an average reduction of 45% and loss is 0.06% less on average. OTTER’s on-demand allocation outperforms greedy baselines by up to 26-45% in allocated bytes (the amount of demand that is satisfied), and comes close to what an infeasible, infinitely-fast optimizer can achieve.

结果。 OTTER在两个商业广域网上进行的路径编排，其性能优于公共路径以及专为企业级隐私和安全设计的拓扑。我们将部署拓扑扩展到整个美国大陆，结果显示吞吐量平均提升13%，最佳情况下提升136%，这可能意味着增加6-10 Gbps的带宽。对于关注往返时间 (RTT) 的流，我们实现了平均15%的降低，最多可减少42毫秒。对抖动敏感的流，平均抖动降低了45%，而丢包率平均降低了0.06%（原文中为减少0.06%，应指绝对值的降低，例如从0.06x%降至0.00x%）。OTTER的按需分配方案与贪心基线相比，在已分配字节数（即已满足的需求量）上高出26-45%，并且其结果接近一个理论上无法实现、速度无限快的优化器所能达到的水平。

Cloudification of Access

Traditionally, vendors have implemented cellular technology on monolithic and proprietary hardware. However, 5G cellular networks are being re-architected using SDN and NF virtualization (NFV) [9]. This new architecture enables operators to scale up and out NFs on statistically multiplexed compute at operator edge sites (at or near cellular base stations and mobile switching centers), cloud edge sites (at hosting centers or peering points), and cloud DCs [55]. The use of the same compute management plane across all three allows NFs and interactive applications (e.g., AR/VR) to be deployed flexibly, as user demand dictates. This allows operators to unlock the elasticity and efficiency of cloud computing. This convergence of access networks and hyperscalers is known as the cloudification of 5G [27,64,65].

传统上，蜂窝技术由供应商在一体化且专有的硬件上实现。然而，5G蜂窝网络正在使用SDN和网络功能虚拟化 (NFV) [9] 进行重新架构。

这种新架构使运营商能够在其 边缘站点 (位于蜂窝基站和移动交换中心或其附近), 云边缘站点 (位于托管中心或对等互联点) 以及 云数据中心 [55] 的统计复用计算资源上，对网络功能进行纵向和横向扩展。

在所有这三类位置使用相同的计算管理平面，使得网络功能和交互式应用（例如AR/VR）可以根据用户需求进行灵活部署。这使得运营商能够释放云计算的弹性和效率。这种接入网与超大规模云服务商的融合被称为5G的云化 [27, 64, 65]。

2.1 Implications for WANs

Performance-sensitive demands over multiple WANs. The deployment of 5G applications and NFs on the cloud puts the cloud WAN on the critical path of 5G traffic that originates in the operator WAN, as shown in Figure 1. Extremely performance-sensitive NFs, such as RAN (Radio Access Network) NFs are hosted in operator far edges [37,64], close to cell sites. Other NFs, such as the 5G Core, are hosted in the cloud, in its edges and/or DCs. For example, Nokia’s 5G Core runs on the AWS cloud [61], Microsoft’s 5G Core runs on both on-prem servers and Azure DCs [56], and its voicemail NF is hosted on Azure DCs [52]. Multiple operators have already moved parts of their 5G infrastructure to the public cloud. AT&T is using Azure [8, 60]. Verizon is using AWS [12], leveraging multiple cloud technologies including Amazon EKS, AWS Wavelength, and HashiCorp Consul. The operator and cloud WANs peer [69] directly in multiple locations. An edge or DC can serve traffic flows from operator cell sites depending on where the destination NF or application is hosted [55]. Applications may include operator applications such as phone storage backup, private enterprise applications such as industrial AR/VR, and consumer applications such as game streaming.

跨多个广域网的性能敏感型需求。 将5G应用和网络功能部署在云端，使得云广域网处于5G流量的关键路径上，这些流量源自运营商的广域网 ，如图1所示。

对性能极其敏感的网络功能，如无线接入网 (RAN) 的网络功能，被托管在运营商的远边缘 [37, 64]，靠近蜂窝基站。其他网络功能，如5G核心网，则托管在云端的边缘和/或数据中心:

例如，诺基亚的5G核心网运行在AWS云上 [61]，微软的5G核心网同时运行在本地服务器和Azure数据中心 [56]，其语音信箱网络功能则托管在Azure数据中心 [52]。

多家运营商已经将其部分5G基础设施迁移到公有云:

AT&T正在使用Azure [8, 60]。Verizon正在使用AWS [12]，利用了包括Amazon EKS、AWS Wavelength和HashiCorp Consul在内的多种云技术。

运营商和云广域网在多个地点直接进行对等互联 [69]。根据目标网络功能或应用的托管位置，一个边缘站点或数据中心可以服务来自运营商蜂窝基站的流量流 [55]。这些应用可能包括运营商的应用（如手机存储备份）、私有企业应用（如工业AR/VR）以及消费者应用（如游戏直播）。

Heterogenous network demands. 5G NFs have different performance requirements. For example, some NFs in the vRAN have stringent RTT, loss, and jitter requirements, while others such as UPF (User Plane Function) in the 5G Core need high throughput, and voicemail is performanceinsensitive. Furthermore, upcoming releases such as 5G NR enable new applications, each with diverse performance requirements. For example, interactive AR/VR requires sub20 ms network RTT for acceptable user experience [79, 91], beyond 4K video streaming needs 100+ Mbps throughput, and remote surgery demands high reliability. The 5G specification supports these diverse requirements through multiple radio-layer modes [65, 67, 79, 84], including: (1) Ultrareliable low latency communication (URLLC) with ∼1 ms latency and 99.999% reliability, (2) Enhanced mobile broadband (eMBB) with multi-Gbps data rates, and (3) Massive machine-type communication (mMTC) for ultra-low energy communication at scales of up to 1 million nodes per KM 2 .

异构网络需求。 5G网络功能有不同的性能要求。

例如，虚拟化无线接入网 (vRAN) 中的一些网络功能有严格的RTT、丢包和抖动要求，而5G核心网中的用户平面功能 (UPF) 等则需要高吞吐量，语音信箱则对性能不敏感。

此外，即将发布的标准如5G新空口 (NR) 将催生具有多样化性能要求的新应用。例如，交互式AR/VR需要低于20毫秒的网络RTT才能获得可接受的用户体验 [79, 91]，超4K视频流需要100+ Mbps的吞吐量，而远程手术则要求极高的可靠性。

5G规范通过多种无线层模式来支持这些多样化的需求 [65, 67, 79, 84]，包括：

(1) 超可靠低延迟通信 (URLLC)，延迟约1毫秒，可靠性达99.999%

(2) 增强移动宽带 (eMBB)，数据速率达数Gbps

(3) 海量机器类通信 (mMTC)，用于每平方公里多达100万个节点的超低功耗通信

Compute selection and routing decisions jointly matter. Placement of applications and NFs on compute that is near vs. far from the user [73], and the choice of network routes between users and this compute both impact performance. While operator edges are closer to users (few ms away) and hence are an ideal target in which to place highly responsive workloads (e.g., vRAN), they have very limited compute due to physical space and power constraints at cellular base stations and mobile switching centers [74]. On the other extreme, the DC is furthest away (tens to hundred ms away) but has far more available compute.

Traditional selection techniques, such as those in CDNs (Content Delivery Networks), fall significantly short here. Directing all traffic from an entire subnet or LDNS (local DNS) coverage area to a single destination leads to undifferentiated performance for multiple flows with different requirements. Similar to DNS, the 5G NRF [33,80] (NF Repository Function) maintains profiles of available instances of NFs, and provides discovery and load balancing. However, it is not network-aware and only considers compute load. The problem is further exacerbated by the use of SFC (Service Function Chaining) [20] in 5G, where instances of different NFs are dynamically chained together to serve a request.

Picking a destination for a network demand based on compute capacity alone can lead to poor E2E performance. Picking a destination for a network demand based on network performance alone can lead to overloaded compute. Performance studies of existing 5G networks [91] have already found that "the untamed latency in the wireline paths, which is beyond mobile carriers’ control, may neutralize 5G’s latency advantage" and argue that "the wireline paths, upperlayer protocols, computing and radio hardware architecture need to co-evolve with 5G."

计算资源选择与路由决策的共同重要性。 将应用和网络功能部署在离用户近还是远的计算资源上 [73]，以及选择用户与这些计算资源之间的网络路由，两者都会影响性能。

虽然运营商边缘离用户更近（几毫秒），因此是部署高响应性工作负载（如vRAN）的理想选择，但由于蜂窝基站和移动交换中心物理空间和电力的限制，其计算资源非常有限 [74]。在另一端，数据中心距离最远（几十到上百毫秒），但拥有更多的可用计算资源。

传统的选择技术，如内容分发网络 (CDN) 中使用的技术，在这里远远不够。将整个子网或本地DNS (LDNS) 覆盖区域的所有流量引导到单个目的地，会导致对具有不同需求的多个流提供无差别的性能。与DNS类似，5G的网络功能存储库功能 (NRF) [33, 80] 维护着可用网络功能实例的配置文件，并提供发现和负载均衡功能。然而，它不具备网络感知能力，只考虑计算负载。在5G中使用服务功能链 (SFC) [20] 使得这个问题进一步加剧，因为不同网络功能的实例会被动态地链接在一起以服务一个请求。

仅根据计算能力为网络需求选择目的地，可能导致端到端性能不佳。仅根据网络性能为网络需求选择目的地，则可能导致计算资源过载。对现有5G网络的性能研究 [91] 已经发现，“有线路径中难以控制的延迟，超出了移动运营商的控制范围，可能会抵消5G的延迟优势”，并主张“有线路径、上层协议、计算和无线硬件架构需要与5G协同发展”。

Need for multi-WAN coordination. Cloudification forces operators and clouds to share responsibility for tight performance guarantees of 5G workloads. Individual routing and compute placement decisions leads to sub-optimal performance. Figure 2 shows a latency sensitive flow traversing two WANs from source R 4 to destination R 6 . Each link has an associated latency. The operator chooses the best egress point-of-presence (PoP) R 3 and path R 4 R 3 to R 3 to minimize RTT on its network. The cloud routes traffic from R 8 to R6 following the optimal RTT path, resulting in the E2E path P 1 . However, a more efficient path P 2 exists. We show in §7 that decision making across both the cloud and operator unlocks such performance improvements.

Given the multitude of 5G networks across the world, multiple cloud providers, and dynamic nature of traffic and operational WANs, manual coordination to address performance problems as they appear does not scale.

多广域网协调的必要性。 云化迫使运营商和云服务商共同承担为5G工作负载提供严格性能保障的责任。 各自独立的路由和计算资源放置决策会导致次优性能。

图2展示了一个延迟敏感的流从源R4穿越两个广域网到达目的地R6: 每条链路都有一个相关的延迟。

• 运营商为了最小化其网络内的RTT，选择了最佳的出网接入点 (PoP) R3和路径R4→R3
• 云服务商则将流量从R8路由到R6，遵循RTT最优的路径，最终形成端到端路径P1

然而，存在一条更高效的路径P2

我们将在第7节中展示，跨云和运营商进行协同决策可以实现这样的性能提升。

鉴于全球有众多的5G网络、多个云提供商，以及流量和运营中广域网的动态性，出现性能问题时再进行手动协调的模式是无法扩展的。

New incentives for multi-WAN co-operation. Historically, it has been challenging to achieve collaborative routing beyond traditional peering agreements on the Internet. However, the cloudification of access has aligned the economic incentives of cloud and operator networks. The operator is not only selling access to end users to the cloud, it is now also a customer of the cloud for hosting 5G applications and NFs. There is a shared responsibility to manage compute and

多广域网合作的新激励。 历史上，在互联网上实现超越传统对等互联协议的协作路由一直很困难。然而，接入网的云化使得云和运营商网络的经济激励趋于一致。 运营商不仅向云端销售最终用户接入服务，它现在也成为了云的客户，托管5G应用和网络功能。双方有共同的责任来管理计算和网络资源。

OTTER

Prior empirical work [91] has shown that as 5G performance over-the-air has improved, WANs have become the bottleneck. We tackle this problem by placing heterogenous performance demands on the most appropriate compute and network paths across operator and cloud WANs. We refer to this as the multi-WAN flow placement problem, and it requires two core capabilities — the ability to direct a flow to a specific compute endpoint over a specific network path, and the ability to calculate that optimal path and endpoint.

先前已有实证研究 [91] 表明，随着5G空中接口性能的提升，广域网（WANs）已成为性能瓶瓶颈。

我们致力于解决这一问题，其方法是 将具有异构性能需求的流量，放置到跨越运营商和云服务商广域网的最合适的计算和网络路径上

我们将此问题称为多广域网流放置问题，它需要两项核心能力:

1. 能够将一条流通过特定的网络路径引导至特定的计算端点
2. 能够计算出该最优路径和端点的能力

3.1 Design Goals

We design OTTER with the following goals:

我们设计OTTER时遵循以下目标：

Co-exist with non-5G WAN traffic. 5G traffic will share operational WANs that serve existing customers. For example, operator WANs already provide wireline connectivity to consumers or eyeballs. Cloud WANs carry enterprise and consumer workloads. Moreover, both types of WANs operate existing TE systems to engineer the capacity of their network efficiently [11,25,26,39,41,89]. However, both operator and cloud TE systems aim to achieve network welfare goals such as high utilization or bandwidth fairness [39,41], rather than optimizing for individual service objectives. Instead of overhauling existing TE systems in cloud and operator WANs, we design OTTER to be an overlay that operates in a layer of abstraction above existing WAN TE.

与非5G广域网流量共存。 5G流量将与服务现有客户的运营中广域网共享网络。例如，运营商广域网已经为消费者或“眼球”（eyeballs）网络提供有线连接。云广域网则承载着企业和消费者的工作负载。此外， 这两种广域网都运行着现有的流量工程（TE）系统，以高效地规划其网络容量 [11, 25, 26, 39, 41, 89]。然而， 运营商和云服务商的TE系统都旨在实现网络整体效益目标，如高利用率或带宽公平性 [39, 41]，而不是为单个服务的具体目标进行优化。

我们设计的OTTER并非要彻底改革云和运营商广域网中现有的TE系统，而是 作为一个运行在现有广域网TE之上抽象层的覆盖网络（overlay）

Dynamic flow placement. This overlay design choice further allows OTTER to place 5G flows on network paths and compute endpoints on-demand. In contrast, existing TE and other optimizations operate on batched traffic demands by periodically (e.g., every 5 minutes) computing flow allocations from a predicted traffic matrix [39, 41], or running a single optimizer instance for a single input demand [40]. A broker [39] enforces those periodic allocations by squelching flows that exceed them. While existing TE will continue to run on each WAN separately, OTTER can dynamically place flows without being subject to periodic allocations nor requiring bandwidth brokers in 5G deployments.

动态流放置。 这种覆盖网络的设计选择进一步允许OTTER按需将5G流放置到网络路径和计算端点上。

相比之下，现有的TE和其他优化方法通过周期性地（例如，每5分钟）从预测的流量矩阵中计算流分配来处理批量流量需求 [39, 41]，或为单个输入需求运行单个优化器实例 [40]。

代理（broker）[39] 通过抑制超出配额的流来强制执行这些周期性分配。

尽管现有的TE将继续在每个广域网上独立运行，但OTTER可以动态地放置流，既不受周期性分配的限制，也无需在5G部署中使用带宽代理。

Support multiple, fine-grained service objectives. Today’s WAN TE systems provide service differentiation using only a small number of priority classes (e.g., high priority vs. low priority) [10, 39]. Such few classes are insufficient to express the more complex and fine-grained objectives of 5G traffic. These coarse-grained priority classes can fail to capture the demands of lower priority flows, even when there exists an allocation that maps all flows to acceptable paths. Consider a simple example in Figure 3, where each link can carry at most one flow. Flow F A has high priority with a latency requirement of 20 ms, and flow F B has low priority with a throughput requirement of 10 Gbps. On the left, we show the allocation where flows are assigned in strict priority order, preferring shorter paths—this is how modern TE systems like OneWAN [45] and SWAN [39] operate. F A will be assigned first to the top path while F B will be assigned to the remaining bottom path and fall short of its throughput requirement. On the right, we show an alternate allocation that can map both flows to paths that meet their service demands. However, it is not trivial to find such an allocation at the scale of hundreds of endpoints and thousands of flows, each with their own performance demands across a variety of metrics.

支持多个细粒度的服务目标。 现今的广域网TE系统仅使用少数几个优先级类别（例如，高优先级与低优先级）来提供服务差异化 [10, 39]。如此少的类别不足以表达5G流量更为复杂和细粒度的目标。这些粗粒度的优先级类别可能无法满足低优先级流的需求，即使存在能够将所有流映射到可接受路径的分配方案。

考虑图3中的一个简单例子，其中每条链路最多只能承载一条流。流 \(F_A\) 具有高优先级，延迟要求为20毫秒；流 \(F_B\) 为低优先级，吞吐量要求为10 Gbps。

• 在左侧，我们展示了严格按照优先级顺序分配流、并偏好更短路径的方案——这是像OneWAN [45]和SWAN [39]这样的现代TE系统的运作方式。\(F_A\) 将首先被分配到顶部路径，而 \(F_B\) 将被分配到剩余的底部路径，从而无法满足其吞吐量要求
• 在右侧，我们展示了另一种分配方案，可以将两条流都映射到满足其服务需求的路径上

然而，在拥有数百个端点和数千条流，且每条流在多种性能指标上都有自身需求的规模下，找到这样的分配方案并非易事。

Take inter-domain routing protocols as a given. InterWAN routing leads to sub-optimal E2E routes due to limited information disclosure and different goals between WANs [47, 48]. While prior work has shown that ASes can exchange information to produce E2E optimal routes, such negotiation entails heavyweight communication overheads that may not be practical [47,48]. In the context of 5G, this overhead is further exacerbated by numerous traffic flows, each with widely varying objectives. So, we design OTTER without requiring changes to inter-domain routing protocols, policies, and implementations.

将域间路由协议视为既定条件。 由于广域网之间有限的信息披露和不同的目标，跨广域网路由（Inter-WAN routing）会导致次优的端到端（E2E）路由 [47, 48]。

尽管先前的工作表明，自治系统（ASes）可以通过交换信息来产生端到端最优路由，但这种协商会带来沉重的通信开销，可能不切实际 [47, 48]。

在5G的背景下，大量的流量以及每条流迥异的目标进一步加剧了这种开销。因此，我们设计OTTER时，无需更改域间路由协议、策略和实现。

Not require private WAN data. While sharing intra-WAN topologies could lead to better multi-WAN paths, ASes have economic incentives to limit such visibility [47, 48, 75, 81]. 5G cloudification presents a unique opportunity for efficient cross-domain routing without sharing such private data. The operator, as a customer of the cloud, owns the workloads deployed on VMs in cloud edges, DCs, and operator edges. The cloud provides the control plane for managing bare metal at these locations, such as provisioning VMs and establishing connectivity. Thus both the operator and the cloud have visibility into network performance at an overlay layer between all compute locations. OTTER uses this visibility to make informed flow and compute placement. We show that the operator can deploy and run OTTER across both WANs, without access to private data or capabilities, and still provide significant performance improvements.

不要求私有广域网数据。 虽然共享广域网内部的拓扑结构可能有助于找到更好的多广域网路径，但自治系统（ASes）有经济动机来限制此类信息的可见性 [47, 48, 75, 81]。

5G的云化为实现高效的跨域路由提供了一个独特的机会，而无需共享此类私有数据:

• 运营商作为云服务商的客户，拥有部署在云边缘、数据中心和运营商 边缘的虚拟机（VMs） 上的工作负载
• 云服务商提供用于管理这些位置裸金属的控制平面，例如配置虚拟机和建立连接

因此，运营商和云服务商在所有计算位置之间的覆盖网络层上，都对网络性能具有可见性。

OTTER利用这种可见性来做出明智的流和计算资源放置决策。

我们证明，运营商可以在两个广域网上部署和运行OTTER，即使无法访问私有数据或能力，仍然可以提供显著的性能提升。

Multi-cloud support with open interfaces. 5G operators want to avoid cloud-lock in and deploy their NFs to one or more cloud providers of their choice. To achieve this, we design OTTER as a multi-cloud network overlay across operator and cloud WANs. OTTER uses publicly available VMs as nodes in the overlay, and a variety of public cloud NFs for routing between the nodes. This allows operators to use OTTER in multi-cloud deployments, without requiring private or special privileges from an underlying cloud.

通过开放接口支持多云。 5G运营商希望避免被单一云厂商锁定，并希望将其网络功能（NFs）部署到一个或多个所选的云提供商。

为实现这一目标，我们将OTTER设计为一个跨越运营商和云广域网的多云网络覆盖。OTTER使用公开可用的虚拟机作为覆盖网络中的节点，并利用各种公共云网络功能在节点间进行路由。这使得运营商可以在多云部署中使用OTTER，而无需底层云服务商提供私有或特殊权限。

3.2 OTTER Overview

OTTER places flows in a multi-WAN environment to satisfy their service demands. It does so by co-optimizing which 5G application / NF compute will serve the flow, and the route that flow will take to it. Figure 4 shows OTTER’s architecture consisting of the Controller and Orchestrator. The Controller accepts flow demands to 5G NFs and applications using APIs [28, 34, 53, 57]. These "Quality on Demand" APIs [3] allow an application to create a session by issuing an HTTP POST to the /sessions API, providing flow identifiers (IP addresses, port ranges, device phone number, etc.) and a QoS profile that best matches the application needs (see the "quality-on-demand v0.11.1" API on Swaggger [3]). Such standardized APIs have been implemented by 5G operators (see the implementations by Deutsche Telekom, Orange, and Spry Fox Networks posted on GitHub [3]) and by clouds that offer to host 5G services [15]. In addition to allowing application developers to specify their 5G flow needs, these APIs allow for termination of a session, extending a session, and getting notified when the requested QoS cannot be fulfilled. A UE that disconnects and reconnects, or incurs an inter-RAT (Radio Access Technology) handover, can create a new session with this API.

OTTER在多广域网环境中放置流以满足其服务需求。它通过协同优化服务于该流的5G应用/网络功能（NF）计算资源，以及该流到达该资源所采用的路由来实现这一目标。

图4展示了OTTER的架构，它由控制器（Controller）和编排器（Orchestrator）组成。

控制器使用API [28, 34, 53, 57] 接收发往5G网络功能和应用的流需求。这些“按需质量”（Quality on Demand）API [3] 允许应用程序通过向/sessions API发出一个HTTP POST请求来创建一个会话，请求中提供流标识符（IP地址、端口范围、设备电话号码等）和一个最匹配应用需求的QoS配置文件（参见Swagger上的"quality-on-demand v0.11.1" API [3]）。

这类标准化API已被5G运营商（参见德国电信、Orange和Spry Fox Networks在GitHub上发布的实现[3]）以及提供5G服务托管的云服务商[15]所实现。

除了允许应用开发者指定其5G流需求外，这些API还允许终止会话、延长会话以及在无法满足所请求的QoS时获得通知。

当用户设备（UE）断开重连，或发生跨无线接入技术（inter-RAT）切换时，可以通过此API创建一个新会话。

The Controller allocates compute to meet these performance requirements within the limits of compute availability on edges and DCs. The Controller calculates the most appropriate network paths to the allocated compute. Unlike TE systems, OTTER relies on available compute capacity at destination sites, as well as periodic measurements of network paths that it makes. This not only allows OTTER to route flows along alternate paths, but also to alternate destinations. The Orchestrator manages cloud resources and implements the forwarding mechanism to steer traffic along routes computed by the Controller. It provisions compute as per the Controller’s optimal placement.

控制器 在边缘和数据中心计算资源可用性的限制内分配计算资源，以满足这些性能要求。控制器 计算出通向所分配计算资源的最优网络路径

与TE系统不同，OTTER不仅依赖于目的站点的可用计算容量，还依赖于它所做的周期性网络路径测量。这不仅使OTTER能够沿着备用路径路由流，还能将其路由到备用目的地。

编排器 管理云资源，并实施转发机制以引导流量沿控制器计算出的路径行进。它 根据控制器的最优放置决策来配置计算资源

While similar challenges can be faced by other applications across WANs, OTTER is designed to serve 5G deployments. The business alignment between "eyeball" networks and cloud networks, where 5G deployments are hosted across both, drives the necessary incentive to orchestrate both WANs. The diversity of 5G applications requires a design that supports multiple classes of traffic, not just bulk traffic [40]. End-user flows have to be placed in real time, not at traditional, periodic TE intervals. OTTER leverages APIs [28,34,53,57] already deployed by 5G providers to accept these application demands, where such APIs have not been deployed at scale for other applications nor networks. Finally, we design OTTER as an overlay without forcing one WAN to have administrative control nor require use of private capabilities – this design hence does not preclude a cloud from supporting multiple 5G operators, and a 5G operator from relying on multiple clouds.

虽然其他跨广域网应用也可能面临类似的挑战，但OTTER是为服务5G部署而设计的。“眼球”网络和云网络之间的商业协同（5G部署横跨两者）为协同编排两个广域网提供了必要的动力。5G应用的多样性要求设计能够支持多种流量类别，而不仅仅是批量流量 [40]。终端用户的流必须实时放置，而不是在传统的周期性TE时间间隔内进行。OTTER利用5G提供商已经部署的API [28, 34, 53, 57]来接收这些应用需求，而这类API尚未在其他应用或网络中得到大规模部署。最后，我们将OTTER设计为一个覆盖网络，其设计无需强制一个广域网拥有管理控制权，或要求使用私有能力——因此，该设计既不妨碍云服务商支持多个5G运营商，也不妨碍5G运营商依赖多个云服务商。

OTTER Controller

The OTTER Controller maps traffic demands to network paths and compute resources. It does so while maximizing the allocated traffic and minimizing the degree to which 5G application-specific service requirements are violated. The resulting mapping of flows to destination compute sites and the network paths to reach them is an input to the OTTER Orchestrator. It informs the Orchestrator at which of the available edge sites or DCs to terminate 5G flows and which network paths to use. In this section, we formalize this problem, referred to as the 5G multi-WAN flow placement problem, explain the constraints on the Optimizer, and describe techniques used to ensure its scalability.

The Optimizer takes as input the set of paths \(P\) that are available in the multi-WAN network. Each path \(p \in P\) is represented by the tuple \((s, d, L, \sigma)_p\), where \(s_p\) is the source, \(d_p\) is the destination, \(L_p\) is the set of links used by the path, and \(\sigma_p\) is a vector of performance metrics for path \(p\). \(\sigma_p^m\) refers to the value of metric \(m\) for path \(p\). For example, the RTT, jitter, and packet loss of path \(p\) are represented by \(\sigma_p^{rtt}\), \(\sigma_p^{jit}\), \(\sigma_p^{loss}\). \(L\) is the set of links in the network, and each link \(l \in L\) has capacity \(c_l\). Let \(P_l\) be the set of paths that share a link \(l\), and \(P_d\) be the set of paths that share a destination \(d\).

The Optimizer takes as input a set of 5G flows \(F\) that need to be allocated. Each flow \(f \in F\) is represented by the tuple \((s, D, D_f, r_f)_f\). \(s_f\) is the source of the demand. \(D_f\) is the set of destinations (specific edges & DCs) that can serve the flow. \(D_f\) represents the service demands of the flow with \(D_f^m\) giving the demand for metric \(m\). \(bw_f\) is the requested bandwidth. \(r_f\) is a vector of destination resource requirements of the flow – compute, memory, and storage. We write \(r_f^\pi\) to refer to the requirements of flow \(f\) with respect to resource type \(\pi\). Unlike TE, an OTTER flow is not limited to a single destination. Rather, we leverage the agility of the cloud to place compute resources for serving a 5G application or NF across multiple destinations. This allows OTTER to further optimize path performance by adjusting the destinations of flows. Compared to resource-constrained edge sites, a destination \(d\) that is a DC has \(\approx \infty\) capacity \(c_d^\pi\). For scalability, individual flows with the same source, destination set, service demands, and resource requirements are treated as part of a single “flow aggregate” for the Optimizer to allocate. The Optimizer solves the flow placement problem such that all flows within the same flow aggregate are routed in the same way and to the same destination.

Modeling service demands. We encode the service demand of a flow that is obtained from operator APIs (§ 3.2) as a vector of demand functions \(D_f\). A demand function \(D_f^m\) maps a flow’s performance expectation, i.e., what the flow expects to achieve for a given allocation, to a tolerance coefficient \(D_f^m(\sigma_p^m) \in [0, 1]\). The tolerance coefficient is a measure of how good the path is for a flow. For example, an application may consider a path RTT under 100 ms acceptable, over 200 ms unacceptable, and linearly decrease its preference for a path between 100 and 200 ms. In this case, the Optimizer computes the tolerance coefficient to be 1 for all paths with an RTT under 100 ms, \(\epsilon\) for all paths greater than 200 ms, and a value inversely proportional to the RTT for paths between 100 and 200 ms: \(D_f^{rtt}(\sigma_p^{rtt}) = \begin{cases} 1, & \text{if } \sigma_p^{rtt} \le 100 \\ -\frac{1}{100}\sigma_p^{rtt} + 2, & \text{if } 100 < \sigma_p^{rtt} \le 200 \\ \epsilon, & \text{if } \sigma_p^{rtt} > 200 \end{cases}\) The Optimizer’s goal is to allocate demands such that the degree of satisfaction (tolerance coefficients) is maximized. All tolerance coefficients are constant, allowing the objective function to be linear and the optimization problem to be a linear program (LP). The use of demand functions allows the Optimizer to scale over numerous path metrics and performance demands, as well as future-proof it for new applications that have arbitrary service demands.

Modeling resource requirements. Each flow specifies a resource-requirement vector \(r_f\). Each element represents compute in vCPUs, RAM, and disk storage needed to serve the 5G workload. While we focus on those three resources, the vector can be expanded and compared component-wise to the resource capacities of destination sites.

Decision variables. The Optimizer assigns network flows to paths in \(P\) and compute resources to destination sites in \(D\). We have two decision variables: \(x_{f,p}\), the amount of bandwidth of flow \(f\) assigned to path \(p\), and \(y_{f,d}^\pi\), the amount of resources of type \(\pi\) for flow \(f\) allocated at destination \(d\). OTTER co-optimizes the flow allocation and the destination placement. See Table 1 for a summary of variables.

Objective function. The Optimizer aims to assign network flows to multi-WAN paths such that as much traffic is allocated as possible while still satisfying performance demands. The cost of a single flow assignment is the product of its allocated bytes and the overall tolerance coefficient summed over all performance metrics. To prevent bias towards large flows, we normalize this cost by the requested bandwidth of the flow \(bw_f\). The objective function is as follows: \(\underset{f, p}{\text{argmax}} \sum (x_{f,p} \cdot \sum_{m \in \{rtt, jit, loss, bw\}} D_f^m(\sigma_p^m))/bw_f\) We note that our objective function does not inherently prioritize one metric over another. Rather, applications can indicate priority through the demand functions.

Constraints. Table 2 summarizes the constraints in OTTER’s optimization. Constraint c1 ensures that the total allocated flow is non-negative. c2 and c3 ensure that the allocated flows are bounded by their requested bandwidth as well as the bottleneck capacity of the paths. Destination sites have resource constraints that limit how many flows can map to them, and c4 prevents overload. Constraint c5 requires that the proportion of resources assigned to a flow at a destination is equal to the proportion of the amount of bandwidth allocated to the flow at the same destination.

On-demand 5G flow allocation. Unlike TE systems which allocate flows periodically (e.g., every 5 minutes) for predicted traffic demands, the OTTER Controller must perform on-demand flow placement. This difference arises due to the dynamic and unpredictable nature of 5G traffic [24] compared to typical WAN workloads. Unfortunately, re-running the Optimizer too frequently incurs a large delay. While the Optimizer computes newly optimal flow allocations, stale routes remain in-use, potentially leading to service demand violations [93]. To handle this, the Controller implements a greedy heuristic that approximates the optimal (i.e., invoking the Optimizer on every new flow) configuration. It greedily allocates flows to available paths and destinations to provide the best demand satisfaction, while adhering to resource constraints. The Controller then periodically runs the full Optimizer in the background to optimally place new flows and those that were greedily assigned since the previous call to the Optimizer. The Controller can then shift those flows to achieve a globally optimal allocation, or it can pin flows to specific paths or destinations to avoid packet re-ordering. While shifting flows could result in an individual flow experiencing lower service demand satisfaction compared to its initial greedy allocation, reallocation still improves overall satisfaction across more flows, as shown in §7.

Computational complexity of OTTER. OTTER’s optimizer is a linear program. Off-the-shelf optimization solvers implement polynomial time algorithms to solve linear programs, making it possible to solve OTTER’s optimization quickly. While linear programs are quick to solve in practice, increasing the size of the problem can increase the run time of the solver. OTTER has two ways of dealing with high run times. First, OTTER reduces the set of possible destinations for each flow to a set of size \(n\) prior to solving the optimization. We estimate the \(n\) “best” destination sites using a flow’s service demands (\(D_f^m\)) to rank candidate destinations by preference. This effectively reduces the number of variables in the linear program. Second, depending on the size of the linear program, OTTER can run the full optimization at longer periodic intervals (e.g., ten minutes) since the greedy flow allocation can assign flows to paths without waiting for the next interval. We evaluate the performance of greedy flow allocation between consecutive runs of the Optimizer, as well as the Optimizer, in detail in §7.

这一部分太长了, gemini 简要概括一下:

• 核心功能： OTTER 控制器的主要任务是将 5G 网络的流量需求，智能地分配到最合适的网络路径和计算资源上，目标是尽可能多地传输流量，同时最大限度地保证服务质量

• 建模方法： 它将这个复杂的分配问题建立成一个线性规划（Linear Programming, LP）模型

  • 其中最关键的是引入了“需求函数”和“容忍系数”的概念，用来量化一条网络路径（如延迟、抖动等指标）对特定应用需求的满足程度

• 动态应对策略： 考虑到 5G 流量的突发性和不可预测性，它不采用传统的周期性分配，而是执行“按需分配”

  • 它通过一种贪心启发式算法来快速处理新流入的流量，先给出一个临时的“次优解”，然后在后台运行完整的优化算法，以找到长期的“全局最优解”

• 复杂度管理技术： 为了避免大规模优化问题带来的巨大计算开销，它采用两种方法来“减负”：

  1. 通过算法预先筛选出少数几个“最佳”的目的地，缩小求解范围
  2. 将完整的、消耗巨大的优化计算放在较长的时间间隔（如十分钟）进行，从而确保系统的敏捷和高效

OTTER Orchestrator

The Orchestrator includes the forwarding mechanism necessary to steer traffic along routes computed by the Controller (§4). It also manages compute as per the Controller’s optimal placement. We design the Orchestrator as a multi-WAN overlay on a cloud WAN and an operator WAN. To evaluate OTTER without impacting an operational 5G network, we substitute the operator WAN with GCP (Google Cloud Platform). Figure 5 shows one small deployment of OTTER, and Figure 19 in Appendix A shows a larger one. The Orchestrator uses only existing, native cloud functionality, which satisfies our design goal of not requiring private WAN data (§3).

编排器包含必要的转发机制，以引导流量沿控制器（§4）计算出的路径传输。它还根据控制器的最优放置方案来管理计算资源。

我们将编排器设计为构建在云广域网和运营商广域网之上的多广域网覆盖网络。

为了在不影响运营中的 5G 网络的情况下评估 OTTER，我们用谷歌云平台（GCP）替代了运营商广域网。

图5展示了一个小型的 OTTER 部署，附录A中的图19展示了一个更大的部署。编排器仅使用现有的、原生的云功能，这满足了我们不需要私有广域网数据的设计目标（§3）:

Connectivity in the OTTER Orchestrator overlay. We host multiple VMs in each Azure region to represent 5G NF and application servers. VMs in GCP serve as 5G cell sites with far edges, while virtual private network (VPN) gateways behave like 5G near edge sites. We create private subnets in each GCP region and virtual private cloud (VPC) peer-ing connections in full-mesh between GCP regions. We use virtual network (VNet) peering between clusters of VMs in different regions to connect them to each other and to VPN gateways. Given that much of the 5G infrastructure, including cell sites and NFs, use private IP address ranges to protect communication, VPN tunnels are necessary to establish private connectivity among them. We have configured the VPN gateways to allow transit with the 5G operator WAN where both clouds have regional DCs and underlying peering (see Appendix A).

OTTER 编排器覆盖网络中的连接性:

• 我们在每个 Azure 区域托管多个虚拟机（VM）来代表 5G 网络功能（NF）和应用服务器
• GCP 中的 VM 充当具有远边缘的 5G 蜂窝基站
• 虚拟专用网络（VPN）网关则扮演 5G 近边缘站点的角色

---

1. 我们在每个 GCP 区域创建私有子网，并在 GCP 区域之间建立全网状的虚拟私有云（VPC）对等连接
2. 我们使用不同区域中 VM 集群之间的虚拟网络（VNet）对等连接，将它们相互连接并连接到 VPN 网关

鉴于包括蜂窝基站和 NF 在内的许多 5G 基础设施都使用私有 IP 地址范围来保护通信，因此必须使用 VPN 隧道来在它们之间建立私有连接。我们已将 VPN 网关配置为允许与 5G 运营商广域网进行传输，前提是两个云都在该区域拥有数据中心和底层的对等连接（见附录A）

Forwarding in the OTTER Orchestrator overlay. The OTTER Orchestrator leverages native functionality in each cloud to achieve high forwarding performance, ease of scale, and lower deployment complexity. To control the path that a 5G flow takes in the overlay, we use a combination of route tables with user-defined routes (UDRs) and choice of multiple VNets in the destination region, something that both GCP and Azure support without requiring the customer to deploy their own BGP speakers and packet forwarders. Other design choices, such as using SNAT and DNAT with VPN gateways and firewalls [51], or MP-TCP on source and destination VMs, incur different tradeoffs in flexibility and overheads. We configure advanced parameters on VNet peering and network firewalls to allow VMs in one region to use a VPN gateway in another and allow transit traffic between regions. Each cloud component supports scale-up and scale-out, allowing the entire topology to easily cover the continental US, for example, without any code changes to individual components. We use symmetric paths for a flow but can easily use different paths in each direction. The use of one system to identify, manage, and use routes across WANs allows us to satisfy the need for multi-WAN coordination (§ 2).

OTTER 编排器覆盖网络中的转发:

OTTER 编排器利用每个云的原生功能来实现高转发性能、易于扩展和较低的部署复杂性。

为了控制 5G 流量在覆盖网络中采用的路径，我们结合使用了带有用户定义路由（UDR）的路由表和在目标区域选择多个 VNet 的方法，GCP 和 Azure 都支持这种方式，无需客户部署自己的 BGP Speaker 和数据包转发器。其他设计选择，例如使用带有 VPN 网关和防火墙的 SNAT 和 DNAT [51]，或在源和目标 VM 上使用 MP-TCP，会在灵活性和开销方面带来不同的权衡。

我们在 VNet 对等连接和网络防火墙上配置了高级参数，以允许一个区域的 VM 使用另一个区域的 VPN 网关，并允许区域间的传输流量。每个云组件都支持纵向和横向扩展，使得整个拓扑可以轻松覆盖例如美国大陆，而无需对单个组件进行任何代码更改。

我们为流量使用对称路径，但也可以轻松地在每个方向上使用不同的路径。

使用一个系统来识别、管理和使用跨广域网的路由，使我们能够满足多广域网协调的需求（§2）。

Secure multi-WAN routing. We use private VNets to host 5G cell sites, edge sites, and NFs. We use VNet peering to allow only authorized traffic to transit between VNets. We use VPN gateways to enforce encryption when transiting WANs. Our design choices were inspired by product demonstrations from industry 5G NF providers [61].

安全的多广域网路由:

我们使用私有 VNet 来托管 5G 蜂窝基站、边缘站点和 NF

• 使用 VNet 对等连接，只允许授权流量在 VNet 之间传输
• 使用 VPN 网关在跨越广域网传输时强制加密

我们的设计选择受到了行业 5G NF 提供商产品演示的启发[61]

Extensible multi-WAN topology. This design allows both scale and flexibility. Given an arbitrary source in a GCP region and a destination in an Azure region, it can use any VPN connection and any intermediate Azure and/or GCP region to send and receive traffic. We achieve this through a combination of VPN connections, VNet peering, and selective route announcements. We trivially increase inter-WAN forwarding throughput by scaling out VPN gateways and tunnels.

可扩展的多广域网拓扑: 这种设计兼具扩展性和灵活性。

对于 GCP 区域中的任意源和 Azure 区域中的任意目的地，它可以使用任何 VPN 连接以及任何中间的 Azure 和/或 GCP 区域来发送和接收流量

我们通过 VPN 连接、VNet 对等连接和选择性路由宣告的组合来实现这一点

通过横向扩展 VPN 网关和隧道，我们可以轻易地增加跨广域网的转发吞吐量

System Implementation

We have implemented the entire OTTER system shown in Figure 4 to operate on top of Azure and GCP, with no special support from either underlying cloud. The Orchestrator is built with HashiCorp’s Terraform [86]. Our evaluation topology (available in an anonymous repository 1 ), listed in Table 3 and described in §7.1, is built in ∼4,100 LoC of HCL (HashiCorp Configuration Language), with ∼2,600 LoC for module deployments, ∼600 LoC for GCP resource declarations, and ∼900 LoC for Azure resource declarations.

We implemented the Measurement Coordinator in ∼1,200 LoC in C#. It uses the Azure and GCP SDKs to automate periodic execution of measurement tools for path bandwidth, RTT, jitter, and loss. We use iPerf3 [23] to measure throughput, configured to use TCP CUBIC with 60 parallel connections, 2 seconds of warm up, and 5 seconds of measurement. For packet loss, we use iPerf3 in UDP mode. For RTT and jitter, we use sockperf [49]. We use round-trip values, and jitter is calculated as the standard deviation of RTT.

The Measurement Coordinator analyzes the output of the Orchestrator’s Terraform execution to determine the deployment topology and the names and IP addresses of its components, including source and destination VMs. It then creates a non-overlapping schedule of measurements. The output of the measurement tools is parsed and converted to JSON. That JSON is then inserted as a new document record into an Azure Cosmos DB NoSQL database [54].

The Optimizer issues simple SQL queries on that database to obtain sliding window values of median throughput, RTT, jitter, and loss for all paths. 5G flow requirements are modeled from a subset of application profiles in the 3GPP standard [79], which we describe in §7.2. However, we expect to integrate with network APIs [28, 34] such as Azure Programmable Connectivity [53,57] that allow 5G applications to specify flow QoS requirements. We have implemented the Optimizer using the Gurobi [35] solver. The Optimizer is an LP of ∼500 LoC. We run our implementation on a VM with a 16 core 3GHz CPU and 48 GB of memory.

我们已经实现了图4所示的整个 OTTER 系统，使其能够在 Azure 和 GCP 之上运行，无需任何来自底层云的特殊支持。编排器是使用 HashiCorp 的 Terraform [86] 构建的。

我们的评估拓扑（可在一个匿名仓库中获取¹），在表3中列出并在 §7.1 中描述，由约 4100 行 HCL（HashiCorp 配置语言）代码构建，其中约 2600 行用于模块部署，约 600 行用于 GCP 资源声明，约 900 行用于 Azure 资源声明。

我们用约 1200 行 C# 代码实现了测量协调器。它使用 Azure 和 GCP 的 SDK 来自动化周期性执行测量工具，以获取路径带宽、RTT、抖动和丢包

1. 我们使用 iPerf3 [23] 来测量吞吐量，配置为使用 TCP CUBIC、60 个并行连接、2 秒预热和 5 秒测量
2. 对于丢包，我们使用 UDP 模式的 iPerf3
3. 对于 RTT 和抖动，我们使用 sockperf [49]。我们使用往返值，抖动计算为 RTT 的标准差

测量协调器分析编排器的 Terraform 执行输出来确定部署拓扑及其组件的名称和 IP 地址，包括源和目标 VM。然后，它创建一个无重叠的测量计划。测量工具的输出被解析并转换为 JSON 格式。该 JSON 随后作为新文档记录插入到 Azure Cosmos DB NoSQL 数据库[54]中。

优化器对该数据库执行简单的 SQL 查询，以获取所有路径的吞吐量、RTT、抖动和丢包的中位数滑动窗口值。5G 流量需求是根据 3GPP 标准[79]中的一部分应用配置文件建模的，我们将在 §7.2 中描述。

然而，我们期望与网络 API [28, 34]（如允许 5G 应用指定流量 QoS 需求的 Azure Programmable Connectivity [53,57]）集成。我们使用 Gurobi [35] 求解器实现了优化器。

优化器是一个约 500 行代码的线性规划问题。我们的实现运行在一台配备 16 核 3GHz CPU 和 48 GB 内存的虚拟机上。

Miscellany

Why does BGP not get in the way? Cellular network operators tend to be large ISPs that directly peer with public clouds in multiple locations. Our evaluation setup in §7 represents that, with both WANs directly peering and our use of VPN gateways at the overlay layer. Not only do we expect 5G traffic to flow over direct connections, we expect 5G cloudification to spur operators to increase the quantity of peering. This has two implications for OTTER. First, many of the oddities of BGP in the wild do not apply here, where intermediate ASes enforce their policies between source and destination ASes. Second, we expect to see even more diversity of inter-domain paths that OTTER can exploit.

为什么 BGP 不会成为阻碍？

蜂窝网络运营商通常是大型的互联网服务提供商（ISP），它们在多个地点与公有云直接对等互联。

我们在 §7 中的评估设置就体现了这一点，其中两个广域网（WAN）直接对等，并且我们在覆盖网络层使用了 VPN 网关。我们不仅预期 5G 流量会流经这些直接连接，还预期 5G 的云化趋势将促使运营商增加对等互联的数量。这对 OTTER 有两层含义:

首先，许多 BGP 在实际应用中的复杂特性（例如中间自治系统 AS 在源和目的 AS 之间强制执行其策略）在这里并不适用

其次，我们预期将看到更多可供 OTTER 利用的、多样化的域间路径

Why not optimize for WAN costs? OTTER does not explicitly incorporate monetary cost as an optimization criteria. Cloud compute and network costs tend to be uniform within countries. Most 5G network deployments are at national scales. Operators and clouds that they depend on use direct peering. Hence, optimizing for WAN costs is not a priority for OTTER and we leave that for future work.

为什么不针对广域网成本进行优化？

OTTER 并未将金钱成本明确地作为一项优化标准。

在单个国家内，云端的计算和网络成本往往是统一的。大多数 5G 网络部署都是国家规模的。运营商及其所依赖的云服务商之间使用直接对等互联。

因此，优化广域网成本对 OTTER 而言并非优先事项，我们将其留作未来的工作。

Will OTTER negatitvely impact existing WAN TE? OTTER works within the confines of TE in both WANs — the routes available to OTTER are what the underlying TE has exposed to applications. OTTER then can only influence the traffic matrix, which TE systems effectively optimize over. If OTTER-induced demands are sufficiently high, TE systems can change the routes available to applications, including OTTER . OTTER will in turn adapt to these new routes and still pick the best ones for 5G workloads. While there is a feedback loop between OTTER and TE, it is not one that can be exploited for reducing the efficiency of existing TE. In Appendix H, we explore the value of using paths that underlying TE systems create but do not expose to applications.

OTTER 会对现有的广域网流量工程（TE）产生负面影响吗？

OTTER 在两个广域网现有的流量工程（TE）框架内工作——OTTER 可用的路由，正是底层 TE 系统暴露给应用的那些。

因此， OTTER 只能影响流量矩阵，而 TE 系统正是针对流量矩阵进行有效优化的

如果 OTTER 引发的流量需求足够高，TE 系统可能会改变提供给应用（包括 OTTER）的路由。OTTER 会反过来适应这些新路由，并仍然为 5G 工作负载选择最佳路径。

虽然 OTTER 和 TE 之间存在一个反馈循环，但这并不会被利用来降低现有 TE 的效率。在附录 H 中，我们探讨了使用那些底层 TE 系统已创建但未暴露给应用的路径所能带来的价值。

Related Work

Overlay networks. Researchers have proposed overlay networks to improve underlying network availability, performance, or security [4, 5, 17, 22, 36, 59, 70, 77]. Recently, Skyplane [40] proposed a cloud overlay for inter-cloud bulk transfers between object stores, and selects routes that balance price and throughput. Routes and VMs are chosen using static throughput and cost measurements and participate in store-and-forward queuing. Hercules [29] is another bulk-transfer tool for overlays. CRONets [18] deploys overlay nodes on cloud networks and leverages MPTCP through overlay paths to improve throughput. CloudCast [72] focuses on latency improvement using cloud overlays.

OTTER also uses an overlay network to interconnect a 5G operator WAN and a cloud WAN. Unlike prior work, OTTER optimizes for various 5G NFs and applications, each of which can have different performance requirements, not just bulk data transfer. OTTER optimizes network paths online by using dynamic measurements for a wide range of metrics, including RTT, jitter, and loss. We achieve higher inter-cloud, per-VM throughput than prior work [40], in part because our security and privacy oriented topology design does not suffer from limits that some clouds impose on bandwidth to pub-lic IPs (Appendix B). OTTER also allocates both the path of flows as well as the compute destination.

覆盖网络（Overlay networks）

研究人员已经提出了多种覆盖网络，用以改善底层网络的可用性、性能或安全性 [4, 5, 17, 22, 36, 59, 70, 77]。最近，Skyplane [40] 提出了一个用于对象存储之间跨云批量传输的云覆盖网络，它通过选择路由来平衡价格和吞吐量。其路由和虚拟机的选择基于静态的吞吐量和成本测量，并参与存储转发排队。Hercules [29] 是另一个用于覆盖网络的批量传输工具。CRONets [18] 在云网络上部署覆盖节点，并利用 MPTCP 通过覆盖路径来提高吞-吐量。CloudCast [72] 则专注于使用云覆盖网络改善延迟。

OTTER 也使用覆盖网络来互连一个 5G 运营商广域网和一个云广域网。与先前的工作不同，OTTER 针对各种 5G 网络功能（NF）和应用进行优化，这些应用各有不同的性能需求，而不仅仅是批量数据传输。OTTER 通过使用包括 RTT、抖动和丢包在内的多种指标的动态测量，来在线优化网络路径。我们实现了比先前工作 [40] 更高的跨云、单虚拟机吞吐量，部分原因是我们以安全和隐私为导向的拓扑设计，避免了某些云对公有 IP 带宽施加的限制（附录 B）。OTTER 还同时分配流量的路径和计算资源的目的地。

Cloud TE. Existing cloud TE systems (such as Taiji [19], COPE [87], SWAN [39], B4 [41], BwE [46], OneWAN [45], Cascara [76], Espresso [94]) maximize network utilization or bandwidth fairness, or minimize cost for traffic grouped by coarse-grained priority classes. OTTER is not a replacement for existing TE, as underlay paths will still be subject existing TE. OTTER is an enhancement that can optimize at the granularity required for 5G applications by selecting overlay paths between multiple destinations across WANs. Additionally, OTTER performs dynamic flow placement and considers server resource capacities at endpoints, while TE periodically allocates batched traffic predictions and only considers network resource constraints. While some recent TE work [50] to optimize more fine-graned flows for applications, this optimization still takes place within a single WAN, and does not consider the multi-WAN setting of OTTER.

云流量工程（Cloud TE）

现有的云流量工程系统（如 Taiji [19], COPE [87], SWAN [39], B4 [41], BwE [46], OneWAN [45], Cascara [76], Espresso [94]）致力于最大化网络利用率或带宽公平性，或者为按粗粒度优先级类别分组的流量最小化成本。

OTTER 并非要替代现有的 TE，因为底层路径仍将受现有 TE 的管辖。

OTTER 是一种增强机制，它通过在跨广域网的多个目的地之间选择覆盖路径，能够实现 5G 应用所需的优化粒度。此外，OTTER 执行动态的流量放置，并考虑端点的服务器资源容量，而 TE 则是周期性地分配批量的流量预测，并且只考虑网络资源约束。

虽然近期一些 TE 工作 [50] 开始为应用优化更细粒度的流量，但这种优化仍局限于单个广域网内，并未考虑 OTTER 所处的多广域网环境。

Multi-WAN collaboration. Unlike TE systems, OTTER optimizes traffic that spans two WANs. Prior work (Nexit [48], Wiser [47], P4P [90], Flow Director [68]) attempt to facilitate coordination between two independent entities by allowing them to exchange information while preserving privacy. Nexit and Wiser allow two ISPs to route traffic efficiently, despite competing interests. Nexit uses a negotiationbased approach, while downstream ISPs using Wiser tag routing advertisements with costs used by upstream ISPs. P4P improves peer-to-peer application performance by allowing ISPs to share performance and topology information with applications. Flow Director uses ALTO [2] to share information between ISPs and CDNs to optimize content placement. However, these works require negotiation to ensure mutually beneficial routing. OTTER makes use of the fact that, for 5G networks, these incentives are already aligned operators pay to deploy resources on cloud networks to facilitate better QoS for 5G flows. OTTER does not require any buy-in from cloud platforms, so routing and resource placement can be done without coordination.

多广域网协作

与 TE 系统不同，OTTER 优化的是跨越两个广域网的流量。先前的工作（Nexit [48], Wiser [47], P4P [90], Flow Director [68]）试图通过允许两个独立实体在保护隐私的同时交换信息，来促进它们之间的协调。Nexit 和 Wiser 允许两个 ISP 在存在利益竞争的情况下高效地路由流量。Nexit 使用基于协商的方法，而使用 Wiser 的下游 ISP 则在其路由通告中标记成本，供上游 ISP 使用。P4P 通过允许 ISP 与应用共享性能和拓扑信息，来提升点对点应用的性能。Flow Director 使用 ALTO [2] 在 ISP 和 CDN 之间共享信息，以优化内容放置。

然而，这些工作都需要通过协商来确保互利共赢的路由。

OTTER 则利用了一个事实：对于 5G 网络，各方激励已经天然一致——运营商付费在云网络上部署资源，以为 5G 流量提供更好的服务质量（QoS）。

OTTER 不需要云平台的任何额外支持，因此可以在无需协调的情况下完成路由和资源放置。

Better routing. Attempts to improve interdomain routing include multipath extensions to BGP [38, 92] and interdomain bandwidth reservation such as N-Tube [88]. New routing protocols that allow nodes to select paths that optimize multiple metrics include Routing on Multiple Optimality Criteria [78]. Such approaches may improve path diversity or even path selection. However, they must still optimize routing based on static metrics such as BGP AS path length or width, and cannot adjust based on dynamic metrics such as loss or latency. Deploying any of these technologies requires changes to BGP, which has proven to be a major practical impediment. SCION [95] is a novel Internet routing architecture that seeks to provide high security and availability. However, it requires ISP buy-in to deploy border routers to participate. RouteScout [7], PAINTER [43] and SCULP-TOR [6] reroute traffic based on traffic performance degradation. However, such optimizations are performed locally or per prefix. In contrast, OTTER optimizes the entire overlay according to the performance demands of all flows, allocates per flow, and does not require deploying new devices.

路由改进

改进域间路由的尝试包括 BGP 的多路径扩展 [38, 92] 和域间带宽预留（如 N-Tube [88]）。允许节点选择能优化多个指标的路径的新路由协议包括“多重最优性标准路由”[78]。这些方法可能会改善路径多样性甚至路径选择。然而，它们仍然必须基于静态指标（如 BGP AS 路径长度或宽度）来优化路由，无法根据动态指标（如丢包或延迟）进行调整。部署这些技术中的任何一种都需要对 BGP 进行更改，这在实践中已被证明是一个主要的障碍。

SCION [95] 是一种新颖的互联网路由架构，旨在提供高安全性和可用性。然而，它需要 ISP 的支持来部署边界路由器以参与其中。RouteScout [7]、PAINTER [43] 和 SCULPTOR [6] 根据流量性能下降情况来重新路由流量。然而，这类优化是在本地或按前缀进行的。

相比之下，OTTER 根据所有流量的性能需求来优化整个覆盖网络，按单个流量进行分配，并且不需要部署新设备。

Better path selection at the edge. In Tango [16], edge networks coordinate to expose more paths, enable better measurements, and route traffic efficiently. Rather than coordinate paths between edge networks, OTTER directly sets up overlay paths between the cloud provider and 5G operators by orchestrating resources on the cloud network. OTTER also routes traffic according to application-specific service demands and relies only on existing cloud network technologies. Other work [42] selects paths between a private WAN and the public Internet for video conferencing applications (e.g., Teams) based on extensive measurement data. Like OTTER, this work jointly optimizes the path and destination DC selection. However, unlike OTTER, it neither considers the general setting of 5G applications nor the coordination of paths among multiple ASNs. Any of these mechanisms that expose multiple paths to an endpoint could be utilized at the transport layer by multipath TCP.

边缘侧的路径选择优化

在 Tango [16] 中，边缘网络相互协调以暴露更多路径、实现更好的测量并高效路由流量。OTTER 不是在边缘网络之间协调路径，而是通过在云网络上编排资源，直接在云提供商和 5G 运营商之间建立覆盖路径。OTTER 还根据应用特定的服务需求来路由流量，并且只依赖现有的云网络技术。其他工作 [42] 基于大量的测量数据，为视频会议应用（如 Teams）在私有广域网和公共互联网之间选择路径。与 OTTER 类似，这项工作也联合优化了路径和目的地数据中心的选择。然而，与 OTTER 不同的是，它既没有考虑 5G 应用的通用场景，也没有考虑多个 ASN 之间的路径协调。任何这些向端点暴露多条路径的机制，都可以在传输层被多路径 TCP（multipath TCP）所利用。

Virtual Network Function (VNF) placement. Systems for VNF placement [62,63,82] optimize for metrics such as cost and CPU utilization of NFs. However, unlike OTTER, they do not consider the QoS needs of individual flows, and are evaluated on simulated setups. VNF placement systems periodically optimize for coarse-grained objectives, according to predicted future load, but OTTER dynamically places flows and resources, on-demand, according to customizable fine-grained service objectives.

虚拟网络功能（VNF）放置

VNF 放置系统 [62, 63, 82] 针对成本和 NF 的 CPU 利用率等指标进行优化。然而，与 OTTER 不同，它们不考虑单个流量的 QoS 需求，并且其评估是在模拟环境中进行的。VNF 放置系统根据预测的未来负载，周期性地针对粗粒度目标进行优化，而 OTTER 则根据可定制的细粒度服务目标，按需、动态地放置流量和资源。

Conclusions

The next generation of 5G networks includes new frequency bands and radio technology, equipping users with access links so fast that their E2E performance will become bottlenecked by inter-domain paths [91] rather than last mile access links. Moreover, next generation 5G networks will rely on hyperscalers to deploy software-based 5G NFs [61] and interactive applications. Thus, we need to improve the performance of paths across operator and cloud WANs.

We demonstrate the value of constructing such paths over two commercial WANs. OTTER achieves 13% higher average throughput or 6-10 Gbps in the maximum, RTT reduction as much as 42 ms within the US, jitter reduction by 45% on average, and 0.06% lower packet loss. We designed and implemented a scalable algorithm to jointly optimize the placement of workloads on available paths and compute resources. It allocates 26%–45% more bytes on the network than good baselines while also satisfying the service demands of more flows. While we focus on the urgency of 5G, our research contributions may be relevant in other up-coming access technologies such as 6G.

OTTER unlocks avenues of future work. Underlay paths that are private to one WAN can provide even more diversity in end-to-end path performance than we have shown – how can they be leveraged? How will OTTER work in topologies where a 5G operator relies on two or more hyperscalers?

下一代 5G 网络包含新的频段和无线电技术，为用户配备了极快的接入链路，以至于他们的 端到端（E2E）性能 瓶颈将更多地出现在 域间路径 上 [91]，而不是“最后一公里”的接入链路。此外，下一代 5G 网络将依赖超大规模云服务商来部署基于软件的 5G 网络功能（NF）[61] 和交互式应用。因此，我们需要提升跨运营商和云广域网路径的性能。

我们展示了在两个商业广域网上构建此类路径的价值。OTTER 实现了平均吞吐量提升 13%（最大可达 6-10 Gbps），在美国境内 RTT 最多减少 42 毫秒，抖动平均降低 45%，丢包率降低 0.06%。我们设计并实现了一种可扩展的算法，用于联合优化工作负载在可用路径和计算资源上的放置。与优秀的基线方法相比，它在网络上多分配了 26%–45% 的字节，同时满足了更多流量的服务需求。虽然我们关注的是 5G 的紧迫性，但我们的研究贡献也可能与其他新兴的接入技术（如 6G）相关。

OTTER 为未来的工作开辟了新的途径。例如，单一广域网私有的底层路径可以提供比我们所展示的更丰富的端到端路径性能多样性——如何才能利用它们？当一个 5G 运营商依赖两个或更多超大规模云服务商时，OTTER 将如何工作？