Mahimahi: Accurate Record-and-Replay for HTTP

依旧是先让gemini读一遍

• 背景: HTTP 已成为客户端-服务器应用(如网页加载, 移动 App, 视频流等)的事实标准
• 挑战: 在受控的网络环境下精确评估这些应用的性能(如页面加载时间 PLT)具有挑战性, 现有工具在模拟现代网页的多服务器架构及环境隔离方面存在不足

Mahimahi 是一个能够录制 HTTP 应用流量, 并在后续模拟的网络条件下进行回放的框架. 它主要具有以下三个特点:

• 高准确性(Accuracy):
  • 不同于将所有请求重定向到单一服务器的传统工具, Mahimahi 会为录制期间涉及的每个不同服务器创建一个对应的虚拟服务器
  • 从而精确还原真实世界中网页的多服务器通信模式
• 环境隔离(Isolation):
  • 利用 Linux 的 network namespaces, Mahimahi 将自身的网络流量与主机系统及其他实例隔离
  • 使得多个 Mahimahi 实例可在同一台机器上并发运行而互不干扰
• 可组合性与扩展性(Composability & Extensibility):
  • 它被设计为一组 UNIX Shell, 用户可以将 unmodified 的应用二进制文件在这些 Shell 中运行, 并根据需要进行嵌套

Mahimahi 的功能通过不同的"Shell"组件实现, 这些 Shell 可以相互嵌套:

• RecordShell: 录制进程产生的所有 HTTP 流量
• ReplayShell: 使用本地模拟服务器回放已录制的内容
• DelayShell: 模拟固定的网络传播延迟
• LinkShell: 模拟具有固定或多变容量的网络链路(如蜂窝网络轨迹)

Introduction

HTTP is the de facto communication protocol for client-server applications today [27]. Beyond its widespread use as an application-layer protocol for loading Web pages, HTTP is now used for mobile apps [22], video streaming [14], and instant messaging [19].

It is useful to evaluate the performance of these applications under controlled experimental conditions. For example, browser developers may wish to evaluate how changes to their document object model (DOM) and JavaScript parsers affect Web page load times, while network-protocol designers might want to understand the application-level impact of new multiplexing protocols like QUIC [30]. Similarly, a mobile app developer may wish to determine the user-perceived latency [28] for user interactions over different wireless networks.

Motivated by such questions, we developed Mahimahi 1 , a framework to record traffic from applications that use HTTP, and later replay recorded traffic under emulated network conditions. Mahimahi works with any application that uses HTTP or HTTPS. Application clients (Web browsers, video players, and apps within mobile-phone emulators) can be run unmodified within Mahimahi. Additionally, Mahimahi's replay semantics can be extended to support the server-side logic of many applications, such as YouTube.

Mahimahi has three notable features that distinguish it from other record-and-replay tools such as Google's web-page-replay [11] and Fiddler [34]:

1. Accuracy: Mahimahi is careful about emulating the multi-server nature of Web applications. Instead of responding to all requests from a single server, Mahimahi creates a separate server for each distinct server contacted while recording. We find that emulating multiple servers is a key factor in accurately measuring Web page load times ( § 4.1).

2. Isolation: Using Linux's network namespaces [7], Mahimahi isolates its traffic from the rest of the host system, allowing multiple instances of its shells to run in parallel with no mutual interference ( § 4.2). Because other tools modify the network configuration of the entire host [11, 34], they cannot provide this feature.

3. Composability and extensibility: Mahimahi is structured as a set of UNIX shells, allowing the user to run unmodified client binaries within each shell. RecordShell allows a user to record all HTTP traffic for any process spawned within it. ReplayShell replays recorded content using local servers that emulate the application servers. To emulate network conditions, Mahimahi includes DelayShell, which emulates a fixed network propagation delay, and LinkShell, which emulates both fixed-capacity and variable-capacity links. These shells can be nested within one another, allowing the user to flexibly experiment with many different network configurations. Mahimahi makes it easy to modify these shells and add new ones; e.g., to record-and-replay YouTube videos, emulate packet losses, implement active queue management algorithms, etc. ( § 4.3). We used Mahimahi to evaluate Web multiplexing protocols. We were able to easily extend Mahimahi to support QUIC, a new protocol in active development at Google. We compared HTTP/1.1, SPDY [3], and QUIC to a hypothetical optimal protocol and found that all three are suboptimal. We then used Mahimahi to understand the shortcomings of these multiplexing protocols. We found that each protocol is suboptimal because of the request serialization caused by source-level object dependencies present in today's Web pages. Resolving each dependency requires an RTT between the client and origin Web servers; Mahimahi allowed us to pinpoint the problem because we were able to conduct a large number of emulation experiments under different network conditions quickly.

We used these findings to develop Cumulus, a new system to improve HTTP application performance, especially on long-delay paths. Cumulus has two components: the "Remote Proxy," a headless browser that the user runs on a well-provisioned cloud server, and the "Local Proxy," a transparent, caching HTTP proxy that runs on the user's computer. These two components cooperate to move the resolution of object dependencies closer to origin Web servers, reducing the effective RTT. Mahimahi's shell structure allowed us to implement Cumulus with ease by adapting RecordShell to implement the Local Proxy.

To evaluate Cumulus, we used Mahimahi yet again, this time on the same large number of network configurations used to understand HTTP/1.1, SPDY, and QUIC. Our key result is that page load times with Cumulus do not degrade dramatically with increasing round-trip times (RTTs), unlike the other multiplexing protocols. Some representative results are shown in Table 1. We have also evaluated Cumulus on AT&T's live cellular network in Boston, finding that it outperforms existing Web accelerators such as Opera Turbo [1] and Chrome Data Compression Proxy [15].

Mahimahi has been used in other projects, including an analysis of mobile app traffic patterns to compare single-path and multi-path TCP [13], and an evaluation of intelligent network selection schemes [12]. Mahimahi has also been used in Stanford's graduate networking course [41] and at Mozilla to understand and improve networking within browsers. Mahimahi and our experimental data are available under an open source license at http://mahimahi.mit.edu. Mahimahi has been queued for inclusion with the Debian distribution.

HTTP 已成为当今客户端-服务器应用事实上的通信协议 [27]. 除了作为加载网页的应用层协议被广泛使用外, HTTP 目前还被应用于移动应用 [22], 视频流媒体 [14] 以及即时通讯 [19] 等领域.

在受控的实验条件下评估这些应用的性能具有重要意义:

1. 浏览器开发人员可能希望评估其文档对象模型(DOM)和 JavaScript 解析器的变更如何影响网页加载时间(PLT)
2. 网络协议设计者则可能希望了解如 QUIC [30] 等新型多路复用协议对应用层产生的影响
3. 移动应用开发者可能需要确定在不同无线网络环境下, 用户交互所产生的用户感知延迟 [28]

受此类问题的启发, 我们开发了 Mahimahi -- 这是一个用于录制 HTTP 应用流量, 并随后在模拟网络条件下进行回放的框架

Mahimahi 适用于任何使用 HTTP 或 HTTPS 的应用程序. 应用客户端(如浏览器, 视频播放器以及手机模拟器内的 App)无需任何修改即可在 Mahimahi 中运行. 此外, Mahimahi 的回放语义具有可扩展性, 能够支持多种应用(如 YouTube)的服务器端逻辑.

与 Google 的 web-page-replay [11] 和 Fiddler [34] 等其他录制回放工具相比, Mahimahi 具有三个显著特点:

1. 准确性(Accuracy):
   • Mahimahi 能够精确模拟 Web 应用的多服务器特性. 它并非由单一服务器响应所有请求, 而是为录制期间涉及的每个独立服务器分别创建对应的虚拟服务器
   • 我们发现, 模拟多服务器环境是准确测量网页加载时间的关键因素(§ 4.1)
2. 隔离性(Isolation):
   • 利用 Linux 的网络命名空间(network namespaces)[7], Mahimahi 将其流量与主机系统的其余部分隔离, 从而允许其多个 Shell 实例并行运行且互不干扰(§ 4.2)
   • 由于其他工具通常需要修改整个主机的网络配置 [11, 34], 因此无法提供此特性
3. 可组合性与扩展性(Composability and Extensibility):
   • Mahimahi 由一系列 UNIX Shell 构成, 允许用户在各 Shell 中运行未经修改的客户端二进制文件:
   • RecordShell 负责录制其衍生的任何进程所产生的所有 HTTP 流量
   • ReplayShell 则利用模拟应用服务器的本地服务器来回放录制内容
   • 为了模拟网络状况, Mahimahi 包含了用于模拟固定网络传播延迟的 DelayShell, 以及用于模拟固定带宽和可变带宽链路的 LinkShell
   • 这些 Shell 可以相互嵌套, 使用户能够灵活地实验多种不同的网络配置
   • 此外, Mahimahi 易于修改和扩展, 例如录制并回放 YouTube 视频, 模拟数据包丢失, 实现主动队列管理(AQM)算法等(§ 4.3)

我们利用 Mahimahi 评估了网页多路复用协议, 并轻松实现了对 Google 正在开发的 QUIC 协议的支持. 通过将 HTTP/1.1, SPDY [3] 和 QUIC 与假设的最优协议进行对比, 我们发现这三者均处于 suboptimal 状态.

借助 Mahimahi, 我们进一步探究了这些多路复用协议的缺陷, 发现其性能不佳的原因在于当今网页中存在的源码级对象依赖性所导致的请求序列化.

解决每一项依赖都需要在客户端与源服务器之间进行一次往返(RTT); 得益于 Mahimahi 能够快速进行大量不同网络条件下的模拟实验, 我们得以精确锁定问题的根源.

基于上述发现, 我们开发了 Cumulus 系统, 旨在提升 HTTP 应用的性能, 特别是在长延迟路径下的表现.

Cumulus 由两个组件组成:

1. 运行在配置良好的云服务器上的无头浏览器"远程代理(Remote Proxy)"
2. 运行在用户计算机上的透明缓存 HTTP 代理"本地代理(Local Proxy)"

两者协作将对象依赖的解析过程移至更接近源服务器的位置, 从而降低了有效 RTT.

Mahimahi 的 Shell 结构使得我们能够通过改造 RecordShell 轻松实现本地代理, 从而完成了 Cumulus 的构建.

在评估 Cumulus 时, 我们再次使用了 Mahimahi, 并采用了与分析 HTTP/1.1, SPDY 及 QUIC 时相同的大量网络配置.

核心实验结果表明, 与其它多路复用协议不同, Cumulus 的网页加载时间不会随着 RTT 的增加而剧烈恶化. 表 1 展示了部分代表性结果. 我们还在波士顿的 AT&T 现网蜂窝网络中对 Cumulus 进行了评估, 结果显示其性能优于 Opera Turbo [1] 和 Chrome 数据压缩代理 [15] 等现有网页加速器.

Mahimahi 已被应用于多个研究项目, 包括分析移动应用流量模式以对比单路径与多路径 TCP [13], 以及评估智能网络选择方案 [12]. 此外, Mahimahi 已被引入斯坦福大学的研究生网络课程 [41], 并被 Mozilla 用于理解和优化浏览器内部网络机制.

Mahimahi 及其实验数据均已通过开源许可证发布(http://mahimahi.mit.edu), 并已排队等待进入 Debian 发行版仓库.

Related Work

This section describes prior work on Web record-andreplay tools and network emulation frameworks.

2.1 Record-and-replay tools

The most prominent Web page record-and-replay tools are Google’s web-page-replay [11] and Telerik’s Fiddler [34]. web-page-replay uses DNS indirection to intercept HTTP traffic during both record and replay, while Fiddler adjusts the system-wide proxy settings in the Windows networking stack. With both tools, all HTTP requests from a browser are sent to a proxy server that records the request and forwards it to the corresponding origin server. Responses also pass through the proxy server and are recorded and sent back to the browser.

Both tools suffer from two shortcomings. First, because they serve all HTTP responses from a single server, neither tool preserves the multi-server nature of Web applications. Consolidating HTTP resources onto a single server during replay allows browsers to use a single connection to fetch all resources, which is impossible when resources are on different servers. Mahimahi faithfully emulates the multi-server nature of Web applications, leading to more accurate measurements ( § 4.1).

Second, these tools do not provide isolation: the network conditions that web-page-replay and Fiddler emulate affect all other processes on the host machine. These include the link rate, link delay, and DNS indirection settings for web-page-replay, and the system proxy address, specified in the Windows networking stack, for Fiddler. During replay, this lack of isolation could lead to inaccurate measurements if cross traffic from other processes reaches the replaying proxy server. The lack of isolation also precludes multiple independent instances of web-page-replay or Fiddler from running concurrently—a useful feature for expediting experiments, or for experimenting with different applications concurrently. Mahimahi overcomes these problems by using Linux’s network namespaces [7].

Other record-and-replay tools such as Timelapse/Dolos [8] and WaRR [6] target reproducible application debugging by capturing program executions (including user input and activity) and replaying them, while providing popular debugging abstractions including breakpoints. These systems are complementary to Mahimahi; they can be run within ReplayShell, which ensures that served HTTP content, including dynamic content such as JavaScript, does not vary during replay.

目前最主流的网页录制与回放工具是 Google 的 web-page-replay [11] 和 Telerik 的 Fiddler [34]:

1. web-page-replay 在录制与回放阶段均采用 DNS 重定向(DNS indirection)技术来拦截 HTTP 流量
2. Fiddler 则通过调整 Windows 网络栈中的系统级代理设置 来实现

在这两种工具中, 来自浏览器的所有 HTTP 请求都会被发送到一个代理服务器, 该服务器负责录制请求并将其转发至相应的源服务器(origin server). 响应内容同样经由该代理服务器进行录制, 随后返回给浏览器.

然而, 这两种工具都存在两个缺陷:

1. 首先, 由于它们通过单一服务器提供所有 HTTP 响应, 因此无法保留 Web 应用的多服务器特性

   • 在回放期间将 HTTP 资源整合到单一服务器上, 会导致浏览器可以使用单个连接获取所有资源, 而这在资源分布于不同服务器的真实场景中是不可能的
   • Mahimahi 能够忠实地模拟 Web 应用的多服务器特性, 从而获得更准确的测量结果(§ 4.1)

2. 其次, 这些工具不具备隔离性: web-page-replay 和 Fiddler 所模拟的网络条件会影响宿主机上的所有其他进程

   • 这包括 web-page-replay 的链路速率, 链路延迟和 DNS 重定向设置, 以及 Fiddler 在 Windows 网络栈中指定的系统代理地址.
   • 在回放期间, 这种隔离性的缺失可能会导致测量结果不准(如果来自其他进程的交叉流量到达了执行回放的代理服务器).
   • 此外, 缺乏隔离性还导致无法并行运行多个独立的 web-page-replay 或 Fiddler 实例, 而并行运行对于加速实验或同时开展不同应用的实验非常有益.
   • Mahimahi 通过利用 Linux 的网络命名空间(network namespaces)[7] 解决了这些问题

其他录制与回放工具, 如 Timelapse/Dolos [8] 和 WaRR [6], 主要针对可重现的应用调试:

它们通过捕获程序执行过程(包括用户输入和活动)并进行回放, 同时提供包括断点在内的常用调试抽象.

这些系统与 Mahimahi 是互补关系: 它们可以在 ReplayShell 中运行, 从而确保在回放期间所提供的 HTTP 内容(包括 JS 等动态内容)保持不变

2.2 Emulation Frameworks

Tools like dummynet [10] and netem [20] emulate network conditions including link rate, one-way delay, and stochastic loss. Mahimahi uses its own network emulation shells, LinkShell and DelayShell. Unlike dummynet and netem, LinkShell can emulate variable-rate cellular links, in addition to static link rates, because it runs over packet-delivery traces. Mahimahi also allows users to evaluate new in-network algorithms (instead of Drop Tail FIFO) by modifying the source code of LinkShell. A similar evaluation using web-page-replay would require developing a new kernel module for dummynet, a more complicated task.

Mahimahi is general enough to record and replay any HTTP client-server application under emulated conditions. It is, however, limited in that it only emulates one physical client connected to an arbitrary number of servers. Mahimahi supports a single shared link from the client to all servers, as well as multi-homed clients ( § 5.5), allowing the evaluation of multipath-capable transport protocols such as MPTCP [25]. Mahimahi cannot emulate arbitrary network topologies such as transit-stub [9]; for emulating applications over such topologies, tools like Mininet [21] are more suitable.

诸如 dummynet [10] 和 netem [20] 等工具可以模拟包括链路速率, 单向延迟和随机丢包在内的网络状况.

Mahimahi 使用其特有的网络模拟 Shell, 即 LinkShell 和 DelayShell.

与 dummynet 和 netem 不同, LinkShell 能够运行在数据包传输轨迹(packet-delivery traces)之上, 因此除了静态链路速率外, 它还能模拟速率波动的蜂窝网络链路.

Mahimahi 还允许用户通过修改 LinkShell 的源代码来评估新的网络内部算法(而非仅限于丢弃尾部 FIFO 算法). 若使用 web-page-replay 进行类似的评估, 则需要为 dummynet 开发一个新的内核模块, 其任务复杂度显著更高.

• Mahimahi 的通用性足以实现在模拟条件下对任何 HTTP 客户端-服务器应用进行录制与回放
  • 然而, 它的局限性在于仅模拟单个物理客户端连接到任意数量服务器的场景
• Mahimahi 支持从客户端到所有服务器的单条共享链路, 同时也支持多宿主(multi-homed)客户端(§ 5.5), 从而允许对多路径传输协议(如 MPTCP [25])进行评估
• Mahimahi 无法模拟复杂的任意网络拓扑(如 transit-stub 拓扑 [9]); 若需在涉及此类拓扑的环境下评估应用, Mininet [21] 等工具则更为适用

MAHIMAHI

Mahimahi is structured as a set of four UNIX shells, allowing users to run unmodified client binaries within each shell. Each shell creates a new network namespace for itself prior to launching the shell. Quoting from the man page, "a network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices" [7]. A separate network namespace minimizes disruption to the host machine during recording, prevents accidental download of resources over the Internet during replay, and ensures that the host machine is isolated from all network configuration changes that are required to evaluate an application.

RecordShell ( § 3.1) records all HTTP traffic for subsequent replay. ReplayShell ( § 3.2) replays previously recorded HTTP content. DelayShell ( § 3.3) delays all packets originating from the shell by a user-specified amount and LinkShell ( § 3.4) emulates a network link by delivering packets according to a user-specified packet-delivery trace. All components of Mahimahi run on a single physical machine (which we call the host machine) and can be arbitrarily composed with each other. For example, to replay recorded content over a cellular network with a 10 ms minimum RTT, one would run a client application inside DelayShell inside LinkShell inside ReplayShell.

Mahimahi 被设计为一组由四个 UNIX Shell 构成的套件, 允许用户在每个 Shell 中运行未经修改的客户端二进制文件.

每个 Shell 在启动前都会为其自身创建一个新的网络命名空间(Network Namespace).

引用自手册页(man page)的定义: "网络命名空间在逻辑上是网络栈的另一份副本, 拥有其独立的路由表, 防火墙规则和网络设备"[7]. 采用独立的网络命名空间能够最大限度地减少录制期间对宿主机的干扰, 防止回放期间意外通过互联网下载资源, 并确保宿主机与评估应用所需的各类网络配置更改保持隔离.

• RecordShell(§ 3.1)负责录制所有 HTTP 流量以供后续回放
• ReplayShell(§ 3.2)用于回放先前录制的 HTTP 内容
• DelayShell(§ 3.3)按用户指定的数值延迟所有源自该 Shell 的数据包
• LinkShell(§ 3.4)通过根据用户指定的数据包传输轨迹(trace)交付数据包来模拟网络链路

Mahimahi 的所有组件均运行在单台物理机器(我们称之为宿主机)上, 且可以相互进行任意组合. 例如, 若要在具有 10 毫秒最小往返时延(RTT)的蜂窝网络上回放录制内容, 用户可以在 DelayShell 中运行客户端应用, 而 DelayShell 嵌套在 LinkShell 中, LinkShell 再嵌套在 ReplayShell 中.

3.1 RecordShell

RecordShell (Figure 1a) records HTTP data and stores it on disk in a structured format for subsequent replay. On startup, RecordShell spawns a man-in-the-middle proxy on the host machine to store and forward all HTTP traffic both to and from an application running within RecordShell. To operate transparently, RecordShell adds an iptable rule that forwards all TCP traffic from within RecordShell to the man-in-the-middle proxy.

When an application inside RecordShell attempts to connect to a server, it connects to the proxy instead. The proxy then establishes a TCP connection with the application, uses the SO ORIGINAL DST socket option to determine the server’s address for the connection, and connects to the server on the application’s behalf. An HTTP parser running at the proxy captures traffic passing through it to parse HTTP requests and responses from TCP segments. Once an HTTP request and its corresponding response have both been parsed, the proxy writes them to disk, associating the request with the response. At the end of a record session, a recorded directory consists of a set of files, one for each HTTP request-response pair seen during that session.

SSL traffic is handled similarly by splitting the SSL connection and establishing two separate SSL connections: one between the proxy and the application and another between the proxy and the server. The proxy can establish a secure connection with the application in two ways. In the first approach, RecordShell’s proxy uses a new Root CA, in the same way Fiddler does [35]. Clients must manually trust this CA once and individual certificates are signed by this Root CA.

Another approach is for RecordShell’s proxy to use a self-signed certificate. This approach may trigger warnings within applications that only accept certificates signed by any one of a list of trusted Certificate Authorities (CAs). Most modern browsers allow users to disable these warnings. Certain applications, such as mobile phone emulators, do not allow these warnings to be disabled; the first approach handles these applications [31].

RecordShell(图 1a)负责录制 HTTP 数据并将其以结构化格式存储在磁盘上. 启动时, RecordShell 会在宿主机上派生一个中间人(MITM)代理, 用于存储并转发与 RecordShell 内运行的应用之间的所有 HTTP 流量.

为了实现透明化操作, RecordShell 添加了一条 iptables 规则, 将所有来自 RecordShell 内部的 TCP 流量转发至该中间人代理.

当 RecordShell 内的应用尝试连接服务器时, 它实际上会连接到代理. 随后, 代理与应用建立 TCP 连接, 并利用 SO_ORIGINAL_DST 套接字选项确定该连接的目标服务器地址, 代表应用与服务器建立连接.

运行在代理端的 HTTP 解析器会捕获流经的流量, 从 TCP 段中解析 HTTP 请求与响应. 一旦 HTTP 请求及其对应的响应均被解析完成, 代理将其写入磁盘, 并将请求与响应进行关联. 在录制会话结束时, 录制目录包含一组文件, 每个文件对应会话期间观测到的一对 HTTP 请求-响应.

对于 SSL 流量, 处理方式类似: 通过拆分 SSL 连接, 建立两个独立的 SSL 连接--一个存在于代理与应用之间, 另一个存在于代理与服务器之间. 代理可以通过两种方式与应用建立安全连接. 第一种方法中, RecordShell 代理使用一个新的根证书颁发机构(Root CA), 其机制与 Fiddler 类似 [35]. 客户端必须手动信任该 CA 一次, 随后各个证书将由该根 CA 签名.

另一种方法是 RecordShell 代理使用自签名证书. 这种方法可能会在仅接受受信任 CA 列表签名的应用中触发警告. 大多数现代浏览器允许用户禁用此类警告; 而对于某些不允许禁用警告的应用(如手机模拟器), 则需采用第一种方法 [31].

3.2 ReplayShell

ReplayShell (Figure 1b) also runs on the test machine and mirrors the server side of Web applications using content recorded by RecordShell. ReplayShell accurately emulates the multi-server nature of most Web applications today by spawning an Apache 2.2.22 Web server for each distinct IP/port pair seen while recording. Each server handles HTTPS traffic using Apache’s mod ssl module and may be configured to speak HTTP/1.1 or SPDY (using mod spdy).

To operate transparently, ReplayShell binds each Apache server to the same IP address and port number as its recorded counterpart. To do so, ReplayShell creates a separate dummy (virtual) interface for each distinct server IP. These interfaces can have arbitrary IPs because they are in a separate network namespace.

All client requests are handled by one of ReplayShell’s servers, each of which can read all of the previously recorded content. Each server redirects all incoming requests to a CGI script using Apache’s mod rewrite module. The CGI script on each server compares each incoming HTTP request to the set of all recorded requestresponse pairs to locate a matching request and return the corresponding response. Incoming requests may be influenced by local state present in the client application (e.g. time-sensitive query string parameters) and may not exactly match any recorded request. We handle such requests using a matching heuristic that enforces that some parts of the request must match exactly, while tolerating some degree of imperfection in other parts.

We expect the Host and User-Agent header fields, along with the requested resource (without the query string), to exactly match the corresponding values in some stored request. If multiple stored requests match on these properties, the algorithm selects the request whose query string has the maximal common substring to the incoming query string.

ReplayShell(图 1b)同样运行在测试机上, 利用 RecordShell 录制的内容来镜像 Web 应用的服务器端.

ReplayShell 通过为录制期间观测到的每个独立 IP/端口对派生一个 Apache 2.2.22 Web 服务器, 从而精确模拟当今大多数 Web 应用的多服务器特性. 每个服务器利用 Apache 的 mod_ssl 模块处理 HTTPS 流量, 并可配置为支持 HTTP/1.1 或 SPDY(使用 mod_spdy).

为了实现透明操作, ReplayShell 将每个 Apache 服务器绑定到与其录制副本相同的 IP 地址和端口号上. 为此, ReplayShell 为每个独立的服务器 IP 创建一个单独的哑(虚拟)接口(dummy interface). 由于这些接口处于独立的网络命名空间中, 因此可以拥有任意 IP 地址.

所有客户端请求均由 ReplayShell 的服务器处理, 每个服务器都能读取所有先前录制的内容. 每个服务器利用 Apache 的 mod_rewrite 模块将所有传入请求重定向至一个 CGI 脚本.

各服务器上的 CGI 脚本会将传入的 HTTP 请求与所有已录制的请求-响应对进行对比, 以定位匹配的请求并返回相应的响应. 由于传入请求可能会受到客户端应用本地状态(如对时间敏感的查询字符串参数)的影响, 可能无法与任何录制请求完全匹配.

我们通过一种匹配启发式算法处理此类请求: 该算法强制要求请求的某些部分必须精确匹配, 同时允许其他部分存在一定程度的不一致.

我们要求 Host 和 User-Agent 头部字段, 以及请求资源路径(不含查询字符串)必须与存储的某个请求完全一致. 如果多个存储请求在这些属性上均匹配, 算法将选择其查询字符串与传入请求查询字符串具有最大公共子串的那个请求.

3.3 DelayShell

DelayShell emulates a link with a fixed minimum one-way delay. All packets sent to and from an application running inside DelayShell are stored in a packet queue. A separate queue is maintained for packets traversing the link in each direction. When a packet arrives, it is assigned a delivery time, which is the sum of its arrival time and the user-specified one-way delay. Packets are released from the queue at their delivery time. This technique enforces a fixed delay on a per-packet basis.

DelayShell 模拟具有固定最小单向延迟的链路:

所有发送至 DelayShell 内运行应用或从其发出的数据包都会被存入一个数据包队列

针对链路上每个方向的流量, 系统分别维护一个独立的队列

当数据包到达时, 系统会为其分配一个交付时间, 即到达时间与用户指定的单向延迟之和

数据包在到达其交付时间时从队列中释放

该技术实现了基于单个数据包的固定延迟强制执行

3.4 LinkShell

LinkShell emulates a link using packet-delivery traces. It emulates both time-varying links such as cellular links and links with a fixed link rate. When a packet arrives into the link, it is directly placed into either the uplink or downlink packet queue. LinkShell is trace-driven and releases packets from each queue based on the corresponding packet-delivery trace. Each line in the trace is a packet-delivery opportunity: the time at which an MTU-sized packet will be delivered in the emulation.2 Accounting is done at the byte-level, and each delivery opportunity represents the ability to deliver 1500 bytes. Thus, a single line in the trace file can correspond to the delivery of several packets whose sizes sum to 1500 bytes. Delivery opportunities are wasted if bytes are unavailable at the instant of the opportunity.

LinkShell supports live graphing of network usage and per-packet queuing delay, giving near-instantaneous feedback on the performance of applications and network protocols. Uplink and downlink capacity are calculated using the input packet-delivery traces, while network usage, in each direction, is based on the amount of data that a client application attempts to transmit or receive. Per-packet queuing delay is computed as the time each packet remains in LinkShell’s uplink or downlink queues.

Figure 2 illustrates the downlink network usage of a single Web page load of http://www.cnn.com, using Google Chrome over an emulated Verizon LTE cellular network with a minimum RTT of 100 ms. As shown, Web servers try to exceed the link capacity at around 9.3 seconds into the trace.

LinkShell 利用数据包传输轨迹(packet-delivery traces)来模拟链路: 它既能模拟时变链路(time-varying links, 如蜂窝链路), 也能模拟具有固定链路速率的链路

当数据包到达链路时, 会被直接置入上行或下行数据包队列中

LinkShell 是轨迹驱动(trace-driven)的, 它根据相应的数据包传输轨迹从各个队列中释放数据包.

轨迹文件中的每一行都代表一次数据包传输机会(packet-delivery opportunity): 即在模拟过程中, 一个 MTU 大小的数据包将被交付的时间点.

流量核算是在字节级(byte-level)进行的, 每一次交付机会代表传输 1500 字节的能力. 因此, 轨迹文件中的单行记录可能对应于多个数据包的交付, 只要这些数据包的大小总和为 1500 字节. 如果在传输机会到来的时刻没有可用的数据字节, 该机会就会被浪费.

LinkShell 支持对网络使用情况和单包排队延迟进行实时绘图(live graphing), 从而就应用程序和网络协议的性能提供近乎即时的反馈:

1. 上行和下行容量是根据输入的数据包传输轨迹计算得出的
2. 各个方向上的网络使用情况则是基于客户端应用尝试发送或接收的数据量来确定的
3. 单包排队延迟计算的是每个数据包在 LinkShell 的上行或下行队列中停留的时间长度

图 2 展示了使用 Google Chrome 加载 http://www.cnn.com 单个网页时的下行网络使用情况, 测试环境为模拟的 Verizon LTE 蜂窝网络, 最小往返时延(RTT)为 100 毫秒. 如图所示, Web 服务器在轨迹运行至约 9.3 秒时, 试图发送超过链路容量的数据.

Conclusion

Mahimahi is an accurate and flexible record-and-replay framework for HTTP applications. Mahimahi’s shellbased design makes it composable and extensible, allowing the evaluation of arbitrary applications and network protocols. It accurately emulates the multi-server nature of Web applications during replay, and by isolating its own traffic, allows several instances to run in parallel without affecting collected measurements.

We presented several case studies to evaluate Mahimahi and demonstrate its benefits. These include a study of HTTP/1.1, SPDY, and QUIC under various emulated network conditions. We used Mahimahi both to conduct the experiments and to understand the reasons for the sub-optimality of these protocols. We then used our key finding—that these protocols are suboptimal due to source-level dependencies in Web pages—to design Cumulus. Mahimahi was useful in our implementation of Cumulus, as well as in our experiments to measure its performance. As round-trip times and link rates increase, the performance of Cumulus degrades much slower than previous HTTP multiplexing protocols.

We have released Mahimahi under an open source license at http://mahimahi.mit.edu.

Mahimahi 是一个面向 HTTP 应用的精准且灵活的录制与回放框架

其基于 Shell 的设计赋予了该框架可组合性与可扩展性, 使其能够支持对任意应用程序及网络协议的评估

1. 在回放过程中, 它能够精确模拟 Web 应用的多服务器特性
2. 同时, 通过隔离自身流量, 它允许并行运行多个实例, 且不会对采集到的测量数据造成干扰

我们通过若干案例研究评估了 Mahimahi 并验证了其优势. 其中包括在多种模拟网络条件下对 HTTP/1.1, SPDY 和 QUIC 的研究. 我们不仅利用 Mahimahi 开展了实验, 还借助它深入剖析了这些协议表现出亚优性(suboptimality)的根本原因. 基于我们的关键发现--即这些协议的非最优性源于网页中存在的源码级依赖关系--我们设计了 Cumulus 系统. Mahimahi 在 Cumulus 的实现过程及其性能测量实验中均发挥了重要作用. 实验结果表明, 随着往返时延(RTT)和链路速率的增加, Cumulus 的性能衰退速度显著低于先前的 HTTP 多路复用协议.

我们已通过开源许可证在 http://mahimahi.mit.edu 发布了 Mahimahi.