Compatibility Layer¶

The first step towards achieving the Sky computing vision is to provide a cloud compatibility layer; i.e., a layer that abstracts away the services provided by a cloud and allows an application developed on top of this layer to run on different clouds without change. In short, the compatibility layer is a set of interfaces or APIs that applications can be built on; this compatibility layer can then be ported to each cloud using the cloud’s set of (perhaps proprietary) interfaces.

实现天空计算愿景的第一步是提供一个云兼容层，即一个抽象化云服务的层，使在该层之上开发的应用程序可以在不同的云上运行而无需修改。简而言之，兼容层是一组接口或API，应用程序可以基于这些接口构建；然后，该兼容层可以使用各云平台的一套（可能是专有的）接口移植到每个云上。

In our Internet analogy, this is similar to the IP layer that enables routers using different underlying (L2) communication technologies to handle IP packets. However, unlike IP which is a narrow waist, the cloud compatibility layer is significantly broader and less well defined since clouds expose a rich (and growing) set of services to the applications, including computation, storage, data transfers, and much more. Thus, the cloud compatibility layer is more similar in spirit to an operating system (e.g., Linux) that manages the computer’s resources and exposes an API to applications.

在我们的互联网类比中，这类似于IP层，它使得使用不同底层（L2）通信技术的路由器能够处理IP数据包。然而，与IP这种“窄腰层”不同，云兼容层要宽得多，定义也更为模糊，因为云向应用程序公开了一套丰富（且不断增长）的服务，包括计算、存储、数据传输等。因此，云兼容层在精神上更类似于操作系统（如Linux），它管理计算机资源并向应用程序暴露API。

How do we build such a cloud compatibility layer? While every cloud offers a set of proprietary low-level interfaces, most users today interact mostly with higher level management and service interfaces. Some of these are proprietary, but a growing number of them are supported by open source software (OSS).

如何构建这样的云兼容层？虽然每个云平台都提供了一套专有的低级接口，但如今大多数用户主要与更高级的管理和服务接口进行交互。这些接口中有一些是专有的，但越来越多的接口由开源软件（OSS）支持。

These OSS projects exist all levels of the software stack, including operating systems (Linux), cluster resource managers (Kubernetes [12], Apache Mesos [31]), application packaging (Docker [10]), databases (MySQL [15], Postgres [17]), big data execution engines (Apache Spark [42], Apache Hadoop [41]), streaming engines (Apache Flink [26], Apache Spark [42], Apache Kafka [5]), distributed query engines and databases (Cassandra [4], MongoDB [14], Presto [18], SparkSQL [22], Redis [19]), machine learning libraries (PyTorch [37], Tensorflow [24], MXNet [27], MLFlow [13], Horovod [40], Ray RLlib [33]), and general distributed frameworks (Ray [36], Erlang [25], Akka [1]).

这些开源项目存在于软件栈的各个层次，包括：

操作系统（Linux）
集群资源管理器（Kubernetes [12]、Apache Mesos [31]）
应用程序打包（Docker [10]）
数据库（MySQL [15]、Postgres [17]）
大数据执行引擎（Apache Spark [42]、Apache Hadoop [41]）
流处理引擎（Apache Flink [26]、Apache Spark [42]、Apache Kafka [5]）
分布式查询引擎和数据库（Cassandra [4]、MongoDB [14]、Presto [18]、SparkSQL [22]、Redis [19]）
机器学习库（PyTorch [37]、Tensorflow [24]、MXNet [27]、MLFlow [13]、Horovod [40]、Ray RLlib [33]）
通用分布式框架（Ray [36]、Erlang [25]、Akka [1]）

Furthermore, a plethora of companies founded by OSS creators have emerged to provide hosted services on multiple clouds. Example are Cloudera (Apache Hadoop), Confluent (Apache Kafka), MongoDB, Redis Labs, HashiCorp (Terraform, Consul), Datastax (Cassandra), and Databricks (Apache Spark, MLFlow, and Delta). These developments make it relatively easy for enterprises to switch from one cloud to another if their applications are using one of these multi-cloud OSS-based offerings.

此外，由开源软件（OSS）创建者创立的大量公司已经涌现出来，提供基于多个云平台的托管服务。例子包括Cloudera（Apache Hadoop）、Confluent（Apache Kafka）、MongoDB、Redis Labs、HashiCorp（Terraform、Consul）、Datastax（Cassandra）和Databricks（Apache Spark、MLFlow和Delta）。这些发展使得企业相对容易地从一个云平台切换到另一个，前提是它们的应用程序使用的是这些基于多云的OSS解决方案之一。

The compatibility layer could be constructed out of some set of these OSS solutions. Indeed, there are already efforts underway to consolidate different OSS components in a single coherent platform. One example is Cloud Foundry [9], an open source multi-cloud application platform that supports all major cloud providers, as well as on-premise clusters. Another example is RedHat’s OpenShift [16], a kubernetesbased platform for multi-cloud and on-premise deployments.

兼容层可以由这些开源软件（OSS）解决方案中的一部分构建而成。实际上，已经有一些工作在进行中，旨在将不同的OSS组件整合到一个统一的平台中。一个例子是Cloud Foundry [9]，这是一个开源的多云应用平台，支持所有主要的云提供商以及本地集群。另一个例子是RedHat的OpenShift [16]，这是一个基于Kubernetes的平台，支持多云和本地部署。

While OSS provides solutions at most layers in the software stack, the one glaring gap is the storage layer. Every cloud provider has its own version of proprietary highlyscalable storage. Examples are AWS’ S3 [2], Microsoft’s Azure Blob Storage [7] and Google’s Cloud Storage [11]. This being said, there are already several solutions providing S3 compatibility APIs for Azure’s Blob Storage and Google’s Cloud Storage, such as S3Proxy [20] and Scality [21]. Some cloud providers offer their own S3 compatibility APIs to help customers transition from AWS to their own cloud. In what follows, we assume that the storage API provided by the compatibility layer allows reading data across clouds.

虽然开源软件（OSS）在软件栈的大多数层面提供了解决方案，但存储层却存在明显的空白。每个云提供商都有自己版本的专有高可扩展存储。例如，AWS的S3 [2]、微软的Azure Blob Storage [7]和谷歌的Cloud Storage [11]。尽管如此，目前已经有几个解决方案提供了Azure Blob Storage和Google Cloud Storage的S3兼容API，如S3Proxy [20]和Scality [21]。一些云提供商还提供自己的S3兼容API，以帮助客户从AWS过渡到他们自己的云。

Thus, we think achieving a widely usable compatibility layer is, on purely technical grounds, easily achievable. The problem is whether the market will support such an effort because, while the compatibility layer has clear benefits for users, it naturally leads to the commoditization of the cloud providers, which may not be in their interests. We will discuss the incentives issues in Section 7.

因此，我们认为，在纯技术层面上，实现一个广泛可用的兼容层是相对容易的。问题在于市场是否会支持这样的努力，因为尽管兼容层对用户有明显的好处，但它自然会导致云提供商的商品化，这可能并不符合他们的利益。我们将在第7节中讨论这些激励问题。