Skadi: Building a Distributed Runtime for Data Systems in Disaggregated Data Centers
Data-intensive systems are the backbone of today's computing and are responsible for shaping data centers. Over the years, cloud providers have relied on three principles to maintain cost-effective data systems: use disaggregation to decouple scaling, use domain-specific computing to battle waning laws, and use serverless to lower costs. Although they work well individually, they fail to work in harmony: an issue amplified by emerging data system workloads. In this paper, we envision a distributed runtime to mitigate current shortcomings. The distributed runtime has a tiered access layer exposing declarative APIs, underpinned by a stateful serverless runtime with a distributed task execution model. It will be the narrow waist between data systems and hardware. Users are oblivious to data location, concurrency, disaggregation style, or even the hardware to do the computing. The underlying stateful serverless runtime transparently evolves with novel data-center architectures, such as disaggregation and tightly-coupled clusters. We prototype Skadi to showcase that the distributed runtime is practical.
WOS:001119203300012
2023-01-01
979-8-4007-0195-5
New York
94
102
REVIEWED
Event name | Event place | Event date |
Providence, RI | JUN 22-24, 2023 | |
Funder | Grant Number |
National Key Research and Development Plan of China | 2022YFB4500400 |
Strategic Priority Research Program of Chinese Academy of Sciences | XDA0320000 |
National Natural Science Foundation of China | 62090022 |
Youth Innovation Promotion Association of Chinese Academy of Sciences | 2020105 |