SQPR: Stream query planning with reuse

When users submit new queries to a distributed stream processing system (DSPS), a query planner must allocate physical resources, such as CPU cores, memory and network bandwidth, from a set of hosts to queries. Allocation decisions must provide the correct mix of resources required by queries, while achieving an efficient overall allocation to scale in the number of admitted queries. By exploiting overlap between queries and reusing partial results, a query planner can conserve resources but has to carry out more complex planning decisions. In this paper, we describe SQPR, a query planner that targets DSPSs in data centre environments with heterogeneous resources. SQPR models query admission, allocation and reuse as a single constrained optimisation problem and solves an approximate version to achieve scalability. It prevents individual resources from becoming bottlenecks by re-planning past allocation decisions and supports different allocation objectives. As our experimental evaluation in comparison with a state-of-the-art planner shows SQPR makes efficient resource allocation decisions, even with a high utilisation of resources, with acceptable overheads.


Published in:
2011 IEEE 27th International Conference on Data Engineering, 840-851
Presented at:
2011 IEEE International Conference on Data Engineering (ICDE 2011), Hannover, Germany, 11-16 04 2011
Year:
2011
Publisher:
IEEE
ISBN:
978-1-4244-8959-6
Keywords:
Laboratories:




 Record created 2014-01-29, last modified 2018-01-28

External link:
Download fulltext
URL
Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)