Agile Views in a Dynamic Data Management System
This paper calls for a new breed of lightweight systems - dynamic data management systems (DDMS). In a nutshell, a DDMS manages large dynamic data structures with agile, frequently fresh views, and provides a facility for monitoring these views and triggering application-level events. We motivate DDMS with applications in large-scale data analytics, database monitoring, and high-frequency algorithmic trading. We compare DDMS to more traditional data management systems architectures. We present the DBToaster project, which is an ongoing effort to develop a prototype DDMS system. We describe its architecture design, techniques for high-frequency incremental view maintenance, storage, scaling up by parallelization, and various key challenges to overcome to make DDMS a reality.
CIDR11_Paper38.pdf
openaccess
815.51 KB
Adobe PDF
f068486b0ff00e3c5a52fe6fb67d6eb6