Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. Storage Management in Smart Data Lake
 
conference paper not in proceedings

Storage Management in Smart Data Lake

Haoqiong Bian  
•
Bikash Chandra  
•
Ioannis Mytilinis  
Show more
March 23, 2021

Data lakes are complex ecosystems where heterogeneity prevails. Raw data of diverse formats are stored and processed, while long and expensive ETL processes are avoided. Apart from data heterogeneity, data lakes also entail hardware heterogeneity. Typical installations involve distributed infrastructures, where each node is possibly equipped with hardware of different characteristics. Especially for the case of storage, the various devices a node possesses can be organized in a hierarchy that defines a spectrum of performance-capacity-cost configurations. Given the various configurations and the volatile workload landscape, taking optimal placement decisions is a cumbersome task. In this work, we propose a storage management solution for the Smart Data Lake [12] platform. The proposed system takes advantage of the available storage devices, while it abstracts away data/hardware characteristics and provides a unified interface for data accesses. This way performance is improved while tiering complexity is hidden from the application layer.

  • Details
  • Metrics
Type
conference paper not in proceedings
Author(s)
Haoqiong Bian  
Bikash Chandra  
Ioannis Mytilinis  
Anastasia Ailamaki  
Date Issued

2021-03-23

Editorial or Peer reviewed

REVIEWED

Written at

EPFL

EPFL units
DIAS  
Available on Infoscience
December 9, 2022
Use this identifier to reference this record
https://infoscience.epfl.ch/handle/20.500.14299/193118
Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés