Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. Rock You like a Hurricane: Taming Skew in Large Scale Analytics
 
conference paper

Rock You like a Hurricane: Taming Skew in Large Scale Analytics

Bindschaedler, Laurent  
•
Malicevic, Jasmina  
•
Schiper, Nicolas
Show more
April 23, 2018
EuroSys '18: Proceedings of the Thirteenth EuroSys Conference
Eurosys '18

Current cluster computing frameworks suffer from load imbalance and limited parallelism due to skewed data distributions, processing times, and machine speeds. We observe that the underlying cause for these issues in current systems is that they partition work statically. Hurricane is a high-performance large-scale data analytics system that successfully tames skew in novel ways. Hurricane performs adaptive work partitioning based on load observed by nodes at runtime. Overloaded nodes can spawn clones of their tasks at any point during their execution, with each clone processing a subset of the original data. This allows the system to adapt to load imbalance and dynamically adjust task parallelism to gracefully handle skew. We support this design by spreading data across all nodes and allowing nodes to retrieve data in a decentralized way. The result is that Hurricane automatically balances load across tasks, ensuring fast completion times. We evaluate Hurricane’s performance on typical analytics workloads and show that it significantly outperforms state- of-the-art systems for both uniform and skewed datasets, because it ensures good CPU and storage utilization in all cases.

  • Files
  • Details
  • Metrics
Loading...
Thumbnail Image
Name

hurricane.pdf

Type

Publisher's Version

Version

http://purl.org/coar/version/c_970fb48d4fbd8a85

Access type

openaccess

Size

2.52 MB

Format

Adobe PDF

Checksum (MD5)

aa82439aa1ffa14a08978bf2a548c186

Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés