SmartJoin: A network-aware multiway join for MapReduce

Kenn Slagter, Ching Hsien Hsu, Yeh Ching Chung, Gangman Yi

Research output: Contribution to journalArticlepeer-review

11 Scopus citations

Abstract

MapReduce is an effective tool for processing large amounts of data in parallel using a cluster of processors or computers. One common data processing task is the join operation, which combines two or more datasets based on values common to each. In this paper, we present a network aware multi-way join for MapReduce (SmartJoin) that improves performance and considers network traffic when redistributing workload amongst reducers. SmartJoin achieves this by dynamically redistributing tuples directly between reducers with an intelligent network aware algorithm. We show that our presented technique has significant potential to minimize the time required to join multiple datasets. In our evaluation, we show that SmartJoin has up to 39 % improvement compared to the non-redistribution method, a 26.8 % improvement over random redistribution and 27.6 % improvement over worst join redistribution.

Original languageEnglish
Pages (from-to)629-641
Number of pages13
JournalCluster Computing
Volume17
Issue number3
DOIs
StatePublished - Sep 2014

Keywords

  • Hadoop
  • MapReduce
  • Multiway join
  • Workload redistribution

Fingerprint

Dive into the research topics of 'SmartJoin: A network-aware multiway join for MapReduce'. Together they form a unique fingerprint.

Cite this