Handling Data Skew in MapReduce Cluster by Using Partition Tuning

Abstract EN

The healthcare industry has generated large amounts of data, and analyzing these has emerged as an important problem in recent years.

The MapReduce programming model has been successfully used for big data analytics.

However, data skew invariably occurs in big data analytics and seriously affects efficiency.

To overcome the data skew problem in MapReduce, we have in the past proposed a data processing algorithm called Partition Tuning-based Skew Handling (PTSH).

In comparison with the one-stage partitioning strategy used in the traditional MapReduce model, PTSH uses a two-stage strategy and the partition tuning method to disperse key-value pairs in virtual partitions and recombines each partition in case of data skew.

The robustness and efficiency of the proposed algorithm were tested on a wide variety of simulated datasets and real healthcare datasets.

The results showed that PTSH algorithm can handle data skew in MapReduce efficiently and improve the performance of MapReduce jobs in comparison with the native Hadoop, Closer, and locality-aware and fairness-aware key partitioning (LEEN).

We also found that the time needed for rule extraction can be reduced significantly by adopting the PTSH algorithm, since it is more suitable for association rule mining (ARM) on healthcare data.