Efficient Performance Optimization on Yarn-Based MapReduce Hadoop Framework

Abstract

Apache Hadoop exposes 180+ configuration parameters for all types of applications and clusters, 10-20% of which has a great impact on performance and efficiency of the execution. The optimal configuration settings for one application may not be suitable for another one leading to poor system resources utilization and long application completion time. Further, optimizing many parameters is a time consuming and a challenging job because configuration parameters and search space are huge, and users require good knowledge of Hadoop framework. The issue is that the user should adjust at least the important parameters, e.g. the number of map tasks that can run in parallel for a given application. This paper introduces the parameter optimization algorithm to the key application level parameter based on input data size and dynamic resource capabilities at any given time for a given application to improve execution time and resource utilization with nearly zero optimization overhead.