Cloud Data Center Resource Demand Prediction Model Development on Apache Spark

Abstract

Dynamic resource allocation in cloud data centers is a challenging problem. Resource prediction is a key feature for on-demand resource planning and efficient resource management of dynamic workload. This requires a highly accurate demand prediction. Hyper-parameter optimization can largely affect the performance of the prediction model. The process of identifying the optimal parameters for a machine learning (ML) algorithm involves the search for a broad range of value combinations of parameter sets. This paper presents a resource demand prediction model with the cloud computational frameworks Apache™ Hadoop® and Apache Spark™. The model is developed on the powerful ML technique, Decision Tree (DT) algorithm, and hyper-parameter optimization for DT algorithm is performed to achieve the prediction model with high accuracy. The evaluation of prediction model is conducted on real data center workload traces and the evaluation results show that hyper-parameter optimization can save the prediction error significantly.