Reinforcement Learning initialization by evolutionary formulation: Application for workflow autoscaling in the Cloud

Abstract

Scientific workflow execution is usually fulfilled through Cloud Computing, but correct autoscaling techniques are needed for proper performance. Reinforcement Learning (RL) has been used for autoscaling, but presents low performance in early stages. Poor initial performance accumulates over episodes, making the learning process more expensive, which is critical in the context of Cloud autoscaling. Solutions to this problem are sparse and difficult to generalize. Here, we present Reinforcement Learning Initialization by Evolutionary Formulation (ReLIEF), which uses evolutionary algorithm to generate an initial pre-optimized RL policy, that is later refined via RL. Proposed initilization aims to reduce the accumulated losses in monetary cost and execution time (i.e. makespan) during learning. In this article two prominent evolutionary algorithm are used: Non-dominated Sorting Genetic Algorithm III (NSGA-III) and Improved Decomposition-Based Evolutionary Algorithm (I-DBEA). On the other hand, for Reinforcement Learning only Q-Learning in tabular form is used. Four benchmark workflows were used to validate savings produced by the proposal. In 3 out of 4 workflows analyzed, ReLIEF outperformed baseline RL agents. In the remaining workflow, competitive performance was obtained.