Model Hyper Parameter Tuning using Ant Colony Optimization

Abstract

The process of adjusting the best hyper-parameters for machine learning models is a complex optimization problem over the models that in reality are ”black-boxes”. This process entails several challenges such as the time complexity of finding optimal results, different types of hyper-parameters (continuous and categorical hyper-parameters) and model over-fitting. The main benefits of the hyper-parameter adjustment process are the improvement of the efficiency and quality of the models. This paper presents an implementation for adjusting hyperparameters of time series prediction regression models, which is based on pheromone paths of ant colonies used as a way of finding food in nature. This approach of leaving pheromone pathways has been used to quickly find values for the hyperparameters of the regression models for which they give the most optimal results. The proposed implementation for Ant Colony optimization is compared with three other parameter optimization approaches (Grid search, Bayesian optimization and Tree Parzen Estimators) on ten different regression datasets according to the time required to perform the optimization and the quality of the model prediction. The experimental evaluation shows that the proposed method needs minimal time to optimize the hyperparameters, while preserving very good predictive performance in comparison to the competing approaches.