Stochastic approximation with adaptive step sizes for optimization in noisy environment and its application in regression models

Abstract

We propose a generalization of recently proposed stochastic approximation method with adaptive step sizes for optimization problems in noisy environment. The adaptive step size scheme uses only a predefined number of last noisy functional values to select a step size for the next iterate and allows different intensities of influence of the past functional values. The almost sure convergence is established under suitable assumptions. Numerical results indicate a good performance of the method. Application of the method in regression models is presented.