Faculty of Natural Sciences and Mathematics

Permanent URI for this communityhttps://repository.ukim.mk/handle/20.500.12188/16

Browse

Settings

Start date: 2010Subject: search.filters.subject.Adaptive step size

Search Results

Now showing 1 - 2 of 2
  • Loading...
    Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Adaptive stochastic approximation algorithm
    (Springer Science and Business Media LLC, 2017-02-27)
    Kresoja, Milena
    ;
    Lužanin, Zorana
    ;
    In this paper, stochastic approximation (SA) algorithm with a new adaptive step size scheme is proposed. New adaptive step size scheme uses a fixed number of previous noisy function values to adjust steps at every iteration. The algorithm is formulated for a general descent direction and almost sure convergence is established. The case when negative gradient is chosen as a search direction is also considered. The algorithm is tested on a set of standard test problems. Numerical results show good performance and verify efficiency of the algorithm compared to some of existing algorithms with adaptive step sizes.
  • Loading...
    Thumbnail Image
    Some of the metrics are blocked by your 
    consent settings
    Item type:Publication,
    Descent Direction Stochastic Approximation Algorithm with Adaptive Step Sizes
    (Global Science Press, 2019-06-01)
    Lužanin, Zorana
    ;
    ;
    Kresoja, Milena
    A stochastic approximation (SA) algorithm with new adaptive step sizes for solving unconstrained minimization problems in noisy environment is proposed. New adaptive step size scheme uses ordered statistics of fixed number of previous noisy function values as a criterion for accepting good and rejecting bad steps. The scheme allows the algorithm to move in bigger steps and avoid steps proportional to 1/k when it is expected that larger steps will improve the performance. An algorithm with the new adaptive scheme is defined for a general descent direction. The almost sure convergence is established. The performance of new algorithm is tested on a set of standard test problems and compared with relevant algorithms. Numerical results support theoretical expectations and verify efficiency of the algorithm regardless of chosen search direction and noise level. Numerical results on problems arising in machine learning are also presented. Linear regression problem is considered using real data set. The results suggest that the proposed algorithm shows promise.