NEO

NEO: NEuro-Inspired Optimization—A Fractional Time Series Approach

Chatterjee, Sarthak (Rensselaer Polytechnic Institute)
Das, Subhro (IBM Research)
Gonçalves Melo Pequito, S.D. (TU Delft Team Sergio Pequito)

2021

Solving optimization problems is a recurrent theme across different fields, including large-scale machine learning systems and deep learning. Often in practical applications, we encounter objective functions where the Hessian is ill-conditioned, which precludes us from using optimization algorithms utilizing second-order information. In this paper, we propose to use fractional time series analysis methods that have successfully been used to model neurophysiological processes in order to circumvent this issue. In particular, the long memory property of fractional time series exhibiting non-exponential power-law decay of trajectories seems to model behavior associated with the local curvature of the objective function at a given point. Specifically, we propose a NEuro-inspired Optimization (NEO) method that leverages this behavior, which contrasts with the short memory characteristics of currently used methods (e.g., gradient descent and heavy-ball). We provide evidence of the efficacy of the proposed method on a wide variety of settings implicitly found in practice.

fractional calculus
iterative optimization algorithms
long memory time series
optimization
time series processes

http://resolver.tudelft.nl/uuid:943d3429-e674-458a-abe1-f77decb47571

https://doi.org/10.3389/fphys.2021.724044

Frontiers in Physiology, 12

Institutional Repository

journal article

© 2021 Sarthak Chatterjee, Subhro Das, S.D. Gonçalves Melo Pequito