Abstract
Given the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness. In this paper, we study an online sketched Newton method that leverages a randomized sketching technique to perform an approximate Newton step in each iteration, thereby eliminating the computational bottleneck of second-order methods. While existing studies have established the asymptotic normality of sketched Newton methods, a consistent estimator of the limiting covariance matrix remains an open problem. We propose a fully online covariance matrix estimator that is constructed entirely from the Newton iterates and requires no matrix factorization. Compared to covariance estimators for first-order online methods, our estimator for second-order methods is batch-free. We establish the consistency and convergence rate of our estimator, and coupled with asymptotic normality results, we can then perform online statistical inference for the model parameters based on sketched Newton methods. We also discuss the extension of our estimator to constrained problems, and demonstrate its superior performance on regression problems as well as benchmark problems in the CUTEst set.
Wei Kuang
PhD in Statistics (2019-2025)
Wei Kuang was a PhD student in the Statistics department at UChicago, working with Mihai Anitescu (supervisor) and Sen Na on randomized second-order methods, with an emphasis on the uncertainty quantification and statistical inference aspects.
Mihai Anitescu
Professor in Statistics and CAM
Mihai Anitescu is a Professor in the Statistics and CAM departments at the University of Chicago, and is also a senior computational mathematician in the Mathematics and Computer Science Division at Argonne. He works on a variety of topics on control, optimization, and computational statistics.
Sen Na
Assistant Professor in ISyE
Sen Na is an Assistant Professor in the School of Industrial and Systems Engineering at Georgia Tech. Prior to joining ISyE, he was a postdoctoral researcher in the statistics department and ICSI at UC Berkeley. His research interests broadly lie in the mathematical foundations of data science, with topics including high-dimensional statistics, graphical models, semiparametric models, optimal control, and large-scale and stochastic nonlinear optimization. He is also interested in applying machine learning methods to problems in biology, neuroscience, and engineering.