How Accurate Can Any Regression Procedure Be?
Iowa State University
Various parametric and nonparametric regression procedures have been constructed according to different possible characteristics of the underlying regression function. To reduce the dependence on subjective assumptions, the theme of adaptive estimation is to construct a procedure that provides an accurate estimate of the regression function for various scenarios without knowing which one describes the data well. A closely related question is: Given a regression procedure, how many regression functions are estimated accurately? In this work, for a given sequence of prescribed estimation accuracy (in sample size), we given an upper bound (in terms of metric entropy) on the number of regression functions for which the accuracy is achieved. A consequence is that if one demands near optimal performance for a target class of regression functions, then the same accuracy can not be achieved for many additional regression functions. This has a negative implication on adaptive estimation. The main result is also applied to show that as far as polynomial rates of convergence are concerned, any regression procedure is essentially no better than a method based on sparse approximation.