Kybernetika 54 no. 5, 958-977, 2018

Rank theory approach to ridge, LASSO, preliminary test and Stein-type estimators: Comparative study

A. K. Md. Ehsanes Saleh and Radim NavrátilDOI: 10.14736/kyb-2018-5-0958


In the development of efficient predictive models, the key is to identify suitable predictors for a given linear model. For the first time, this paper provides a comparative study of ridge regression, LASSO, preliminary test and Stein-type estimators based on the theory of rank statistics. Under the orthonormal design matrix of a given linear model, we find that the rank based ridge estimator outperforms the usual rank estimator, restricted R-estimator, rank-based LASSO, preliminary test and Stein-type R-estimators uniformly. On the other hand, neither LASSO nor the usual R-estimator, preliminary test and Stein-type R-estimators outperform the other. The region of domination of LASSO over all the R-estimators (except the ridge R-estimator) is the interval around the origin of the parameter space. Finally, we observe that the L$_2$-risk of the restricted R-estimator equals the lower bound on the L$_2$-risk of LASSO. Our conclusions are based on L$_2$-risk analysis and relative L$_2$-risk efficiencies with related tables and graphs.


efficiency of LASSO, penalty estimators, preliminary test, Stein-type estimator, ridge estimator, L$_2$-risk function


62G05, 62J05, 62J07


  1. A. Belloni A and V. Chernozhukov: Least squares after model selection in high-dimensional sparse models. Bernoulli 19 (2013), 521-547.   DOI:10.3150/11-bej410
  2. L. Breiman: Heuristics of instability and stabilization in model selection. Ann. Statist. 24 (1996), 2350-2383.   DOI:10.1214/aos/1032181158
  3. D. L. Donoho and I. M. Johnstone: Minimax estimation via wavelet shrinkage. Ann. Statist. 26 (1994), 879-921.   DOI:10.1214/aos/1024691081
  4. N. R. Draper and R. C. Van Nostrand: Ridge regression and James-Stein estimation: review and comments. Technometrics 21 (1979), 451-466.   DOI:10.2307/1268284
  5. J. Fan and R. Li: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 (2001), 1348-1360.   DOI:10.1198/016214501753382273
  6. L. E. Frank and J. H. Friedman: A statistical view of some chemometrics regression tools. Technometrics 35 (1993), 109-135.   DOI:10.1080/00401706.1993.10485033
  7. E. Hoerl and R. W. Kennard: Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12 (1970), 55-67.   DOI:10.1080/00401706.1970.10488634
  8. W. James and C. Stein: Estimation with quadratic loss. In: Proc. Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, University of California Press 1961, pp. 361-379.   CrossRef
  9. J. Jurečková: Nonparametric estimate of regression coefficients. Ann. Math. Statist. 42 (1971), 1328-1338.   DOI:10.1214/aoms/1177693245
  10. B. E. Hansen: The risk of James-Stein and Lasso shrinkage. Econometric Rev. 35 (2015), 456-470.   CrossRef
  11. A. K. Md. E. Saleh: Theory of Preliminary test and Stein-Type Estimators with Applications. John Wiley and Sons, New York 2006.   DOI:10.1002/0471773751
  12. A. K. Md. E. Saleh, M. Arashi, M. Norouzirad and B. M. G. Kibria: On shrinkage and selection: ANOVA MODEL. J. Statist. Res. 51 (2017), 165-191.   CrossRef
  13. C. Stein: Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In: Proc. Third Berkeley Symposium on Mathematical Statistics and Probability, University of California Press 1956, pp. 197-206.   CrossRef
  14. R. Tibshirani: Regression shrinkage and selection via the lasso. J. Royal Statist. Soc., Series B (Methodological) 58 (1996), 267-288.   CrossRef
  15. A. N. Tikhonov: Solution of incorrectly formulated problems and the regularization method. Doklady Akademii Nauk SSSR 151 (1963), 501-504.   CrossRef
  16. H. Zou: The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 (2006), 1418-1429.   DOI:10.1198/016214506000000735
  17. H. Zou and T. Hastie: Regularization and variable selection via the elastic net. J. Royal Stat. Soc. Ser. B Stat. Methodol. 67 (2005), 301-320.   DOI:10.1111/j.1467-9868.2005.00503.x