Kybernetika 60 no. 1, 38-59, 2024

Highly robust training of regularized radial basis function networks

Jan Kalina, Petra Vidnerová and Patrik JanáčekDOI: 10.14736/kyb-2024-1-0038

Abstract:

Radial basis function (RBF) networks represent established tools for nonlinear regression modeling with numerous applications in various fields. Because their standard training is vulnerable with respect to the presence of outliers in the data, several robust methods for RBF network training have been proposed recently. This paper is interested in robust regularized RBF networks. A robust inter-quantile version of RBF networks based on trimmed least squares is proposed here. Then, a systematic comparison of robust regularized RBF networks follows, which is evaluated over a set of 405 networks trained using various combinations of robustness and regularization types. The experiments proceed with a particular focus on the effect of variable selection, which is performed by means of a backward procedure, on the optimal number of RBF units. The regularized inter-quantile RBF networks based on trimmed least squares turn out to outperform the competing approaches in the experiments if a highly robust prediction error measure is considered.

Keywords:

robustness, quantile regression, regression neural networks, robust training, effective regularization

Classification:

68T37, 68W25, 62J02

paper.pdf

References:

  1. G. Blokdyk: Artificial Neural Network: A Complete Guide. Createspace Independent Publishing Platform, Scotts Valey 2017.   CrossRef
  2. A. G. Borş and I. Pitas: Robust RBF networks. In: Radial basis function networks 1. (R. J. Howlett, L. C. Jain, and J. Kacprzyk, eds.). Recent developments in theory and applications, Physica Verlag Rudolf Liebing KG, Vienna 2001, pp. 123-133.   CrossRef
  3. K. Boudt, P. J. Rousseeuw, S. Vanduffel and T. Verdonck: The minimum regularized covariance determinant estimator. Statist. Computing 30 (2020), 113-128.   DOI:10.1007/s11222-019-09869-x
  4. S. Chatterjee and A. S. Hadi: Regression Analysis by Example. (Fifth edition.) Wiley, Hoboken 2012.   CrossRef
  5. Y. B. Cheng, X. H. Chen, H. L. Li, Z. Y. Cheng and R. Jiang et al.: Analysis and comparison of Bayesian methods for measurement uncertainty evaluation. Math. Problems Engrg. (2018), 7509046.   DOI:10.1155/2018/7509046
  6. F. Chollet: Keras.    CrossRef
  7. Y. Dodge, J and Jurečková: Adaptive Regression. Springer, New York 2000.   CrossRef
  8. J. Dong, Y. Zhao, C. Liu, Z. F. Han and C. S. Leung: Orthogonal least squares based center selection for fault-tolerant RBF networks. Neurocomputing 339 (2019), 217-231.   DOI:10.1016/j.neucom.2019.02.039
  9. D. Dua and C. Graff: UCI Machine Learning Repository.    CrossRef
  10. E. Egrioglu, E. Bas and O. Karahasan: Winsorized dendritic neuron model artificial neural network and a robust training algorithm with Tukey's biweight loss function based on particle swarm optimization. Granular Comput. 8 (2023), 491-501.   DOI:10.1007/s41066-022-00345-y
  11. A. H. Fath, F. Madanifar and M. Abbasi: Implementation of multilayer perceptron (MLP) and radial basis function (RBF) neural networks to predict solution gas-oil ratio of crude oil systems. Petroleum 6 (2020), 80-91.   DOI:10.1016/j.petlm.2018.12.002
  12. M. Hallin, D. Paindaveine and M. Šiman: Multivariate quantiles and multiple-output regression quantiles: From $L_1$ optimization to halfspace depth. Ann. Statist. 38 (2010), 635-669.   DOI:10.1214/09-aos723
  13. I. Han, X. Qian, H. Huang and T. Huang: Efficient design of multicolumn RBF networks. Neurocomputing 450 (2021), 253-263.   DOI:10.1016/j.neucom.2021.04.040
  14. T. Hastie, R. Tibshirani and J. Friedman: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. (Second edition.). Springer, New York 2009.   CrossRef
  15. J. Jurečková, J. Picek and M. Schindler: Robust Statistical Methods with R. (Second edition.) Chapman and Hall/CRC, Boca Raton 2019.   CrossRef
  16. J. Kalina and J. Tichavský: On robust estimation of error variance in (highly) robust regression. Measurement Sci. Rev. 20 (2020), 6-14.   DOI:10.2478/msr-2020-0002
  17. J. Kalina, A. Neoral and P. Vidnerová: Effective automatic method selection for nonlinear regression modeling. Int. J. Neural Syst. 31 (2021), 2150020.   DOI:10.1142/S0129065721500209
  18. J. Kalina, J. Tumpach and M. Holeňa: On combining robustness and regularization in training multilayer perceptrons over small data. In: 2000 International Joint Conference on Neural Networks (IJCNN), IEEE 2022.   DOI:10.1109/ijcnn55064.2022.9892510
  19. M. E. Karar: Robust RBF neural network-based backstepping controller for implantable cardiac pacemakers. Int. J. Adaptive Control Signal Process. 32 (2018), 1040-1051.   DOI:10.1002/acs.2884
  20. I. A. Khan, T. Hussain, A. Ullah, S. Rho, M. Lee, M. and S. W. Baik: Towards efficient electricity forecasting in residential and commercial buildings: A novel hybrid CNN with a LSTM-AE based framework. Sensors 20 (2020), 1399.   DOI:10.3390/s20051399
  21. G. Klambauer, T. Unterthiner, A. Mayr and S. Hochreiter: Self-normalizing neural networks. In: NIPS'17: Proc. 31st International Conference on Neural Information Processing Systems, Curran Associates, New York 2017, pp. 972-981.   CrossRef
  22. M. A. Knefati, P. E. Chauvet, S. N'Guyen and B. Daya: Reference curves estimation using conditional quantile and radial basis function network with mass constraint. Neural Process. Lett. 43 (2016), 17-30.   DOI:10.1007/s11063-014-9399-9
  23. R. Koenker: Quantile regression: 40 years on. Annual Rev. Econom. 9 (2917), 155-176.   DOI:10.1146/annurev-economics-063016-103651
  24. M. Kordos and A. Rusiecki: Reducing noise impact on MLP training -- Techniques and algorithms to provide noise-robustness in MLP network training. Soft Comput. 20 (2016), 46-65.   DOI:10.1007/s00500-015-1690-9
  25. C. C. Lee, P. C. Chung, J. R. Tsai and C. I. Chang: Robust radial basis function neural networks. IEEE Trans. Systems Man Cybernet. B 29 (1999), 674-685.   DOI:10.1109/3477.809023
  26. X. Li and Y. Sun: Application of RBF neural network optimal segmentation algorithm in credit rating. Neural Computing Appl. 33 (2021), 8227-8235.   DOI:10.1007/s00521-020-04958-9
  27. Z. Liu, C. S. Leung and H. C. So: Formal convergence analysis on deterministic $\ell_1$-regularization based mini-batch learning for RBF networks. Neurocomputing 532 (2023), 77-93.   DOI:10.1016/j.neucom.2023.02.012
  28. National Institute of Standards and Technology (NIST): Nonlinear Regression Datasets.    CrossRef
  29. C. Paul and G. K. Vishwakarma: Back propagation neural networks and multiple regressions in the case of heteroscedasticity. Commun. Statist. Simul. Comput. 46 (2017), 6772-6789.   DOI:10.1080/03610918.2016.1212066
  30. G. Petneházi: Quantile convolutional neural networks for value at risk forecasting. Machine Learning Appl. 6 (2021), 100096.   DOI:10.1016/j.mlwa.2021.100096
  31. T. Poggio and S. Smale: The mathematics of learning: Dealing with data. Notices Amer. Math. Soc. 50 (2003), 537-544.   DOI:10.1109/icnnb.2005.1614546
  32. B. Procházka: Regression quantiles and trimmed least squares estimator in the nonlinear regression model. Comput. Statist. Data Anal. 6 (1988), 385-391.   DOI:10.1016/0167-9473(88)90078-3
  33. Q. Que and M. Belkin: Back to the future: Radial basis function network revisited. IEEE Trans. Pattern Anal. Machine Intell. 42 (2020), 1856-1867.   DOI:10.1109/TPAMI.2019.2906594
  34. Y. Romano, E. Patterson and E. J. Candès: Conformalized quantile regression. ArXiv 2019.   https://arxiv.org/abs/1905.03222
  35. A. Rusiecki: Trimmed categorical cross-entropy for deep learning with label noise. Electron. Lett. 55 (2019), 319-320.   DOI:10.1049/el.2018.7980
  36. A. Rusiecki: Robust learning algorithm based on LTA estimator. Neurocomputing 120 (2013), 624-632.   DOI:10.1016/j.neucom.2013.04.008
  37. A. K. M. E. Saleh, J. Picek and J. Kalina: R-estimation of the parameters of a multiple regression model with measurement errors. Metrika 75 (2012), 311-328.   DOI:10.1007/s00184-010-0328-2
  38. A. K. Seghouane and N. Shokouhi: Adaptive learning for robust radial basis function networks. IEEE Trans. Cybernet. 51 (2021), 2847-2856.   DOI:10.1109/TCYB.2019.2951811
  39. J. Šíma, P. Vidnerová and V. Mrázek: Energy complexity model for convolutional neural networks. Lecture Notes Computer Sci. 14263 (2023), 186-198.   DOI:10.1016/j.comcom.2022.10.029
  40. M. J. Su and W. Deng: A fast robust learning algorithm for RBF network against outliers. Lecture Notes Computer Sci. 4113 (2006), 280-285.   DOI:10.1007/11816157\_28
  41. V. Sze, Y. B. Chen, T. J. Yang and J. S. Emer: Efficient Processing of Deep Neural Networks. Morgan and Claypool, San Rafael 2020.   CrossRef
  42. J. Tukey: Comparing individual means in the analysis of variance. Biometrics 5 (1949), 99-114.   DOI:10.2307/3001913
  43. I. Ullah, H. Y. Youn and Y. H. Han: An efficient data aggregation and outlier detection scheme based on radial basis function neural network for WSN. J. Ambient Intell. Humanized Comput., in press (2022).   CrossRef
  44. J.Á. Víšek: Consistency of the least weighted squares under heteroscedasticity. Kybernetika 47 (2011), 179-206.   CrossRef
  45. T. Werner: Quantitative robustness of instance ranking problems. ArXiv 2021.   https://arxiv.org/abs/2103.07198
  46. R. R. Wilcox: Introduction to Robust Estimation and Hypothesis Testing. Fourth edition. Academic Pres, London 2017.   CrossRef
  47. C. Yang, S. K. Oh, W. Pedrycz, Z. Fu and B. Yang: Design of reinforced fuzzy radial basis function neural network classifier driven with the aid of iterative learning techniques and support vector-based clustering. IEEE Trans. Fuzzy Systems 29 (2021), 2506-2520.   DOI:10.1109/TFUZZ.2020.3001740
  48. P. Yerpude and V. Gudur: Predictive modelling of crime dataset using data mining. Int. J. Data Mining Knowledge Management Process 7 (2017), 43-58.   DOI:10.5121/ijdkp.2017.7404
  49. D. Zhang, G. Zang, J. Li, K. Ma and H. Liu: Prediction of soybean price in China using QR-RBF neural network model. Computers Electron. Agriculture 154 (2018), 10-17.   DOI:10.1016/j.compag.2018.08.016
  50. Y. Zuo: New algorithms for computing the least trimmed squares estimator. ArXiv 2022.   https://arxiv.org/abs/2203.10387