Predictive Approach to Model Selection and Validation in Statistical Learning Networks
Abstract
The best model selection and the validation of the model are key issues in any model-building process. The present paper summarizes the results from international research done in Europe, Australia, and most recently in the United States. It discusses the model selection and validation in deep neural networks based on their prediction errors and provides some insights how to improve their accuracy in a very cost-effective way.
References
Beer, S. (1959) Cybernetics and Management. London: English University Press.
Brown, S. (2023) ‘Machine Learning, Explained’ Available at: https://mitsloan.mit.edu/ ideas-made-to-matter/machine-learning-explained (Accessed: 02 November 2023).
Burns, Ed. (2017) ‘Deep learning models hampered by black box functionality’. Available at: http://searchbusinessanalytics.techtarget. com/feature/Deep-learning-models-hampered-by-black-box-functionality (Accessed: 04 May 2017).
Cureton, E. (1950) ‘Validity, reliability and boloney’. Educ. & Psych. Meas., No. 10, pp. 94-96.
Cureton, E. (1951) ‘Symposium: The need and means of cross-validation. II. Approximate linear restraints and best predictor weights’. Educ. & Psychol. Measurement, No. 11, pp. 12-15.
Fayyad, U., Piatetsky-Shapiro, G., and Smyth P. (1996) ‘From Data Mining to Knowledge Discovery in Databases’. American Association for Artificial Intelligence Magazine, Fall, pp. 37-54. Available at: http://www.kdnuggets.com/gpspubs/aimag-kdd-overview-1996-Fayyad.pdf (Accessed: 02 November 2023).
Günnemann, St., Kremer, H., and Seidl, Th. (2011) ‘An extension of the PMML standard to subspace clustering models’. Proceedings of the 2011 workshop on Predictive markup language modeling, p. 48. doi:10.1145/2023598.2023605.
Hastie, T., Tibshirani, R., and Friedman J. (2017) The Elements of Statistical Learning: data mining, inference, and prediction. New York: Springer.
Herzberg, P. A. (1969) ‘The Parameters of Cross-Validation’ (Psychometric Monograph No. 16, Supplement to Psychometrika, 34). Richmond, VA: Psychometric Society. Available at: http://www.psychometrika.org/journal/online/MN16.pdf (Accessed: 02 November 2023).
Hornik, K., Stinchcombe, M. and White, H. (1989) ‘Multilayer feed-forward networks are universal approximators’. Neural Networks 2, pp. 359–366.
Horst, P. (1941) ‘Prediction of Personal Adjustment’. New York: Social Science Research Council (Bulletin No. 48)
IBM newsletter. (2023) ‘What is machine learning?’ Available at: https://www.ibm.com/topics/machine-learning (Accessed: 02 November 2023).
ISO 5725-1. (1994) ‘Accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions’, p. 1. Available at: https://www.iso.org/obp/ui/#iso: std:iso:5725:-1:ed-1:v1:en (Accessed: 02 November 2023).
Ivakhnenko, A., and Müller, J-A. (1996) ‘Recent Developments of Self-Organizing Modeling Prediction and Analysis of Stock Market’. http://www.gmdh.net/articles/index.html (Accessed: 02 November 2023).
Ivakhnenko, A. G. 1968. ‘Group Method of Data Handling - A Rival of the Method of Stochastic Approximation’. Soviet Automatic Control, Vol. 1, No. 3, pp. 43-55.
Ivakhnenko, A., and Lapa, V. (1967) Cybernetics and forecasting techniques. American Elsevier Pub. Co
Ivakhnenko, A.G. (1971) ‘Polynomial Theory of Complex Systems’, IEEE (Institute of Electrical and Electronics Engineers, Inc.) TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS Vol. SMC-1, No. 4, pp. 364-378.
Katzell, R. A. (1951) ‘Symposium: The need and means of cross-validation. III. Cross-validation of item analyses’. Educ. & Psychol. Measurement, 11, pp. 16-22.
Gödel, Kurt (1931) ‘Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I’. Monatshefte für Mathematik und Physik, Vol. 38, No. 1, pp. 173–198. doi:10.1007/BF01700692
Gödel, Kurt (2001) ‘Collected works’. Oxford University Press, Vol. I. Publications 1929-1936, pp. 144–195. ISBN 978-0195147209. The original German with a facing English translation, preceded by an introductory note by Stephen Cole Kleene. May 21, 2001, Oxford University Press, USA
Larson, S. (1931) ‘The shrinkage of the coefficient of multiple correlation’. Journal Educ. Psychol., 22, pp. 45-55.
Lenox, J. (2020) 2084 (Artificial Intelligence and The Future of Humanity). Zondervan Reflective, Grand Rapids, MI, p. 15.
Lovell, Michael C. (1983) ‘Data Mining’. The Review of Economics and Statistics, 65 (1): pp. 1–12. doi:10.2307/1924403. JSTOR 1924403
Lucey, T. (1991) Management Information Systems. DP Publications Lim.
Madala, H. and Ivakhnenko, A. G. (1994) Inductive Learning Algorithms for Complex Systems Modelling. CRC Press Inc., Boca Raton.
McCarthy, J. (1956) ‘What Is Artificial Intelligence’, p. 2. Available at: http://www-formal.stanford.edu/jmc/whatisai.pdf (Accessed: 02 November 2023).
MicroStrategy, (2005) ‘An Architecture for Enterprise Business Intelligence’. White Paper., pp. 162-173. Available at: http://www.microstrategy.com/Publications/Whitepapers (Accessed: 02 November 2023).
Mohri, M., Rostamizadeh, A. and Talwalkar, A. (2012) Foundations of Machine Learning. The MIT Press
Mosier, C. I. (1951) ‘Problem and designs of cross-validation‘. Symposium: The need and means of cross-validation. I. Educ. & Psychol. Measurement, 11, pp. 5-11
Motzev, M. and Lemke, Fr. (2015) ‘SELF-ORGANIZING DATA MINING TECHNIQUES IN MODEL BASED SIMULATION GAMES FOR BUSINESS TRAINING AND EDUCATION’. Vanguard Scientific Instruments in Management, Vol. 11.
Motzev, M. (2018a) ‘A Framework for Developing Multi-Layered Networks of Active Neurons for Simulation Experiments and Model-Based Business Games Using Self-Organizing Data Mining with the Group Method of Data Handling’. In: Lukosch H., Bekebrede G., Kortmann R. (eds) Simulation Gaming. Applications for Sustainable Cities and Smart Infrastructures. Lecture Notes in Computer Science, vol 10825. Springer, pp.191-199. DOI: 10.1007/978-3-319-91902-7_19.
Motzev, M. (2018b) ‘Statistical Learning Networks in Simulations for Business Training and Education’. In: Developments in Business Simulation and Experiential Learning. Vol. 45. Proceedings of the Annual ABSEL Conference, Seattle, WA, pp. 291-301.
Motzev, M. (2019) ‘Prediction Accuracy - A Measure of Simulation Reality’. Vanguard Scientific Instruments in Management, Vol. 15.
Motzev, M. and Pamukchieva, O. (2021) ‘Accuracy in Business Simulations’. In: Wardaszko M. (ed.) SIMULATION & GAMING Through Times and Across Disciplines. Lecture Notes in Computer Science, vol 11988. Springer, pp. 115-126. DOI: 10.1007/978-3-030-72132-9.
Motzev, M. (2021) Business Forecasting: A Contemporary Decision Making Approach. 2nd edition. Amazon Kindle. ASIN B09438TP9M, ISBN 978-1-7370258-0-1
Müller, J-A. and Lemke, F. (1995) ‘Self-Organizing modelling and decision support in economics’. In: Proceedings of the IMACS Symposium on Systems Analysis and Simulation. Gordon and Breach Publ., pp. 135-138.
Müller, J-A. and Lemke, F. (2003) Self-Organizing Data Mining: An Intelligent Approach To Extract Knowledge From Data. Trafford Publishing, Canada.
Nicholson, G. E. (1960) ‘Prediction in future samples’. In: Olkin I. et al. (eds) Contributions to Probability and Statistics. Stanford University Press
Onwubolu, G. (ed) (2009) Hybrid Self-Organizing Modeling Systems. Springer-Verlag Berlin Heidelberg.
Samuel, A. (1959) ‘Some Studies in Machine Learning Using the Game of Checkers’. IBM Journal of Research and Development, Vol. 3, No. 3, July, pp. 210-229, DOI: 10.1147/rd.33.0210.
Schlesinger, S. et al. (1979) ‘Terminology for Model Credibility’. Simulation 32(3), pp. 103-104.
Stone, M. (1974) ‘Cross-Validatory Choice and Assessment of Statistical Predictions, Cross-Validation and Multinomial Prediction’. Journal of the Royal Statistical Society, Vol.36, No.2, pp. 111-147
Stone, M. (1977a) ‘An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike's Criterion’. Journal of the Royal Statistical Society, Vol. 39, No.1, pp. 44–47.
Stone, M. (1977b) ‘Asymptotics For and Against Cross-Validation’. Biometrika, Vol. 64, No.1, pp. 29-35.
Symposium (1951). ‘The need and means of cross-validation’. Educ. & Psychol. Measurement, 11.
Turing, A. (1950) ‘I.—COMPUTING MACHINERY AND INTELLIGENCE’. Mind, Volume LIX, Issue 236, October, pp. 433–460, DOI: 10.1093/mind/LIX.236.433. Available at: https://academic.oup.com/mind/article/LIX/236/433/986238 (Accessed: 02 November 2023).
Wherry, R. J. (1931) ‘A new formula for predicting the shrinkage of the multiple correlation coefficient’. Ann. Math. Statist., 2, pp. 440-457.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
By submitting a paper for publishing the authors hereby comply with the following provisions: 1. The authors retain the copyrights and only give the journal the right for first publication while licensing the work under Creative Commons Attribution License, which grants permissions to others to share the contribution citing this journal as first publication of the text. 2. The authors may enter separate, additional contractual relations for non-exclusive distribution of the published version of the work in this journal (e.g. to upload it in an institutional depository, or to be published in a book), given that they cite the first publication in this journal. 3. The authors are allowed and are encouraged to publish their works online (e.g. to upload it in an institutional depository, personal websites, social networks, etc.) before, during, and after the submission of the paper here, because this may lead to productive exchange, as well as earlier and larger referencing of the published works (see The Effect of Open Access).