Stacked Regression with a Generalization of the Moore-Penrose Pseudoinverse

• Autorzy: Tomasz Górecki , Maciej Łuczak
• Języki publikacji: EN

Abstrakty

EN

In practice, it often happens that there are a number of classification methods. We are not able to clearly determine which method is optimal. We propose a combined method that allows us to consolidate information from multiple sources in a better classifier. Stacked regression (SR) is a method for forming linear combinations of different classifiers to give improved classification accuracy. The Moore-Penrose (MP) pseudoinverse is a general way to find the solution to a system of linear equations. This paper presents the use of a generalization of the MP pseudoinverse of a matrix in SR. However, for data sets with a greater number of features our exact method is computationally too slow to achieve good results: we propose a genetic approach to solve the problem. Experimental results on various real data sets demonstrate that the improvements are efficient and that this approach outperforms the classical SR method, providing a significant reduction in the mean classification error rate. (original abstract)

Słowa kluczowe

EN

Classification methods; Genetic algorithms; Experimental analysis

PL

Metody klasyfikacyjne; Algorytmy genetyczne; Analiza eksperymentalna

• Czasopismo: Statistics in Transition
• Rocznik: 2017
• Tom: 18
• Numer: nr 3
• Strony: 443--458

Twórcy

autor

Tomasz Górecki

• Adam Mickiewicz University in Poznań, Poland

autor

Maciej Łuczak

• Koszalin University of Technology, Poland

Bibliografia

• BAUER, E., KOHAVI, R., (1999). An experimental comparison of voting classification algorithms: bagging, boosting, and variants. Machine Learning, 36, 105-139.
• BEN-ISRAEL, A., GREVILLE, T.N.E. (2003). Generalized inverses. Theory and applications. Springer.
• BERGMANN, G., HOMMEL, G., (1988). Improvements of general multiple test procedures for redundant systems of hypotheses. In Multiple Hypotheses Testing. P. Bauer, G. Hommel and E. Sonnemann (eds.) Springer, 110- 115.
• BREIMAN, L., (1996a). Stacked regression. Machine Learning, 24, 49-64.
• BREIMAN, L., (1996b). Bagging predictors. Machine Learning, 24, 123-140.
• BREIMAN, L., (2001). Random forests. Machine Learning, 45, 5-32.
• BREIMAN, L., FRIEDMAN, J.H., OLSHEN, R.A., STONE, C.J., (1984). Classification and regression trees, Wadsworth, California.
• CHO, S.B., KIM, J.H., (1995). Multiple network fusion using fuzzy logic. IEEE Transactions on Systems, Man, and Cybernetics, 6,497-501.
• DOUMPOS, M., ZOPOUNIDIS, C., (2007). Model combination for credit risk assessment: A stacked generalization approach. Annals of Operations Research, 151,289-306.
• DUIN, R., TAX, D., (2000). Experiments with classifier combining rules. Lecture Notes in Computer Science, 1857, 16-29.
• DZEROSKI, S., ZENKO, B., (2004). Is combining classifiers with stacking better than selecting the best one? Machine Learning, 54, 255-273.
• FRANK, A., ASUNCION, A., (2010). UCI Machine Learning Repository. http: //archive.ics.uci.edu/ml Irvine, CA: University of California, School of Information and Computer Science.
• GARCIA, S., HERRERA, F., (2008). An extension on "statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons. Journal of Machine Learning Research, 9, 2677-2694.
• GLEIM, G., (1984). The profiling of professional football players. Clinical Sport Medicine, 3(1), 185-97.
• GÓRECKI, T., (2005). Effect of choice of dissimilarity measure on classification efficiency with nearest neighbor method. Discussiones Mathematicae Probability and Statistics, 25(2), 217- 239.
• GÓRECKI, T., ŁUCZAK, M., (2013). Linear discriminant analysis with a generalization of Moore-Penrose pseudoinverse. International Journal of Applied Mathematics and Computer Science, 26(2), 463-471.
• HUANG, Y.S., SUEN, C.Y., (1995). A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17, 90-93.
• IMAN, R., DAVENPORT, J., (1980). Approximations of the critical region of the Friedman statistic. Communications in Statistics - Theory and Methods, 9(6), 571-595.
• KUNCHEVA, L.I., (2004). Combining Pattern Classifiers: Methods and Algorithms. Wiley.
• KUNCHEVA, L., WHITAKER, C., (2003). Measures of diversity in classifier ensembles. Machine Learning, 51, 181-207.
• KYRCHEI, I., (2015). Cramer's rule for generalized inverse solutions. In Advances in Linear Algebra Research I. Kyrchei (ed.) Nova Science Publishers, 79-132.
• LEBLANC, M., TIBSHIRANI, R., (1996). Combining estimates in regression and classification. Journal of the American Statistical Association, 91,1641-1650.
• LEDEZMA, A., ALER, R., SANCHIS, A., BORRAJO, D., (2010). GA-stacking: evolutionary stacked generalization. Intelligent Data Analysis, 14, 89-119.
• MARQUES, A., GARCIA, V., SANCHEZ, J., (2012). Exploring the behavior of base classifiers in credit scoring ensembles. Expert Systems with Applications, 39(11), 10244-10250.
• MICHALEWICZ, Z., FOGEL, D.B., (2004). How To Solve It: Modern Heuristics. Springer.
• MOJIRSHEIBANI, M., (2002). A comparison study of some combined classifiers. Communications in Statistics - Simulation and Computation, 31(2), 245-260.
• MORRISON, D.F., (1976). Multivariate Statistical Methods. McGraw- Hill.
• NI, W., BROWN, S., MAN, R., (2009). Stacked partial least squares regression analysis for spectral calibration and prediction. Journal of Chemometrics, 23, 505-517.
• OZAY, M., VURAL, F.T.Y., (2008). On the performance of stacked generalization classifiers. Lecture Notes in Computer Science, 5112, 445-454.
• RAO, C.R., MITRA, S.K., (1971). Generalized Inverse of Matrices and its Applications. Wiley.
• ROKACH, L., (2010). Ensemble-based classifiers. Artificial Intelligence Review, 33, 1-39.
• ROONEY, N., PATTERSON, D., ANAND, S., TSYMBAL, A., (2004). Dynamic integration of regression models. Lecture Notes in Computer Science, 3077, 64-173.
• ROONEY, N., PATTERSON, D., NUGENT, C., (2004). Reduced ensemble size stacking. Tools with Artificial Intelligence. ICTAI 6th IEEE International Conference, 266-271.
• SCHAPIRE, R.E., (1990). The strength of weak learnability. Machine Learning, 5, 197-227.
• SEBER, G.A.F., (1984). Multivariate Observations. New York: Wiley.
• SEHGAL, M.S.B., GONDAL, I., DOOLEY, L., (2005). Stacked regression ensemble for cancer class prediction. Industrial Informatics INDIN 3rd IEEE International Conference, 831-835.
• SESMERO, M., LEDEZMA, A., SANCHIS, A., (2015). Generating ensembles of heterogeneous classifiers using Stacked Generalization. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(1), 21-34.
• SHUNMUGAPRIYA, P., KANMANI, S., (2013). Optimization of stacking ensemble configurations through artificial bee colony algorithm. Swarm and Evolutionary Computation, 12, 24-32.
• SIGLETOS, G., PALIOURAS, G., SPYROPOULOS, C.D., HATZOPOULOS, M., (2005). Combining information extraction systems using voting and stacked generalization. Journal of Machine Learning Research, 6, 1751-1782.
• WERNECKE, K., (1992). A coupling procedure for discrimination of mixed data. Biometrics, 48, 497-506.
• WOLPERT, D., (1992). Stacked generalization. Neural Networks, 5, 241-259.
• VAN DER HEIJDEN, F., DUIN, R.P.W., DE RIDDER, D., TAX. D.M.J., (2004). Classification, Parameter Estimation and State Estimation: An Engineering Approach Using Matlab, New York: Wiley.
• WEBB, A., (2002). Statistical Pattern Recognition. New York: Wiley.
• XU L., JIANG J.H., ZHOU Y.P., WU H.L., SHEN G.L., YU R.Q., (2007). MCCV stacked regression for model combination and fast spectral interval selection in multivariate calibration. Chemometrics and Intelligent Laboratory Systems, 87(2), 226-230.
• ZHANG, H., (2004). The Optimality of Naive Bayes. 17. FLAIRS Conference 2004: Miami Beach, Florida, USA 562-567.