A Comparison between Different Chess Rating Systems for Ranking Evolutionary Algorithms

• Autorzy: Niki Veček , Marjan Mernik , Matej Črepinšek , Dejan Hrnčič
• Języki publikacji: EN

Abstrakty

EN

Chess Rating System for Evolutionary algorithms (CRS4EAs) is a novel method for comparing evolutionary algorithms which evaluates and ranks algorithms regarding the formula from the Glicko-2 chess rating system. It was empirically shown that CRS4EAs can be compared to the standard method for comparing algorithms - null hypothesis significance testing. The following paper examines the applications of chess rating systems beyond Glicko-2. The results of 15 evolutionary algorithms on 20 minimisation problems obtained using the Glicko-2 system were empirically compared to the Elo rating system, Chessmetrics rating system, and German Evaluation Number (DWZ). The results of the experiment showed that Glicko-2 is the most appropriate choice for evaluating and ranking evolutionary algorithms. Whilst other three systems' benefits were mainly the simple formulae, the ratings in Glicko-2 are proven to be more reliable, the detected significant differences are supported by confidence intervals, the inflation or deflation of ratings is easily detected, and the weight of individual results is set dynamically.(original abstract)

Słowa kluczowe

EN

Algorithms; Ranking; Scientific experiment

PL

Algorytmy; Ranking; Eksperyment badawczy

• Czasopismo: Annals of Computer Science and Information Systems
• Rocznik: 2014
• Tom: 2
• Strony: 511--518

Twórcy

autor

Niki Veček

• University of Maribor, Slovenia

autor

Marjan Mernik

• University of Maribor, Slovenia

autor

Matej Črepinšek

• University of Maribor, Slovenia

autor

Dejan Hrnčič

• University of Maribor, Slovenia

Bibliografia

• Brest J., Greiner S., Boskovic B., Mernik M., Zumer V. Self-adapting control parameters in differential evolution: A comparative study on numerical benchmark problems. IEEE Transactions on Evolutionary Computation, 10(6):646-657, 2006.
• Cohen J. The earth is round (p < .05). American psychologist, 49(12):997-1003, 1994.
• Crepinsek M., Liu S. H., Mernik L. A Note on Teaching-Learning-Based Optimization Algorithm. Information Sciences, 212:79-93, 2012.
• Das S., Suganthan P. N. Differential evolution: A survey of the stateof-the-art. IEEE Transactions on Evolutionary Computation, 15(1):4-31, 2011.
• Deutscher Schachbund [Online]. Available: http://www.schachbund.de/wertungsordnung.html
• Dowell M., Jarratt P. A modified regula falsi method for computing the root of an equation. BIT Numerical Mathematics, 11(2):168-174, 1971.
• Dyba T., Kampenes V. B., Sjoberg D. I. A systematic review of statistical power in software engineering experiments. Information and Software Technology, 48(8):745-755, 2006.
• Elo A. E. The rating of chessplayers, past and present (Vol. 3). Batsford, 1978.
• Epitropakis M. G., Plagianakos V. P., Vrahatis M. N.. Balancing the exploration and exploitation capabilities of the differential evolution algorithm. IEEE World Congress on Computational Intelligence 2008, 2686-2693, 2008.
• Evolutionary Algorithms Rating System (Github) [Online]. Available: https://github.com/matejxxx/EARS
• Evolutionary Algorithms Rating System [Online]. Available: http://earatingsystem.appspot.com/
• Friedman M. A comparison of alternative tests of significance for the problem of m rankings. Annals of Mathematical Statistics, 11:86-92, 1940.
• Friedman M. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32:675-701, 1937.
• Gill J. The insignificance of null hypothesis significance testing. Political Research Quarterly, 52(3):647-674, 1999.
• Glickman M. E. A comprehensive guide to chess ratings. American Chess Journal, 3:59-102, 1995.
• Glickman M. E. Dynamic paired comparison models with stochastic variances. Journal of Applied Statistics, 28(6):673-689, 2001.
• Glickman M. E. Example of the Glicko-2 system. Boston University, 2012.
• Glickman M. E. Parameter estimation in large dynamic paired comparison experiments. Journal of the Royal Statistical Society: Series C (Applied Statistics), 48(3):377-394, 1999.
• Glickman M. E. The glicko system. Boston University, 1995.
• Hansen N. The CMA Evolution Strategy: A Comparing Review. Towards a new evolutionary computation, Springer, 75-102, 2006.
• Hooper D, Whyld K. The Oxford Companion to Chess. Oxford University Press, 1992.
• Kitchenham B. A., Pfleeger S. L., Pickard L. M., Jones P. W., Hoaglin D. C., El Emam K., Rosenberg J. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8):721-734, 2002.
• Neyman J., Pearson E. On the problem of the most efficient test of statistical hypothesis. Philosophical Transaction of the Royal Society of London - Series A, 231:289-337, 1933.
• Rao R. V., Savsani V. J., Vakharia D. P. Teaching-Learning-Based Optimization: An optimization method for continuous non-linear large scale problems. Information Sciences, 183(1):1-15, 2012.
• Rastrigin L .A. The convergence of the random search method in the extremal control of a many-parameter system. Automation and Remote Control, 24(10):1337-1342, 1963.
• Rechenberg I. Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Frommann-Holzboog, 1973.
• Sonas J. http://www.chessmetrics.com, Februar 2014.
• Tang K., Li X., Suganthan P. N., Yang z., Weise T. Benchmark Functions for the CEC2010 Special Session and Competition on Large-Scale Global Optimization. Nature Inspired Computation and Applications Laboratory, 2009.
• Tvrdik J. Adaptive differential evolution: application to nonlinear regression. In Proceedings of the International Multiconference on Computer Science and Information Technology, 193-202, 2007.
• Vecek N., Mernik M., Crepinsek M. A Chess Rating System for Evolutionary Algorithms - A New Method for the Comparison and Ranking of Evolutionary Algorithms. Information Sciences, 277:656-679, 2014.
• Zaharie D. A comparative analysis of crossover variants in differential evolution. Proceedings of IMCSIT, 171-181, 2007.