Dynamic network functional comparison via approximate-bisimulation

• Autorzy: Francesco Donnarumma , Aniello Murano , Roberto Prevete
• Języki publikacji: EN

Abstrakty

EN

It is generally unknown how to formally determine whether different neural networks have a similar behaviour. This question intimately relates to the problem of finding a suitable similarity measure to identify bounds on the input-output response distances of neural networks, which has several interesting theoretical and computational implications. For example, it can allow one to speed up the learning processes by restricting the network parameter space, or to test the robustness of a network with respect to parameter variation. In this paper we develop a procedure that allows for comparing neural structures among them. In particular, we consider dynamic networks composed of neural units, characterised by non-linear differential equations, described in terms of autonomous continuous dynamic systems. The comparison is established by importing and adapting from the formal verification setting the concept of δ-approximate bisimulations techniques for non-linear systems. We have positively tested the proposed approach over continuous time recurrent neural networks (CTRNNs). (original abstract)

Słowa kluczowe

EN

Network methods; Dynamical systems; Neural networks

PL

Metody sieciowe; Układy dynamiczne; Sieci neuronowe

• Czasopismo: Control and Cybernetics
• Rocznik: 2015
• Tom: 44
• Numer: nr 1
• Strony: 99--127

Twórcy

autor

Francesco Donnarumma

• Institute of Cognitive Sciences and Technologies, National Research Council of Italy Via S. Martino della Battaglia, Rome, Italy

autor

Aniello Murano

• Dipartimento di Ingegneria Elettrica e Tecnologie dell'Informazione, Universit`a degli Studi di Napoli Federico II Via Claudio, Napoli, Italy

autor

Roberto Prevete

• Dipartimento di Ingegneria Elettrica e Tecnologie dell'Informazione, Universit`a degli Studi di Napoli Federico II Via Claudio, Napoli, Italy

Bibliografia

• Albertini, F. and Sontag, E.D. (1993) For neural networks, function determines form. Neural Networks, 6 (7), 975-990.
• Albertini, F., Sontag, E.D. and Maillot, V. (1993) Uniqueness of weights for neural networks. In: Artificial Neural Networks with Applications in Speech and Vision. Chapman and Hall, 115-125.
• Beer, R.D. (1995) On the dynamics of small continuous-time recurrent neural networks. Adaptive Behavior, 3 (4), 469-509.
• Beer, R.D. (2006) Parameter space structure of continuous-time recurrent neural networks. Neural Computation, 18 (12), 3009-3051.
• Birch, M.C., Quinn, R.D., Hahm, G., Phillips, S.M., Drennan, B.T., Fife, A.J., Beer, R.D., Yu, X., Garverick, S.L., Laksanacharoen, S., Pollack, A.J. and Ritzmann, R.E. (2002) Cricket-based robots. IEEE Robotics and Automation Magazine, 9 (4), 20-30.
• Bishop, C.M. (2006) Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA.
• Brihaye, T. (2006) Verification and Control of O minimal hybrid systems and weighted timed automata. Ph.D. thesis, Universit´e de Mons-Hainaut.
• Cao, J., Huang, D.S. and Qu, Y. (2005) Global robust stability of delayed recurrent neural networks. Chaos, Solitons and Fractals, 23 (1), 221 - 229.
• Cao, J. and Wan, Y. (2014) Matrix measure strategies for stability and synchronization of inertial bam neural network with time delays. Neural Networks, 53, 165-172.
• Casagrande, A., Dreossi, T. and Piazza, C. (2012) Hybrid automata and ǫ-analysis on a neural oscillator. In: Proceedings of the First International Workshop on Hybrid Systems and Biology. Newcastle Upon Tyne, UK, EPTCS, 58-72.
• Casagrande, A., Piazza, C. and Policriti, A. (2009) Discrete semantics for hybrid automata. Discrete Event Dynamic Systems, 19 (4), 471-493.
• Chandrasekar, A., Rakkiyappan, R., Cao, J. and Lakshmanan, S. (2014) Synchronization of memistor based recurrent neural networks with two delay components based on second-order reciprocally convex approach. Neural Networks, 57, 79-93.
• Chersi, F., Donnarumma, F. and Pezzulo, G. (2013) Mental imagery in the navigation domain: a computational model of sensory-motor simulation mechanisms. Adaptive Behavior, 21 (4), 251-262.
• Clarke, E.M., Grumberg, O. and Peled, D.A. (2000) Model Checking. The MIT Press.
• De Falco, I., Della Cioppa, A., Donnarumma, F., Maisto, D., Prevete, R. and Tarantino, E. (2008) CTRNN parameter learning using Differential Evolution. In: M. Ghallab, C.D. Spyropoulos, N. Fakotakis and N. Avouris, eds., 18th European Conference on Artificial Intelligence of Frontiers in Artificial Intelligence and Applications. Frontiers in Artificial Intelligence and Applications, 178. IOS Press, Amsterdam, 783-784.
• De Jong, H., Gouz´e, J.L., Hernandez, C., Page, M., Sari, T. and Geiselmann, J. (2004) Qualitative simulation of genetic regulatory networks using piecewise-linear models. Bulletin of Mathematical Biology, 66 (2), 301- 340.
• DiMattina, C. and Zhang, K. (2010) How to modify a neural network gradually without changing its input-output functionality. Neural Computation, 22 (1), 1-47.
• Donnarumma, F., Prevete, R. and Trautteur, G. (2010) How and over what timescales does neural reuse actually occur? Commentary on "neural re-use as a fundamental organizational principle of the brain", by Michael L. Anderson. Behavioral and Brain Sciences, 33 (04), 272-273.
• Donnarumma, F., Prevete, R. and Trautteur, G. (2012) Programming in the brain: a neural Network theoretical framework. Connection Science, 24 (2-3), 71-90.
• Dunn, N.A., Lockery, S.R., Pierce-Shimomura, J.T. and Conery, J.S. (2004) A neural network model of chemotaxis predicts functions of synaptic connections in the nematode caenorhabditis elegans. Journal of Computational Neuroscience, 17 (2), 137-147.
• Fainekos, G.E., Girard, A. and Pappas, G.J. (2007) Hierarchical synthesis of hybrid controllers from temporal logic specifications. In: Hybrid systems: computation and control. Springer, 203-216.
• Fefferman, C. and Markel, S. (1994) Recovering a feed-forward net from its output. In: Advances In Neural Information Processing Systems (NIPS). Morgan Kaufmann, 335-342.
• Franzle, M. (1999) Analysis of hybrid systems: An ounce of realism can save an infinity of states. In: Computer Science Logic. Springer, 126-139.
• Funahashi, K. and Nakamura, Y. (1993) Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6 (6), 801-806.
• Gardner, M. and Dorling, S. (1998) Artificial neural networks (the multilayer perception) - a review of applications in the atmospheric sciences. Atmospheric Environment, 32 (14-15), 2627-2636.
• Ghosh, R. and Tomlin, C.J. (2001) Lateral inhibition through delta-notch signaling: A piecewise affine hybrid model. In: Hybrid Systems: Computation and Control. Springer, 232-246.
• Girard, A. and Pappas, G. (2007a) Approximation metrics for discrete and continuous systems. Automatic Control, IEEE Transactions on, 52 (5), 782-798.
• Girard, A. and Pappas, G.J. (2005) Approximate bisimulations for nonlinear dynamical systems. In: Proceedings of the 44th IEEE Conference on Decision and Control, and the European Control Conference 2005.. IEEE Press, 684- 689.
• Girard, A. and Pappas, G.J. (2007b) Approximation metrics for discrete and continuous systems. Automatic Control, IEEE Transactions on, 52 (5), 782-798.
• Gupta, M.M., Homma, N. and Jin, L. (2003) Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory. John Wiley & Sons, Inc., New York, NY, USA.
• Hale, J.K. and Koc¸ac, H. (1991) Dynamics and Bifurcations. SpringerVerlag.
• Haschke, R. (2004) Input space bifurcation manifolds of recurrent neural networks. Ph.D. thesis, Bielefeld University, Neuroinformatics Group, Faculty of Technology, Bielefeld, Germany.
• He, W. and Cao, J. (2009) Exponential synchronization of chaotic neural networks: a matrix measure approach. Nonlinear Dynamics, 55 (1-2), 55-65.
• Hopfield, J.J. and Tank, D.W. (1986) Computing with neural circuits: A model. Science, 233, 625-633.
• Horne, B.G. and Giles, C.L. (1995) An experimental comparison of recurrent neural networks. In: G. Tesauro, D. Touretzky and T. Leen, eds., Advances in Neural Information Processing Systems, 7. MIT Press, 697-704.
• Izhikevich, E.M. (2007) Dynamical Systems in Neuroscience: the Geometry of Excitability and Bursting. Computational Neuroscience. MIT Press, Cambridge, Mass., London.
• Lall, S., Marsden, J.E. and Glavascaronki, S. (2002) A subspace approach to balanced truncation for model reduction of nonlinear control systems. International Journal of Robust and Nonlinear Control, 12 (6), 519- 535.
• Liberzon, D., Sontag, E.D. and Wang, Y. (2002) Universal construction of feedback laws achieving iss and integral-iss disturbance attenuation. Systems & Control Letters, 46 (2), 111-127.
• MacGregor, R. (2012) Neural and Brain Modeling. Elsevier.
• MacGregor, R.J. and Tajchman, G. (1988) Theory of dynamic similarity in neuronal systems. Journal of Neurophysiology, 60 (2), 751-768.
• Marder, E. and Goaillard, J.M. (2006) Variability, compensation and homeostasis in neuron and network function. Nature Reviews Neuroscience, 7 (7), 563-574.
• Montone, G., Donnarumma, F. and Prevete, R. (2011) A robotic scenario for programmable fixed weight neural networks exhibiting multiple behaviors. In: A. Dobnikar, U. Lotric and B. ˇSter, eds., Adaptive and Natural Computing Algorithms, Lecture Notes in Computer Science, 6593. Springer Berlin / Heidelberg, 250-259.
• Munakata, T. (1997) Fundamentals of the New Artificial Intelligence: Beyond Traditional Paradigms. Springer-Verlag New York, Inc., Secaucus, NJ, USA.
• Neruda, R. (2000) Genetic algorithms and neural networks: Making use of parameter space symmetries. In: Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE - INNS - ENNS International Joint Conference on, 1, 293-298.
• Paine, R.W. and Tani, J. (2004) Motor primitive and sequence selforganization in a hierarchical recurrent neural network. Neural Networks, 17 (8-9), 1291-1309. New Developments in Self Organizing Systems.
• Parrilo, P.A. (2003) Structured semidefinite programs and semialgebraic geometry methods In robustness and optimization. Ph.D. thesis, California Institute of Technology, Pasadena, CA.
• Pezzulo, G., Donnarumma, F. and Dindo, H. (2013) Human sensorimotor communication: A theory of signaling in online social interactions. PLoS ONE, 8 (11), e79876.
• Prabhakar, P., Vladimerou, V., Viswanathan, M. and Dullerud, G.E. (2009) Verifying tolerant systems using polynomial approximations. In: Proceedings of the 30th IEEE Real-Time Systems Symposium. IEEE Press, 181- 190.
• Prajna, S., Papachristodoulou, A. and Parrilo, P.A. (2002) Introducing sostools: a general purpose sum of squares programming solver. In: Decision and Control 2002, Proceedings of the 41st IEEE Conference on, 1. IEEE Press, 741-746.
• Prinz, A.A., Bucher, D. and Marder, E. (2004) Similar network activity from disparate circuit parameters. Nature Neuroscience, 7 (12), 1345-1352.
• Ratschan, S. (2010) Safety verification of non-linear hybrid systems is quasisemidecidable. In: Theory and Applications of Models of Computation. Springer, 397-408.
• Rolls, E.T., Stringer, S.M. and Elliot, T. (2006) Entorhinal cortex grid cells can map to hippocampal place cells by competitive learning. Network: Computation in Neural Systems, 17 (4), 447-465.
• Sporns, O. (2011) The human connectome: a complex network. Annals of the New York Academy of Sciences, 1224 (1), 109-125.
• Tino, P., Horne, B.G. and Giles, C.L. (2001) Attractive periodic sets in discrete time recurrent networks (with emphasis on fixed point stability and bifurcations in two-neuron networks). Neural Computation, 13, 1379-1414.
• Wei, H. and Amari, S.i. (2008) Dynamics of learning near singularities in radial basis function networks. Neural Networks, 21 (7), 989-1005.
• Wilson, H.R. (1999) Simplified dynamics of human and mammalian neocortical neurons. Journal of Theoretical Biology, 200 (4), 375-388.
• Wu, M.C., David, S.V. and Gallant, J.L. (2006) Complete functional characterization of sensory neurons by system identification. Annual Review of Neuroscience, 29, 477-505.
• Yu, W. and Yao, L. (2007) Global robust stability of neural networks with time varying delays. Journal of Computational and Applied Mathematics, 206 (2), 679 - 687.
• Zhang, Y. (1994) A foundation for the design and analysis of robotic systems and behaviors. Ph.D. thesis, University of British Columbia.