Eigen Artificial Neural Networks
Description
This work has its origin in intuitive physical and statistical considerations. The problem of optimizing an artificial neural network is treated as a physical system, composed of a conservative vector force field. The derived scalar potential is a measure of the potential energy of the network, a function of the distance between predictions and targets.
Starting from some analogies with wave mechanics, the description of the system is justified with an eigenvalue equation that is a variant of the Schrõdinger equation, in which the potential is defined by the mutual information between inputs and targets. The weights and parameters of the network, as well as those of the state function, are varied so as to minimize energy, using an equivalent of the variational theorem of wave mechanics. The minimum energy thus obtained implies the principle of minimum mutual information (MinMI). We also propose a definition of the work produced by the force field to bring a network from an arbitrary probability distribution to the potential-constrained system. At the end of the discussion we expose a recursive procedure that allows to refine the state function and bypass some initial assumptions.
The results demonstrate how the minimization of energy effectively leads to a decrease in the average error between network and target predictions.
Files
EANN-article-en-2-columns.pdf
Files
(319.4 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:965e0a0fcba8486340b999b5e95b09d6
|
319.4 kB | Preview Download |
Additional details
Identifiers
References
- Francisco Yepes Barrera. Búsqueda de la estructura óptima de redes neurales con algoritmos genéticos y simulated annealing. verificación con el benchmark proben1. In- teligencia Artificial, Revista Iberoamericana de IA, 11(34):41–61, 2007.
- Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
- Alex Finnegan and Jun S. Song. Maximum entropy methods for extracting the learned features of deep neural networks. PLOS Computational Biology, 13(10):e1005836, October 2017. ISSN 1553-7358. doi: 10. 1371/journal.pcbi.1005836 .
- Jeffrey D. Fitzgerald, Lawrence C. Sin- cich, and Tatyana O. Sharpee. Mini- mal Models of Multidimensional Computa- tions. PLOS Computational Biology, 7(3): e1001111, March 2011. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1001111 .
- Lan Gao and Youwei Hu. Multi-target match- ing based on niching genetic algorithm. JCSNS International Journal of Computer Science and Network Security, 6(7A), July 2006.
- Amir Globerson and Naftali Tishby. The Min- imum Information Principle for Discriminative Learning. In Proceedings of the 20th Confer- ence on Uncertainty in Artificial Intelligence, UAI '04, pages 193–200, Arlington, Virginia, United States, 2004. AUAI Press. ISBN 978- 0-9749039-0-3. event-place: Banff, Canada.
- Amir Globerson, Eran Stark, Eilon Vaadia, and Naftali Tishby. The minimum informa- tion principle and its application to neural code analysis. Proceedings of the National Academy of Sciences of the United States of America, PNAS, 106(9), march 2009.
- Ira N. Levine. Quantum chemistry. Pearson, Boston, seventh edition edition, 2014. ISBN 978-0-321-80345-0.
- Javier R. Movellan and James L. McClel- land. Learning Continuous Probability Distri- butions with Symmetric Diffusion Networks. Cognitive Science, 17(4):463–496, October 1993. ISSN 03640213. doi: 10.1207/ s15516709cog1704-1 .
- Joseph C. Park and Salahalddin T. Abusalah. Maximum Entropy: A Special Case of Minimum Cross-entropy Applied to Nonlinear Estimation by an Artificial Neural Network. Complex Systems, 11, 1997.
- Carlos A. L. Pires and Rui A. P. Perdi- gao. Minimum Mutual Information and Non- Gaussianity Through the Maximum Entropy Method: Theory and Properties. Entropy, 14(6):1103–1126, June 2012. ISSN 1099- 4300. doi: 10.3390/e14061103 .
- Lutz Prechelt. Proben1 - a set of neural net- work benchmark problems and benchmark- ing rules. Technical Report 21/94, Fakültat für Informatik, Universität Karlsruhe, 76128 Karlsruhe, Germany, September 1994.
- Zhang Xiaodong. Evaluation model and simulation of basketball teaching quality based on maximum entropy neural network. page 5, 2014.
- Dongxin Xu. Energy, entropy and informa- tion potential for neural computation. PhD thesis, University of Florida, 1999.
- Yan Zhang, Mete Ozay, Zhun Sun, and Takayuki Okatani. Information Potential Auto-Encoders. arXiv:1706.04635 [cs, math, stat], June 2017. arXiv: 1706.04635.