# American Institute of Mathematical Sciences

doi: 10.3934/dcdsb.2022097
Online First

Online First articles are published articles within a journal that have not yet been assigned to a formal issue. This means they do not yet have a volume number, issue number, or page numbers assigned to them, however, they can still be found and cited using their DOI (Digital Object Identifier). Online First publication benefits the research community by making new scientific discoveries known as quickly as possible.

Readers can access Online First articles via the “Online First” tab for the selected journal.

## Permutation binary neural networks: Analysis of periodic orbits and its applications

 Department of Electrical and Electronic Engineering, HOSEI University, Japan

* Corresponding author: Toshimichi Saito

Received  January 2022 Revised  April 2022 Early access May 2022

This paper presents a permutation binary neural network characterized by local binary connection, global permutation connection, and the signum activation function. The dynamics is described by a difference equation of binary state variables. Depending on the connection, the network generates various periodic orbits of binary vectors. The binary/permutation connection brings benefits to precise analysis and to FPGA based hardware implementation. In order to consider the periodic orbits, we introduce three tools: a composition return map for visualization of the dynamics, two feature quantities for classification of periodic orbits, and an FPGA based hardware prototype for engineering applications. Using the tools, we have analyzed all the 6-dimensional networks. Typical periodic orbits are confirmed experimentally.

Citation: Hotaka Udagawa, Taiji Okano, Toshimichi Saito. Permutation binary neural networks: Analysis of periodic orbits and its applications. Discrete and Continuous Dynamical Systems - B, doi: 10.3934/dcdsb.2022097
##### References:
 [1] M. Adachi and K. Aihara, Associative dynamics in a chaotic neural network, Neural Networks, 10 (1997), 83-98. [2] S. Anzai, T. Suzuki and T. Saito, Dynamic binary neural networks with time-variant parameters and switching of desired periodic orbits, Neurocomputing, 457 (2021), 357-364. [3] S. Aoki, S. Koyama and T. Saito, Theoretical analysis of dynamic binary neural networks with simple sparse connection, Neurocomputing, 341 (2019), 149-155. [4] L. Appeltant, M. C. Soriano, G. Van der Sande, J. Danckaert, S. Massar, J. Dambre, B. Schrauwen, C. R. Mirasso and I. Fischer, Information processing using a single dynamical node as complex system, Nat. Commun., 2 (2011), 468. [5] P. Arena, L. Patané and A. G. Spinosa, A nullcline-based control strategy for PWL-shaped oscillators, Nonlinear Dyn., 97 (2019), 1011-1033. [6] L. O. Chua, A Nonlinear Dynamics Perspective of Wolfram's New Kind of Science I, World Scientific, 2006. [7] W. Holderbaum, Application of neural network to hybrid systems with binary inputs, IEEE Trans. Neural Netw., 18 (2007), 1254-1261. [8] J. J. Hopfield, Neural networks and physical systems with emergent collective computation abilities, Proc. Nat. Acad. Sci. U.S.A., 79 (1982), 2554-2558.  doi: 10.1073/pnas.79.8.2554. [9] J. J. Hopfield and D. W. Tank, "Neural" computation of decisions optimization problems, Biol. Cybernet., 52 (1985), 141-152.  doi: 10.1007/BF00339943. [10] N. Horimoto and T. Saito, Analysis of digital spike maps based on bifurcating neurons, NOLTA, IEICE, 3 (2012), 596-605. [11] S. Koyama and T. Saito, Guaranteed storage and stabilization of desired binary periodic orbits in three-layer dynamic binary neural networks, Neurocomputing, 416 (2020), 12-18. [12] C. G. Langton, Computation at the edge of chaos: Phase transition and emergent computation, Physica D, 42 (1990), 12-37. [13] A. Lasota and M. C. Mackey, Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics, Springer-Verlag, 1994. doi: 10.1007/978-1-4612-4286-4. [14] M. Lodi, A. L. Shilnikov and M. Storace, Design principles for central pattern generators with preset rhythms, IEEE Trans. Neural Netw. Learn. Syst., 31 (2020), 3658-3669.  doi: 10.1109/TNNLS.2019.2945637. [15] R. May, Simple mathematical models with very complicated dynamics, Nature, 261 (1976), 259-467. [16] A. N. Michel and J. A. Farrell, Associative memories via artificial neural networks, IEEE Control Systems Magazine, 10 (1990), 6-17. [17] A. N. Michel, J. A. Farrell and H.-F. Sun, Analysis and synthesis techniques for Hopfield type synchronous discrete time neural networks with application to associative memory, IEEE Trans. Circuits Systs., 37 (1990), 1356-1366.  doi: 10.1109/31.62410. [18] E. Ott, Chaos in Dynamical Systems, Cambridge, 2002. doi: 10.1017/CBO9780511803260. [19] J. Rodriguez, M. Rivera, J. W. Kolar and and P. W. Wheeler, A review of control and modulation methods for matrix converters, IEEE Trans. Ind. Electron., 59 (2012), 58-70.  doi: 10.1109/TIE.2011.2165310. [20] D. Roy Chowdhury, S. Basu, I. Sengupta and P. Pal Chaudhuri, Design of CAECC - cellular automata based error correcting code, IEEE Trans. Comput., 43 (1994), 759-764.  doi: 10.1109/12.286310. [21] T. Saito, On a coupled relaxation oscillator, IEEE Trans. Circuits. Syst., 35 (1988), 1147-1155.  doi: 10.1109/31.7575. [22] R. Sato and T. Saito, Stabilization of desired periodic orbits in dynamic binary neural networks, Neurocomputing, 248 (2017), 19-27. [23] E. J. Schiessler, R. C. Aydin, K. Linka and C. J. Cyron, Neural network surgery: Combining training with topology optimization, Neural Networks, 144 (2021), 384-393. [24] M. Schüle and R. Stoop, A full computation-relevant topological dynamics classification of elementary cellular automata, Chaos, 22 (2012), 043143, 10 pp. doi: 10.1063/1.4771662. [25] T. Suzuki and T. Saito, Synthesis of three-layer dynamic binary neural networks for control of hexapod walking robots, Proceedings of the IEEE/CNNA, 2021. [26] G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano and A. Hirose, Recent advances in physical reservoir computing: A review, Neural Networks, 115 (2019), 100-123. [27] M. Thor, T. Kulvicius and P. Manoonpong, Generic neural locomotion control framework for legged robots, IEEE Trans. Neural Netw. Learn. Syst., 32 (2021), 4013-4025. [28] H. Uchida, Y. Oishi and T. Saito, A simple digital spiking neural network: synchronization and spike-train approximation, Discrete Contin. Dyn. Syst. Ser. S, 14 (2021), 1479-1494.  doi: 10.3934/dcdss.2020374. [29] M. Wada, J. Kuroiwa and S. Nara, Completely reproducible description of digital sound data with cellular automata, Phys. Lett. A, 306 (2002), 110-115.  doi: 10.1016/S0375-9601(01)00610-7. [30] S. Wolfram, Cellular Automata and Complexity: Collected Papers, CRC Press, 2018. [31] O. Yilmaz, Symbolic computation using cellular automata-based hyperdimensional computing, Neural Computation, 27 (2015), 2661-2692.  doi: 10.1162/NECO_a_00787. [32]

show all references

##### References:
 [1] M. Adachi and K. Aihara, Associative dynamics in a chaotic neural network, Neural Networks, 10 (1997), 83-98. [2] S. Anzai, T. Suzuki and T. Saito, Dynamic binary neural networks with time-variant parameters and switching of desired periodic orbits, Neurocomputing, 457 (2021), 357-364. [3] S. Aoki, S. Koyama and T. Saito, Theoretical analysis of dynamic binary neural networks with simple sparse connection, Neurocomputing, 341 (2019), 149-155. [4] L. Appeltant, M. C. Soriano, G. Van der Sande, J. Danckaert, S. Massar, J. Dambre, B. Schrauwen, C. R. Mirasso and I. Fischer, Information processing using a single dynamical node as complex system, Nat. Commun., 2 (2011), 468. [5] P. Arena, L. Patané and A. G. Spinosa, A nullcline-based control strategy for PWL-shaped oscillators, Nonlinear Dyn., 97 (2019), 1011-1033. [6] L. O. Chua, A Nonlinear Dynamics Perspective of Wolfram's New Kind of Science I, World Scientific, 2006. [7] W. Holderbaum, Application of neural network to hybrid systems with binary inputs, IEEE Trans. Neural Netw., 18 (2007), 1254-1261. [8] J. J. Hopfield, Neural networks and physical systems with emergent collective computation abilities, Proc. Nat. Acad. Sci. U.S.A., 79 (1982), 2554-2558.  doi: 10.1073/pnas.79.8.2554. [9] J. J. Hopfield and D. W. Tank, "Neural" computation of decisions optimization problems, Biol. Cybernet., 52 (1985), 141-152.  doi: 10.1007/BF00339943. [10] N. Horimoto and T. Saito, Analysis of digital spike maps based on bifurcating neurons, NOLTA, IEICE, 3 (2012), 596-605. [11] S. Koyama and T. Saito, Guaranteed storage and stabilization of desired binary periodic orbits in three-layer dynamic binary neural networks, Neurocomputing, 416 (2020), 12-18. [12] C. G. Langton, Computation at the edge of chaos: Phase transition and emergent computation, Physica D, 42 (1990), 12-37. [13] A. Lasota and M. C. Mackey, Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics, Springer-Verlag, 1994. doi: 10.1007/978-1-4612-4286-4. [14] M. Lodi, A. L. Shilnikov and M. Storace, Design principles for central pattern generators with preset rhythms, IEEE Trans. Neural Netw. Learn. Syst., 31 (2020), 3658-3669.  doi: 10.1109/TNNLS.2019.2945637. [15] R. May, Simple mathematical models with very complicated dynamics, Nature, 261 (1976), 259-467. [16] A. N. Michel and J. A. Farrell, Associative memories via artificial neural networks, IEEE Control Systems Magazine, 10 (1990), 6-17. [17] A. N. Michel, J. A. Farrell and H.-F. Sun, Analysis and synthesis techniques for Hopfield type synchronous discrete time neural networks with application to associative memory, IEEE Trans. Circuits Systs., 37 (1990), 1356-1366.  doi: 10.1109/31.62410. [18] E. Ott, Chaos in Dynamical Systems, Cambridge, 2002. doi: 10.1017/CBO9780511803260. [19] J. Rodriguez, M. Rivera, J. W. Kolar and and P. W. Wheeler, A review of control and modulation methods for matrix converters, IEEE Trans. Ind. Electron., 59 (2012), 58-70.  doi: 10.1109/TIE.2011.2165310. [20] D. Roy Chowdhury, S. Basu, I. Sengupta and P. Pal Chaudhuri, Design of CAECC - cellular automata based error correcting code, IEEE Trans. Comput., 43 (1994), 759-764.  doi: 10.1109/12.286310. [21] T. Saito, On a coupled relaxation oscillator, IEEE Trans. Circuits. Syst., 35 (1988), 1147-1155.  doi: 10.1109/31.7575. [22] R. Sato and T. Saito, Stabilization of desired periodic orbits in dynamic binary neural networks, Neurocomputing, 248 (2017), 19-27. [23] E. J. Schiessler, R. C. Aydin, K. Linka and C. J. Cyron, Neural network surgery: Combining training with topology optimization, Neural Networks, 144 (2021), 384-393. [24] M. Schüle and R. Stoop, A full computation-relevant topological dynamics classification of elementary cellular automata, Chaos, 22 (2012), 043143, 10 pp. doi: 10.1063/1.4771662. [25] T. Suzuki and T. Saito, Synthesis of three-layer dynamic binary neural networks for control of hexapod walking robots, Proceedings of the IEEE/CNNA, 2021. [26] G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano and A. Hirose, Recent advances in physical reservoir computing: A review, Neural Networks, 115 (2019), 100-123. [27] M. Thor, T. Kulvicius and P. Manoonpong, Generic neural locomotion control framework for legged robots, IEEE Trans. Neural Netw. Learn. Syst., 32 (2021), 4013-4025. [28] H. Uchida, Y. Oishi and T. Saito, A simple digital spiking neural network: synchronization and spike-train approximation, Discrete Contin. Dyn. Syst. Ser. S, 14 (2021), 1479-1494.  doi: 10.3934/dcdss.2020374. [29] M. Wada, J. Kuroiwa and S. Nara, Completely reproducible description of digital sound data with cellular automata, Phys. Lett. A, 306 (2002), 110-115.  doi: 10.1016/S0375-9601(01)00610-7. [30] S. Wolfram, Cellular Automata and Complexity: Collected Papers, CRC Press, 2018. [31] O. Yilmaz, Symbolic computation using cellular automata-based hyperdimensional computing, Neural Computation, 27 (2015), 2661-2692.  doi: 10.1162/NECO_a_00787. [32]
An example of 6-dimensional ECA (RN212) and a sequence of binary vectors. Black and white squares denote output $+1$ and $-1$, respectively
Networks and BPOs. (a) SBNN in Example 1. Red and blue branches denote positive and negative connections, respectively. (b) PBNN in Example 2 (CN6, P231465). Black branches denote permutation connection
1st map $f_1$ for SBNN, CN6. (a) 1st map of 64 points. (b) BPO(red) with period 6 and EPPs (green)
Cmaps for PBNN. (a) The 1st map $f_1$ for CN6. (b) The 2nd map $f_2$ for P231465. (c) Cmap $f$ of 64 points. (d) BPO with period 12 (red) and EPPs (green). BPO with period 4 (blue)
Feature plane and three segments $S_d$, $S_t$, and $S_l$. Red cross: SBNN, Example 1 ($\alpha = 6/64$, $\beta = 12/64$). Red circle: PBNN, Example 2 ($\alpha = 12/64$, $\beta = 40/64$)
1st map of 8 SBNNs with MBPO (red) and EPPs (green)
Cmap of PBNNs with MBPO (red) and EPPs (green). CN0: P513246. CN1: P413625. CN2: P524361. CN3: P315462. CN4: P254136. CN5: P461253. CN6: P126354. CN7: P651324
Feature planes. Red cross: SBNN, Red circle: PBNN in Fig. 7. CN0(19 points). CN1(79). CN2(49). CN3(78). CN4(79). CN5(53). CN6(78). CN7(26)
Experimental setup: FPGA and Analog discovery
Measured waveforms of MBPOs in the FPGA board. (a) MBPO1 with period 6 from SBNN CN6. (b) MBPO2 with period 20 from PBNN CN6 P126354
 [1] Larry Turyn. Cellular neural networks: asymmetric templates and spatial chaos. Conference Publications, 2003, 2003 (Special) : 864-871. doi: 10.3934/proc.2003.2003.864 [2] Leslaw Skrzypek, Yuncheng You. Feedback synchronization of FHN cellular neural networks. Discrete and Continuous Dynamical Systems - B, 2021, 26 (12) : 6047-6056. doi: 10.3934/dcdsb.2021001 [3] T.K. Subrahmonian Moothathu. Homogeneity of surjective cellular automata. Discrete and Continuous Dynamical Systems, 2005, 13 (1) : 195-202. doi: 10.3934/dcds.2005.13.195 [4] Achilles Beros, Monique Chyba, Oleksandr Markovichenko. Controlled cellular automata. Networks and Heterogeneous Media, 2019, 14 (1) : 1-22. doi: 10.3934/nhm.2019001 [5] Ying Sue Huang, Chai Wah Wu. Stability of cellular neural network with small delays. Conference Publications, 2005, 2005 (Special) : 420-426. doi: 10.3934/proc.2005.2005.420 [6] Marcus Pivato. Invariant measures for bipermutative cellular automata. Discrete and Continuous Dynamical Systems, 2005, 12 (4) : 723-736. doi: 10.3934/dcds.2005.12.723 [7] Cheng-Hsiung Hsu, Suh-Yuh Yang. Structure of a class of traveling waves in delayed cellular neural networks. Discrete and Continuous Dynamical Systems, 2005, 13 (2) : 339-359. doi: 10.3934/dcds.2005.13.339 [8] Cheng-Hsiung Hsu, Suh-Yuh Yang. Traveling wave solutions in cellular neural networks with multiple time delays. Conference Publications, 2005, 2005 (Special) : 410-419. doi: 10.3934/proc.2005.2005.410 [9] Adnène Arbi, Jinde Cao, Mohssine Es-saiydy, Mohammed Zarhouni, Mohamed Zitane. Dynamics of delayed cellular neural networks in the Stepanov pseudo almost automorphic space. Discrete and Continuous Dynamical Systems - S, 2022  doi: 10.3934/dcdss.2022136 [10] Benedetta Lisena. Average criteria for periodic neural networks with delay. Discrete and Continuous Dynamical Systems - B, 2014, 19 (3) : 761-773. doi: 10.3934/dcdsb.2014.19.761 [11] Achilles Beros, Monique Chyba, Kari Noe. Co-evolving cellular automata for morphogenesis. Discrete and Continuous Dynamical Systems - B, 2019, 24 (5) : 2053-2071. doi: 10.3934/dcdsb.2019084 [12] Ricai Luo, Honglei Xu, Wu-Sheng Wang, Jie Sun, Wei Xu. A weak condition for global stability of delayed neural networks. Journal of Industrial and Management Optimization, 2016, 12 (2) : 505-514. doi: 10.3934/jimo.2016.12.505 [13] Suh-Yuh Yang, Cheng-Hsiung Hsu. Existence of monotonic traveling waves in modified RTD-based cellular neural networks. Conference Publications, 2005, 2005 (Special) : 930-939. doi: 10.3934/proc.2005.2005.930 [14] Chuangxia Huang, Hedi Yang, Jinde Cao. Weighted pseudo almost periodicity of multi-proportional delayed shunting inhibitory cellular neural networks with D operator. Discrete and Continuous Dynamical Systems - S, 2021, 14 (4) : 1259-1272. doi: 10.3934/dcdss.2020372 [15] Yong Zhao, Qishao Lu. Periodic oscillations in a class of fuzzy neural networks under impulsive control. Conference Publications, 2011, 2011 (Special) : 1457-1466. doi: 10.3934/proc.2011.2011.1457 [16] Bernard Host, Alejandro Maass, Servet Martínez. Uniform Bernoulli measure in dynamics of permutative cellular automata with algebraic local rules. Discrete and Continuous Dynamical Systems, 2003, 9 (6) : 1423-1446. doi: 10.3934/dcds.2003.9.1423 [17] Marcelo Sobottka. Right-permutative cellular automata on topological Markov chains. Discrete and Continuous Dynamical Systems, 2008, 20 (4) : 1095-1109. doi: 10.3934/dcds.2008.20.1095 [18] Xinxin Tan, Shujuan Li, Sisi Liu, Zhiwei Zhao, Lisa Huang, Jiatai Gang. Dynamic simulation of a SEIQR-V epidemic model based on cellular automata. Numerical Algebra, Control and Optimization, 2015, 5 (4) : 327-337. doi: 10.3934/naco.2015.5.327 [19] Jan Sieber, Matthias Wolfrum, Mark Lichtner, Serhiy Yanchuk. On the stability of periodic orbits in delay equations with large delay. Discrete and Continuous Dynamical Systems, 2013, 33 (7) : 3109-3134. doi: 10.3934/dcds.2013.33.3109 [20] Răzvan M. Tudoran. On the control of stability of periodic orbits of completely integrable systems. Journal of Geometric Mechanics, 2015, 7 (1) : 109-124. doi: 10.3934/jgm.2015.7.109

2021 Impact Factor: 1.497