A novel reinforcement learning based routing algorithm for energy management in networks

Çiğdem Eriş; Ömer Melih Gül; Pinar Sarisaray Bölük

doi:10.3934/jimo.2024049

Article Contents

2024, Volume 20, Issue 12: 3678-3696. Doi: 10.3934/jimo.2024049

This issue Previous Article Deep neural networks for probability of default modelling Next Article Online retail platform cooperative promotion strategies with manufacturer competition

A novel reinforcement learning based routing algorithm for energy management in networks

1.
Bahçeşehir University, Department of Computer Engineering, İstanbul, Türkiye
2.
Istanbul Technical University, Informatics Institute, İstanbul, Türkiye
3.
Bahçeşehir University, Department of Software Engineering, İstanbul, Türkiye

^*Corresponding author: ÇİĞDEM ERİŞ
^*Corresponding author: ÇİĞDEM ERİŞ

Received: December 2023

Revised: February 2024

Early access: April 2024

Published: December 2024

Abstract / Introduction Full Text(HTML) Figure(7) / Table(4) Related Papers Cited by

Abstract

Underwater wireless sensor networks (UWSNs) have the potential to provide environmental data for various applications, including studies related to environmental changes, early warning systems, and monitoring in industry. Continuous delivery of information in these contexts is paramount. UWSNs comprise the fundamental assets in these applications. However, the peculiar characteristics of underwater require sensor nodes to rely on their limited battery reserves. Consequently, energy management in these networks becomes a critical resource allocation problem within underwater sensor networks. To address this decision making problem, cluster-based network routing protocols have been extensively explored as a technology to minimize network energy consumption. Cluster heads (CHs) are employed to aggregate data and reduce overall energy usage, thus prolonging the network's lifespan. On the other hand, the focus on harvesting energy from ambient resources underwater has gained attention as a means to extend the operational life of sensor nodes in the distributed communication network system. This paper considers the stochastic energy harvesting process at each sensor node, specifically addressing the energy-aware routing problem in underwater acoustic sensor networks (UASNs). The contribution of this work lies in proposing a novel reinforcement learning-based algorithm for determining cluster heads (CHs), which involves not only considering the nodes' positions and residual energy but also accounting for the expected harvested energy. Numerical results validate that our introduced approach significantly decreases energy consumption and substantially extends the network's operational lifetime considerably.

Keywords:

Mathematics Subject Classification: Primary: 90B50, 68M12, 68M14, 68M18, 68M20; Secondary: 90B18, 68M10, 91B32.

Citation:

Full Text(HTML)

Figure 1. Sample 3D network architecture showing surface gateway and node positions

Download: Full-size image PowerPoint slide

Figure 2. Timeline showing round based data collection in clustered networks

Download: Full-size image PowerPoint slide

Figure 3. Simulated Vibrations Piezoelectric Harvested Power

Download: Full-size image PowerPoint slide

Figure 4. Number of alive nodes versus simulation rounds of different cluster head selection policy

Download: Full-size image PowerPoint slide

Figure 5. Lifetime comparison of different cluster head selection policy in simulation time

Download: Full-size image PowerPoint slide

Figure 6. Number of alive nodes per data items received

Download: Full-size image PowerPoint slide

Figure 7. Number of data signals received over time for k-means clustering with different cluster head selection policy

Download: Full-size image PowerPoint slide

Table 1. Comparison of Related Research Studies

Related Studies	Distributed	Energy Harvesting (EH)	Clustering	Machine Learning Methods	Stochastic EH
[25], [47]	✔	✘	✔	✔	✘
[38]	✔	✘	✔	✘	✘
[8], [9]	✔	✔	✘	✘	✘
[20], [14]	✔	✔	✘	✔	✔
[31], [30], [28], [11]	✔	✘	✔	✘	✘
[1]	✘	✘	✔	✘	✘
[6], [35], our previous work [10]	✔	✔	✔	✘	✔
Our Work	✔	✔	✔	✔	✔

| Show Table

DownLoad: CSV

Table 4. Simulation Parameters

Network Parameters
$ N_{node} $	Number of nodes	100
$ M $	Network Size ($ m^3 $)	$ 250\times250\times250 $
$ h $	Network Depth ($ mt $)	250
$ S_{(x,y,z)} $	Sink Location	(125,125, 0)
$ T_{round} $	Round Duration (seconds)	500
$ T_{frame} $	Frame Duration (seconds)	25

Communication Channel Parameters
R	Data rate (kbps)	2
$ E_0 $	Initial Energy (kJ)	1
$ P_{rx} $	Reception Power (W)	1.3
$ P_{idle} $	Idle Power (mW)	2.5
f	Frequency (kHz)	25
$ p_{ij}^{b} $	Bit error rate on link-(i, j)	$ 10^{-10} $
L	Data-Packet size in single frame (bits)	1024
$ \kappa $	Spreading Factor	1.5
$ B_N $	Noise Bandwidth (kHz)	1
$ E_{DA} $	Data Aggregation Energy (nJ/bit/packet)	5

Piezoelectric Harvester Parameters
$ L $	Cantilever length (mm)	50
$ B $	Cantilever width (mm)	3
$ \varepsilon_r $	Relative permittivity	4000
$ \varepsilon_0 $	Absolute permittivity ($ F/m $)	$ 8.85\times10^{-12} $
$ T_{pzt} $	Thickness ($ \mu m $)	60
$ d_{31}^p $	Piezoelectric constant ($ C/N $)	$ 300\times10^{-12} $
$ S_t $	Strouhal number	0.2
$ \lambda $	Mean Event Rate	5
$ v_f $	Fluid velocity (m/s)	0.65

| Show Table

DownLoad: CSV

Table 2. Energy State, $ State_{E_{ratio_n}} $

Energy State
0	$ E_{res}(t-1)> 0.9 $
1	$ E_{res}(t-1)> 0.5 $
2	$ E_{res}(t-1)> 0.2 $
3	$ E_{res}(t-1)> 0.1 $
4	$ E_{res}(t-1)< 0 $

| Show Table

DownLoad: CSV

Table 3. Harvested Energy State, $ State_{EH_n} $

Energy Harvesting State
0	$ Harv_{(t-1)}< 0.01 $
1	$ Harv_{(t-1)} \le 0.03 $
2	$ Harv_{(t-1)}> 0.03 $

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	G. Ahmed, J. Zou, X. Zhao and M. M. Sadiq Fareed, Markov chain model-based optimal cluster heads selection for wireless sensor networks, Sensors, 17 (2017), 440.
[2]	B. Altaf, S. S. Ali and G. W. Weber, Modeling the relationship between organizational performance and green supply chain practices using canonical correlation analysis, Wireless Networks, 26 (2020), 5835-5853.
[3]	I. F. Akyildiz, D. Pompili and T. Melodia, Underwater acoustic sensor networks: Research challenges, Ad Hoc Networks, 3 (2005), 257-279.
[4]	K. M. Awan, P. A. Shah, K. Iqbal, S. Gillani, W. Ahmad and Y. Nam, Underwater wireless sensor networks: A review of recent issues and challenges, Wireless Communications and Mobile Computing, (2019).
[5]	P. Bholowalia and A. Kumar, EBK-means: A clustering technique based on elbow method and k-means in WSN, International Journal of Computer Applications, 105 (2014), 17-24.
[6]	S. M. Bozorgi, A. S. Rostami, A. A. R. Hosseinabadi and V. E. Balas, A new clustering protocol for energy harvesting-wireless sensor networks, Computers and Electrical Engineering, 64 (2017), 233-247.
[7]	E. Dror, C. Avin and Z. Lotker, Fast randomized algorithm for 2-hops clustering in vehicular ad-hoc networks, Ad Hoc Networks, 11 (2013), 2002-2015.
[8]	H. E. Erdem and V. C. Gungor, Lifetime analysis of energy harvesting underwater wireless sensor nodes, 2017 25th Signal Processing and Communications Applications Conference (SIU), Antalya, Turkey, (2017), 1-4.
[9]	H. E. Erdem, H. U. Yildiz and V. C. Gungor, On the lifetime of compressive sensing based energy harvesting in underwater sensor networks, IEEE Sensors Journal, 19 (2019), 4680-4687.
[10]	C. Eris, O. M. Gul and P. S. Boluk, An energy-harvesting aware cluster head selection policy in underwater acoustic sensor networks, Proceedings of the 2023 International Balkan Conference on Communications and Networking (BalkanCom), Istanbul, Turkey, (2023), 1-5.
[11]	W. Fei, B. Hexiang, L. Deyu and W. Jianjun, Energy-efficient clustering algorithm in underwater sensor networks based on fuzzy C means and Moth-flame optimization method, IEEE Access, 8 (2020), 97474-97484.
[12]	M. Felemban and E. Felemban, Energy-delay tradeoffs for underwater acoustic sensor networks, Proceedings of the 1st International Black Sea Conference on Communications and Networking (BlackSeaCom), Nantes France, (2013), 45-49.
[13]	R. D. Gawade and S. L. Nalbalwar, A centralized energy efficient distance based routing protocol for wireless sensor networks, Journal of Sensors, (2016).
[14]	Y. Ge, Y. Nan and X. Guo, Maximizing network throughput by cooperative reinforcement learning in clustered solar-powered wireless sensor networks, International Journal of Distributed Sensor Networks, 17 (2021), 15501477211007411.
[15]	O. M. Gul and A. M. Erkmen, Energy-aware UAV-driven data collection with priority in robotic wireless sensor network, IEEE Sensors Journal, (2023), 17667-17675.
[16]	O. M. Gul and A. M. Erkmen, Energy-efficient cluster-based data collection by a UAV with a limited-capacity battery in robotic wireless sensor networks, Sensors, 20 (2020), 5865.
[17]	O. M. Gul, A. M. Erkmen and B. Kantarci, NTN-aided quality and energy-aware data collection in time-critical robotic wireless sensor networks, IEEE Internet of Things Magazine, 7 (2024).
[18]	O. M. Gul, A. M. Erkmen and B. Kantarci, UAV-Driven sustainable and Quality-Aware data collection in robotic wireless sensor networks, IEEE Internet of Things Journal, 9 (2022), 25150-25164.
[19]	B. Han, F. Ran, J. Li, L. Yan, H. Shen and A. Li, A novel adaptive cluster based routing protocol for energy-harvesting wireless sensor networks, Sensors, 22 (2022), 1564.
[20]	M. Han, J. Duan, S. Khairy and L. X. Cai, Enabling sustainable underwater IoT networks with energy harvesting: A decentralized reinforcement learning approach, IEEE Internet of Things Journal, 7 (2020), 9953-9964.
[21]	A. F. Harris and M. Zorzi, Modeling the underwater acoustic channel in ns2, Proceedings of the 2nd international conference on Performance evaluation methodologies and tools (ValueTools '07), Brussels, Belgium, (2007), 1-8.
[22]	W. Heinzelman, Application-Specific Protocol Architectures for Wireless Networks, Ph.D thesis, Massachusetts Institute of Technology in Massachusetts, 2000.
[23]	W. Heinzelman, A. Chandrakasan and H. Balakrishnan, Energy-efficient communication protocol for wireless microsensor networks, Proceedings of the 33rd Hawaii International Conference on System Sciences (HICSS), Maui, Hawaii, (2000), 10.
[24]	W. B. Heinzelman, A. P. Chandrakasan and H. Balakrishnan, An application-specific protocol architecture for wireless microsensor networks, IEEE Trans. Wirel. Commun., 1 (2002), 660-670.
[25]	D. W. HK, A. Kurniawan and M. S. Arifianto, Improving topology of LEACH cluster using reinforcement learning method, Proceedings of the 2019 IEEE International Conference on Sensors and Nanotechnology, Penang, Malaysia, (2019), 1-4.
[26]	E. Jarosz, W. J. Teague, J. W. Book and S. T. Besiktepe, On flow variability in the bosphorus strait, Journal of Geophysical Research, 116.
[27]	J. M. Jornet, M. Stojanovic and M. Zorzi, On joint frequency and power allocation in a cross-layer protocol for underwater acoustic networks, IEEE J. Ocean. Eng., 35 (2010), 936-947.
[28]	P. Li, S. Wang, H. Zhang and E. Zhang, Improved particle swarm optimization algorithm of clustering in underwater acoustic sensor networks, Proceedings of the OCEANS 2017-Aberdeen, Aberdeen, UK, (2017), 1-7.
[29]	M. Molins and M. Stojanovic, Slotted fama: A mac protocol for underwater acoustic networks, Proceedings of the OCEANS 2006 - Asia Pacific, Singapore, (2006), 1-7.
[30]	V. Narayan, A. K. Daniel and P. Chaturvedi, FGWOA: An efficient heuristic for cluster head selection in WSN using fuzzy based grey wolf optimization algorithm.,
[31]	K. G. Omeke, et al., DEKCS: A dynamic clustering protocol to prolong underwater sensor networks, IEEE Sensors Journal, 21 (2021), 9457-9464.
[32]	S. Pobering and N. Schwesinger, A novel hydropower harvesting device, International Conference on MEMS, NANO and Smart Systems (ICMENS04), Banff, AB, Canada, (2004), 480-485.
[33]	J. Qin, W. Fu, H. Gao and W. X. Zheng, Distributed k-means algorithm and fuzzy c-means algorithm for sensor networks based on multiagent consensus theory, IEEE Transactions on Cybernetics, 47 (2016), 772-783.
[34]	S. Redhu and R. M. Hegde, Multi-sensor data fusion for cluster-based data aggregation in IoT applications, Proceedings of the IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), Goa, India, (2019), 1-6.
[35]	Q. Ren and G. Yao, Enhancing harvested energy utilization for energy harvesting wireless sensor networks by an improved uneven clustering protocol, IEEE Access, 9 (2021), 119279-119288.
[36]	M. Rida, A. Makhoul, H. Harb, D. Laiymani and M. Barhamgi, EK-means: A new clustering approach for datasets classification in sensor networks, Ad Hoc Networks, 84 (2019), 158-169.
[37]	A. Stefanov and M. Stojanovic, Design and performance analysis of underwater acoustic networks, IEEE J. Sel. Areas Commun., 29 (2011), 2012-2021.
[38]	Y. Sun, M. Zheng, X. Han, S. Li and J. Yin, Adaptive clustering routing protocol for underwater sensor networks, Ad Hoc Networks, 136 (2022), 102953.
[39]	D. M. Toma, J. del Rio, M. Carbonell-Ventura and J. M. Masalles, Underwater energy harvesting system based on plucked-driven piezoelectrics, Proceedings of OCEANS 2015-Genova, Genova, Italy, (2015), 1-5.
[40]	K. Ugurlu, A new coherent multivariate average-value-at-risk, Optimization, 72 (2023), 493-519. doi: 10.1080/02331934.2021.1970755.
[41]	K. Ugurlu, Refinements of Kusuoka representations on $L^\infty$, Optimization, 71 (2022), 3351-3362. doi: 10.1080/02331934.2022.2038152.
[42]	K. Ugurlu and T. Brzeczek, Distorted probability operator for dynamic portfolio optimization in times of socio-economic crisis, Central European Journal of Operations Research, 31 (2023), 1043-1060. doi: 10.1007/s10100-022-00834-0.
[43]	R. Urick, Underwater Acoustic Systems, Halsted Pr, 1989.
[44]	R. Urick, Principles of Underwater Sound, 2$^{nd}$ edition, New York, NY (USA) McGraw-Hill Book, USA 1983.
[45]	O. Younis and S. Fahmy, HEED: A hybrid, energy-efficient, distributed clustering approach for ad hoc sensor networks, IEEE Transactions on Mobile Computing, 3 (2004), 366-379.
[46]	T. Zhao, M. Xu, X. Xiao, Y. Ma, Z. Li and Z. L. Wang, Recent progress in blue energy harvesting for powering distributed sensors in ocean, Nano Energy, 88 (2021), 106199.
[47]	J. Zhu, Y. Chen, X. Sun, J. Wu, Z. Liu and X. Xu, ECRKQ: Machine learning-based energy-efficient clustering and cooperative routing for mobile underwater acoustic sensor networks, IEEE Access, 9 (2021), 70843-70855.