Deep reinforcement learning for the heat transfer control of pulsating impinging jets

Sajad Salavatidezfouli; Saeid Moradi Zadeh; Giovanni Stabile; Gianluigi Rozza

doi:10.3934/acse.2023016

Article Contents

2023, Volume 1, Issue 4: 401-423. Doi: 10.3934/acse.2023016

This issue Previous Article Review of multi-fidelity models Next Article Numerical solution of the $ L^1 $-optimal transport problem on surfaces

Deep reinforcement learning for the heat transfer control of pulsating impinging jets

1.
Mathematics Area, MathLab, International School for Advanced Studies (SISSA), Trieste, Italy
2.
Department of Mechanical Engineering, Islamic Azad University of Takestan, Takestan, Iran
3.
Department of Pure and Applied Sciences, Informatics and Mathematics Section, University of Urbino Carlo Bo, Urbino, Italy

^*Corresponding author: Gianluigi Rozza
^*Corresponding author: Gianluigi Rozza

Received: November 2023

Revised: December 2023

Early access: December 20, 2023

Published online: December 20, 2023

Abstract / Introduction Full Text(HTML) Figure(14) / Table(1) Related Papers Cited by

Abstract

This research study explored the applicability of deep reinforcement learning (DRL) for thermal control based on computational fluid dynamics. To accomplish that, the forced convection on a hot plate prone to a pulsating cooling jet with variable velocity has been investigated. We begin with evaluating the efficiency and viability of a vanilla deep Q-network (DQN) method for thermal control. Subsequently, a comprehensive comparison between different variants of DRL was conducted. Soft double and duel DQN achieved better thermal control performance among all the variants due to their efficient learning and action prioritization capabilities. Results demonstrated that the soft double DQN outperformed the hard double DQN. Moreover, soft double and duel can maintain the temperature in the desired threshold for more than 98% of the control cycle. These findings demonstrated the promising potential of DRL in effectively addressing thermal control systems.

Keywords:

Mathematics Subject Classification: Primary: 58F15, 58F17; Secondary: 53C35.

Citation:

FullText(HTML)

Figure 1. General overview of the DRL-CFD framework

Download: Full-size image PowerPoint slide

Figure 2. Formation of different regions in the context of the impinging jet on a flat plate [60]

Download: Full-size image PowerPoint slide

Figure 3. Schematic representation of the computational domain along with the dimension data of the jet and hot plate

Download: Full-size image PowerPoint slide

Figure 4. Representation of the structured mesh for the domain

Download: Full-size image PowerPoint slide

Figure 5. Probes location for measurement of states during DRL are shown with blue dots

Download: Full-size image PowerPoint slide

Figure 6. Comparison between total reward for different sensor layouts

Download: Full-size image PowerPoint slide

Figure 7. Temporal variation of dimensionless surface average temperature for off- and on-control (classic DQN) systems

Download: Full-size image PowerPoint slide

Figure 8. Temperature contour for off- and on-control (classic DQN) systems

Download: Full-size image PowerPoint slide

Figure 9. Time history of surface temperature and jet velocity for different episode numbers

Download: Full-size image PowerPoint slide

Figure 10. Temperature contour for different episode numbers

Download: Full-size image PowerPoint slide

Figure 11. Evolution of the total reward value for soft and hard double DQN in terms of the episode number

Download: Full-size image PowerPoint slide

Figure 12. Evolution of the total reward value for different variants of DQN in terms of the episode number

Download: Full-size image PowerPoint slide

Figure 13. Temperature contour for different variants of DQN

Download: Full-size image PowerPoint slide

Figure 14. Time history of surface temperature and jet velocity for on-control simulation of the DQN variants

Download: Full-size image PowerPoint slide

Table 1. Parameters used in the 3-D thermal control

d	$ V_\infty $	$ T_\infty $	$ T_d $	$ \rho $	$ \mu $	$ k $	$ C_p $
($ m $)	($ m/s $)	($ K $)	($ K $)	($ kg/m^3 $)	($ Pa.s $)	($ W/mK $)	($ J/kg.K $)
0.025	1	288	303	1.225	1.789e-5	0.024	1006

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. Al-Salaymeh, S. Taamneh and S. Nada, Characteristics of turbulent boundary layer flow along a horizontal heated flat plate, International Journal of Heat and Mass Transfer, 48 (2005), 5419-5430.
[2]	H. AlSofi, A. Larese, I. Padilla, R. Rossi, R. Scotta and R. W¨uchner, Finite elements and finite volumes methods in wind engineering applications, Advances in Computational Science and Engineering, 1(1), 2023, 82-122.
[3]	J. Anderson, G. Degrez, E. Dick and R. Grundmann, Computational Fluid Dynamics: An Introduction, Springer-Verlag, Berlin, 1992.
[4]	K. Arulkumaran, M. Deisenroth, M. Brundage and A. Bharath, A brief survey of deep reinforcement learning, IEEE Signal Processing Magazine, 34 (2017), 26-38, arXiv: 1708.05866. doi: 10.1109/MSP.2017.2743240.
[5]	Z. Bai, S. Brunton, B. Brunton, J. Kutz, E. Kaiser, A. Spohn and B. Noack, Data-driven methods in fluid dynamics: Sparse classification from experimental data, Whither Turbulence and Big Data in the 21st Century?, (2017), 323-342. doi: 10.1007/978-3-319-41217-7_17.
[6]	A. Bejan, Convection Heat Transfer, John Wiley & Sons, 2013. doi: 10.1002/9781118671627.
[7]	T. Bergman, F. Incropera, D. DeWitt and A. Lavine, Introduction to Heat Transfer, John Wiley & Sons, 2011.
[8]	M. Brenner, J. Eldredge and J. Kutz, A perspective on data-driven algorithms: Past, present, and future, Computers & Fluids, 188 (2019).
[9]	S. Brunton and J. Kutz, Machine learning for fluid mechanics, Annual Review of Fluid Mechanics, 52 (2020), 477-508.
[10]	R. Bunker, J. Dees and P. Palafox, Impingement cooling in gas turbines: Design, applications, and limitations, Impingement Jet Cooling in Gas Turbines, 25 (2014).
[11]	C.ÇCalı, A. Cevik, A. Karakurt and E. Mumcuoǧlu, Model predictive control based energy-efficient demand side management in smart grids, IET Cyber-Physical Systems: Theory & Applications, 3 (2019).
[12]	P. Childs, Heat Exchanger Design, CRC press, 1999.
[13]	M. Cracco, A. Mola, G. Noselli, M. Zanon and G. Rozza, On the thermal welding of paper-based polylaminate packages: Modelling, numerical implementation and sensitivity analysis, Advances in Computational Science and Engineering, 1 (2023), 218-248. doi: 10.3934/acse.2023010.
[14]	K. Davalath, M. Rahman and D. O'Neal, Forced convection heat transfer from a finned horizontal circular cylinder, International Journal of Heat and Mass Transfer, 30 (1987), 1113-1122.
[15]	B. Doll, D. Simon and N. Daw, The ubiquity of model-based reinforcement learning, Current Opinion in Neurobiology, 22 (2012), 1075-1081. doi: 10.1016/j.conb.2012.08.003.
[16]	D. Fan, L. Yang, Z. Wang, M. Triantafyllou and G. Karniadakis, Reinforcement learning for bluff body active flow control in experiments and simulations, Proceedings of the National Academy of Sciences, 117 (2020), 26091-26098. doi: 10.1073/pnas.2004939117.
[17]	C. Fricke, D. Wolff, M. Kemmerling and S. Elgeti, Investigation of reinforcement learning for shape optimization of 2D profile extrusion die geometries, Advances in Computational Science and Engineering, 1 (2023), 1-35. doi: 10.3934/acse.2023001.
[18]	S. Gao, H. Sun, H. Zhang and Y. Wu, DeepComfort: A deep learning approach for estimating indoor thermal comfort in buildings, Energy and Buildings, 216 (2020).
[19]	S. O. Giwa, M. Sharifpur, M. H. Ahmadi and J. P. Meyer, A review of magnetic field influence on natural convection heat transfer performance of nanofluids in square cavities, Journal of Thermal Analysis and Calorimetry, 145 (2021), 2581-2623. doi: 10.1007/s10973-020-09832-3.
[20]	S. Göttlich, N. Heuer, G. Stabile and G. Rozza, A reduced order model based ensemble Kalman filter for parameter identification in PDEs, preprint, 2023, arXiv: 2301.00047.
[21]	F. Guéniat, M. Bakar and N. Nair, Statistical modelling of aerodynamic loads on wind turbines using mixed-effects models, Wind Energy, 19 (2016).
[22]	T. Haarnoja, G. Tucker, J. Bayer, B. Peng, P. Abbeel and C. Finn, Composable deep reinforcement learning for robotic manipulation, Conference on Robot Learning, 2018.
[23]	H. Hasselt, A. Guez and D. Silver, Double Q-learning, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010.
[24]	M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar and D. Silver, Rainbow: Combining improvements in deep reinforcement learning, Proceedings of the AAAI Conference on Artificial Intelligence, 32 (2018), 11796. doi: 10.1609/aaai.v32i1.11796.
[25]	S. Hijazi, G. Stabile, A. Mola and G. Rozza, Data-Driven POD-Galerkin reduced order model for turbulent flows, J. Comput. Phys., 416 (2020), 109513, 30 pp. doi: 10.1016/j.jcp.2020.109513.
[26]	S. Hube, R. Pohlmann and S. Elgeti, Seamless integration of analysis and design: Automatic CAD reconstruction of post-analysis geometries, Advances in Computational Science and Engineering, 1 (2023), 162-179. doi: 10.3934/acse.2023007.
[27]	ANSYS Inc, ANSYS Fluent User's Guide, Version 2019 R3, 2019, Publisher: ANSYS, Inc. Canonsburgh, PA, USA.
[28]	A. Ivagnes, G. Stabile, A. Mola, T. Iliescu and G. Rozza, Residual-based variational multiscale turbulent flow simulations in the reduced order modeling framework, Computers & Fluids, 225 (2022), 105767.
[29]	E. Jang, S. Gu, R. Turner, J. Schulman and Y. Duan, Q-prop: Sample-efficient policy gradient with an off-policy critic, preprint, 2019, arXiv: 1611.02247.
[30]	M. Jena, P. Mishra and S. Sahoo, Comparative performance of jet and spray impingement cooling in steel strip run-out table: experimental results, Australian Journal of Mechanical Engineering, 2022. doi: 10.1080/14484846.2022.2066854.
[31]	J. Kaiser, M. Brechtel, A. Bockand and A. Meesmann, Model predictive control of a wind farm: A comparison between individual pitch control and coordinated control, Wind Energy, 22 (2019).
[32]	A. Keramati, J. Tang and J. Liu, Deep reinforcement learning-based control of microgrids: A comprehensive review, Renewable and Sustainable Energy Reviews, 154 (2022).
[33]	H. Kim, C. Lee and D. Kim, Enhancing convective heat transfer by generating longitudinal vortices using circular cylinders, International Journal of Heat and Mass Transfer, 189 (2022), 121-414.
[34]	D. Kingma and J. Ba, Adam: A method for stochastic optimization, preprint, 2017, arXiv: 1412.6980.
[35]	A. Kuprat, T. Johnson, B. Smith, R. Kurulugama and R. Kelly, Research in data-independent acquisition for mass spectrometry applications in proteomics: A summary, Journal of the American Society for Mass Spectrometry, 32 (2020), 2125-2132.
[36]	J. Li, M. Zhang, J. Martins and C. Shu, Efficient aerodynamic shape optimization with deep-learning-based geometric filtering, AIAA Journal, 58 (2020). doi: 10.2514/1.J059254.
[37]	J. Li and M. Zhang, Reinforcement-learning-based control of confined cylinder wakes with stability analyses, Journal of Fluid Mechanics, 932 (2022), Paper No. A44, 35 pp. doi: 10.1017/jfm.2021.1045.
[38]	Z. Li, J. Li, X. Jiang, J. Qiu and W. Li, Recent advances in indoor environmental quality control with intelligent technologies, Building and Environment, 204 (2022).
[39]	J. Ma and F. Wu, Feudal multi-agent deep reinforcement learning for traffic signal control, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Auckland, New Zealand, (2020), 816-824.
[40]	Y. Mei, C. Zheng, N. Aubry, M. LiW. Wu and X. Liu, Active control for enhancing vortex-induced vibration of a circular cylinder based on deep reinforcement learning, Physics of Fluids, 33 (2021), 103604. doi: 10.1063/5.0063988.
[41]	H. Miao, J. Zhang, Z. Yao, W. Liu, C. Li and W. Li, Spacecraft thermal control using an active vibration isolator, Journal of Thermal Analysis and Calorimetry, 146 (2021), 2327-2336.
[42]	V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra and M. Riedmiller, Playing atari with deep reinforcement learning, preprint, 2013, arXiv: 1312.5602.
[43]	V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, Al ex, M. Riedmiller, F. Fidjeland and G. Ostrovski, Human-level control through deep reinforcement learning, Nature, 518 (2015), 529-533. doi: 10.1038/nature14236.
[44]	S. Mondal, D. Ghosh and T. Das, A survey on thermal comfort prediction in smart buildings using deep learning and IoT, Sustainable Cities and Society, 63 (2020).
[45]	M. Novati, G. Iannello and M. Ortolano, Synchronised state regulation of two mutually connected dynamical systems, International Journal of Control, 89 (2016).
[46]	H. Oztop and E. Abu-Nada, Turbulence modeling of buoyant flows in confined enclosures heated from below, International Journal of Heat and Mass Transfer, 49 (2006), 3068-3082.
[47]	S. Pawar, S. San, A. Rasheed and P. Vedula, A priori analysis on deep learning of subgrid-scale parameterizations for Kraichnan turbulence, Theoretical and Computational Fluid Dynamics, 34 (2020), 429-455. doi: 10.1007/s00162-019-00512-z.
[48]	J. Peng, S. Chen, N. Aubry, Z. Chen and W. Wu, Time-variant prediction of flow over an airfoil using deep neural network, Physics of Fluids, 32 (2020), 123602. doi: 10.1063/5.0022222.
[49]	J. Peng, N. Aubry, S. Zhu, Z. Chen and W. Wu, Geometry and boundary condition adaptive data-driven model of fluid flow based on deep convolutional neural networks, Physics of Fluids, 33 (2021), 123602. doi: 10.1063/5.0073419.
[50]	A. Polydoros and L. Nalpantidis, Survey of model-based reinforcement learning: Applications on robotics, Journal of Intelligent & Robotic Systems, 86 (2017), 153-173. doi: 10.1007/s10846-017-0468-y.
[51]	J. Rabault, M. Kuchta, A. Jensen, U. Réglade and N. Cerardi, Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control, Journal of Fluid Mechanics, 865 (2019), 281-302. doi: 10.1017/jfm.2019.62.
[52]	C. Razzetta, V. Candiani, M. Crocco and F. Benvenuto, A hybrid time-frequency parametric modelling of medical ultrasound signal transmission, Advances in Computational Science and Engineering, 1 (2023), 249-270. doi: 10.3934/acse.2023011.
[53]	F. Ren, J. Rabault and H. Tang, Applying deep reinforcement learning to active flow control in weakly turbulent conditions, Physics of Fluids, 33 (2021), 037121. doi: 10.1063/5.0037371.
[54]	L. Ren, X. Jin, H. Xu, D. Wang and Y. Liu, Active control of broadband noise using a smart distributed acoustic sensor network, Journal of Sound and Vibration, 482 (2020).
[55]	G. Rozza, G. Stabile and F. Ballarin, Advanced Reduced Order Methods and Applications in Computational Fluid Dynamics, Comput. Sci. Eng., 27, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2023.
[56]	O. San, R. Maulik and M. Ahmed, An artificial neural network framework for reduced order modeling of transient flows, Communications in Nonlinear Science and Numerical Simulation, 77 (2019), 271-287. doi: 10.1016/j.cnsns.2019.04.025.
[57]	M. Sewak, A. Sharma, A. Krishna, Alok and P. Kumar, Deep reinforcement learning for autonomous UAV navigation and obstacle avoidance, Journal of Intelligent & Robotic Systems, 94 (2019).
[58]	M. Shahabadi, S. Mehryan, M. Ghalambaz and M. Ismael, Controlling the natural convection of a non-Newtonian fluid using a flexible fin, Applied Mathematical Modelling, 92 (2021), 669-686. doi: 10.1016/j.apm.2020.11.029.
[59]	S. Shimomura, S. Sekimoto, A. Oyama, K. Fujii and H. Nishida, Experimental study on the application of distributed deep reinforcement learning to closed-loop flow separation control over an airfoil, AIAA, 2020. doi: 10.2514/6.2020-0579.
[60]	K. Sodjavi, B. Montagné, P. Bragança, A. Meslem, F. Bode and M. Kristiawan, Impinging cross-shaped submerged jet on a flat plate: a comparison of plane and hemispherical orifice nozzles, Meccanica, 50 (2015), 2927-2947. doi: 10.1007/s11012-015-0181-5.
[61]	B. Swartz, L. Wu, Q. Zhou and Q. Hao, Machine learning predictions of critical heat fluxes for pillar-modified surfaces, International Journal of Heat and Mass Transfer, 180 (2021), 121744. doi: 10.1016/j.ijheatmasstransfer.2021.121744.
[62]	K. Taira, S. Brunton and J. Kutz, Modal analysis of fluid flows: An overview, AIAA Journal, 55 (2017), 1-17. doi: 10.2514/1.J056060.
[63]	K. Taira, M. Hemati, S. Brunton, Y. Sun, K. Duraisamy, S. Bagheri, S. Dawson and C. Yeh, Modal analysis of fluid flows: Applications and outlook, AIAA Journal, 58 (2020), 998-1022. doi: 10.2514/1.J058462.
[64]	P. Tan, T. Xu and F. Gao, General coordinated active thermal control for parallel-connected inverters with switching frequency control, 2021 IEEE 1st International Power Electronics and Application Symposium (PEAS), 2021, 1-6. doi: 10.1109/PEAS53589.2021.9628578.
[65]	H. Tang, J. Rabault, A. Kuhnle, Y. Wang and T. Wang, Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning, Physics of Fluids, 32 (2020), 053605. doi: 10.1063/5.0006492.
[66]	M. Tavakoli, Y. Shoukry, T. Nguyen and S. Krishnan, Action-decision networks for visual tracking with deep reinforcement learning, preprint, 2018, arXiv: 1806.04856.
[67]	Y. Tian, X. Zhou and J. Yang, Networked H-infinity control for systems with a finite number of quantizers: A survey, International Journal of Robust and Nonlinear Control, 31 (2021).
[68]	H. Van Hasselt, A. Guez and D. Silver, Deep reinforcement learning with double Q-learning, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 30 (2016). doi: 10.1609/aaai.v30i1.10295.
[69]	S. Verma, R. Kumar and A. Mukherjee, Efficient fault diagnosis of HVAC systems using vibration analysis, Applied Acoustics, 137 (2018).
[70]	K. Vignon, M. Ferrando, S. Casale Brunet, M. Pepe and G. Bolzon, Effective thermal resistance of textile materials: An innovative in situ measurement method for thermal protective clothing, Measurement, 193 (2023).
[71]	A. Vu, S. Gulati, P. Vogel, T. Grunwald and T. Bergs, Machine learning-based predictive modeling of contact heat transfer, International Journal of Heat and Mass Transfer, 174 (2021), 121300. doi: 10.1016/j.ijheatmasstransfer.2021.121300.
[72]	J. Wan, M. Huang, R. Huang, W. Jiang, Z. Xu and W. Deng, Robust reinforcement learning control of an air conditioning system in a smart building, Energy and Buildings, 209 (2020).
[73]	S. Wang, S. Gao, D. Xie, Y. Liu, X. Shi and X. Huang, Performance analysis and optimization of a novel exhaust gas-driven R744 heat pump system for vehicle air conditioning, Energy Conversion and Management, 263 (2023).
[74]	X. Wang, W. Zhang, Y. Liu, J. Li, J. Qiu and W. Li, Reinforcement learning-based adaptive control for comfort and energy management in buildings, Energy and Buildings, 227, 2020.
[75]	F. Wang, F. Li, J. Wang, J. Cao and Y. Liu, Control of power system voltage stability based on adaptive dynamic programming, IEEE Transactions on Power Systems, 2024.
[76]	Y. Wang, Y. Mei, N. Aubry, Z. Chen, P. Wu and W. Wu, Deep reinforcement learning based synthetic jet control on disturbed flow over airfoil, Physics of Fluids, 34 (2022), 033606. doi: 10.1063/5.0080922.
[77]	Y. Wang, B. Wang, Z. Wang, Y. Chen and W. Wang, Closed-loop feedback control for the stabilization of a foiling sailing yacht with an adjustable T-foil, Control Engineering Practice, 113 (2023).
[78]	Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot and N. de Freitas, Dueling network architectures for deep reinforcement learning, Proceedings of the 33rd International Conference on International Conference on Machine Learning, 2016.
[79]	J. Wang, W. Guo, K. Xiong and S. Wang, Review of aerospace-oriented spray cooling technology, Progress in Aerospace Sciences, 116 (2020), 100635. doi: 10.1016/j.paerosci.2020.100635.
[80]	E. Wei, D. Wicke, D. Freelan and S. Luke, Multiagent soft q-learning, preprint, 2018, arXiv: 1804.09817.
[81]	Y. Wu, L. Fang and Y. Xu, Predicting interfacial thermal resistance by machine learning, npj Computational Materials, 5 (2019), Article number: 56. doi: 10.1038/s41524-019-0193-0.
[82]	P. Xiong, A. Hamid, K. Iqbal, M. Irfan and M. Khan, Numerical simulation of mixed convection flow and heat transfer in the lid-driven triangular cavity with different obstacle configurations, International Communications in Heat and Mass Transfer, 123 (2021), 105-202.
[83]	M. Xiong, C. Zhang, P. Tang, J. Hao and X. Gao, Application of a convolutional neural network in real-time indoor environment perception for mobile robot navigation, Applied Sciences, 12 (2022).
[84]	D. Xu, J. Zhao and L. Liu, Near-field radiation assisted smart skin for spacecraft thermal control, International Journal of Thermal Sciences, 165 (2021), 106934. doi: 10.1016/j.ijthermalsci.2021.106934.
[85]	H. Xu, W. Zhang, J. Deng and J. Rabault, Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning, Journal of Hydrodynamics, 32 (2020), 254-258. doi: 10.1007/s42241-020-0027-z.
[86]	D. Zhang, Y. Zhang, Y. Lu and S. Chen, Switched model predictive control of a HVAC system with a variable-air-volume terminal unit for office buildings, Applied Energy, 250 (2019), 821-834.
[87]	S. Zou, Y. Xia and Z. Wang, Towards event-triggered networked control systems with wireless communication constraints, Automatica, 117 (2020).