An over complete deep learning method for inverse problems

Moshe Eliasof; Eldad Haber; Eran Treister

doi:10.3934/fods.2024036

Article Contents

2025, Volume 7, Issue 1: 50-71. Doi: 10.3934/fods.2024036

This issue Previous Article A system identification approach for non-intrusive reduced order modeling of radiation-induced photocurrents Next Article A practical existence theorem for reduced order models based on convolutional autoencoders

An over complete deep learning method for inverse problems

1.
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
2.
Department of EOAS, University of British Columbia, Vancouver, BC, Canada
3.
Department of Computer Science, Ben-Gurion University of the Negev, Beer Sheva, Israel

Received: February 2024

Revised: July 2024

Early access: August 14, 2024

Published online: August 14, 2024

This research was supported by the Israeli Council for Higher Education (CHE) via the Data Science Research Center at Ben-Gurion University of the Negev.

Abstract / Introduction Full Text(HTML) Figure(12) / Table(2) Related Papers Cited by

Abstract

Obtaining meaningful solutions for inverse problems has been a major challenge with many applications in science and engineering. Recent machine learning techniques based on proximal and diffusion-based methods have shown promising results. However, as we show in this work, they can also face challenges when applied to some exemplary problems. We show that, similar to previous works on over-complete dictionaries, it is possible to overcome these shortcomings by embedding the solution into higher dimensions. The novelty of the work proposed is that we jointly design and learn the embedding and the regularizer for the embedding vector. We demonstrate the merit of this approach on several exemplary and common inverse problems.

Keywords:

Mathematics Subject Classification: 68Q25, 68U05, 68T09.

Citation:

Full Text(HTML)

Figure 1. The duathlon problem, estimate the bike and run time from the total competition time.

Download: Full-size image PowerPoint slide

Figure 2. Experiments with the duathlon problem. The prior is made of three Gaussians and the data (yellow point) are presented on the left along with all possible points that fit the data (red line). Recovering a model given the data by sampling the posterior using diffusion is presented on the right (magenta points).

Download: Full-size image PowerPoint slide

Figure 3. A non-convex prior with 2 local minima in $ x $ is replaced with a learned quasi-convex prior in higher dimensions. Plots (c) and (d) are in log scale.

Download: Full-size image PowerPoint slide

Figure 4. The potential function $ \phi $ with a different number of layers.

Download: Full-size image PowerPoint slide

Figure 5. An example of the recovery of deblurred images from the STL-10 data set. (a) Ground truth (same for all rows) (b) Observed data, (c) Diffusion, (d) Proximal, (e) EUnet. Table 2 reports numerical recovery results. Additional examples are provided in Appendix A.

Download: Full-size image PowerPoint slide

Figure 6. Models and corresponding data (red line above) for the inverse magnetics experiment.

Download: Full-size image PowerPoint slide

Figure 7. Recovery of MNIST images from magnetic data using Diffusion, Proximal, and EUnet.

Download: Full-size image PowerPoint slide

Figure 8. An additional example of the recovery of deblurred images from the STL-10 data set. (a) Ground truth (same for all rows) (b) Observed data, (c) Diffusion, (d) Proximal, (e) EUnet. Table 2 reports numerical recovery results.

Download: Full-size image PowerPoint slide

Figure 9. Additional example of the recovery of deblurred images from the STL-10 data set. (a) Ground truth (same for all rows) (b) Observed data, (c) Diffusion, (d) Proximal, (e) EUnet. Table 2 reports numerical recovery results.

Download: Full-size image PowerPoint slide

Figure 10. An additional example of the recovery of deblurred images from the STL-10 data set. (a) Ground truth (same for all rows) (b) Observed data, (c) Diffusion, (d) Proximal, (e) EUnet. Table 2 reports numerical recovery results.

Download: Full-size image PowerPoint slide

Figure 11. Additional example of the recovery of deblurred images from the STL-10 data set. (a) Ground truth (same for all rows) (b) Observed data, (c) Diffusion, (d) Proximal, (e) EUnet. Table 2 reports numerical recovery results.

Download: Full-size image PowerPoint slide

Figure 12. Convergence history for different methods in the magnetics experiment.

Download: Full-size image PowerPoint slide

Table 1. Mean-Square-Error on validation data for the duathlon problem.

Method	Proximal	OPTEnet	EUnet
MSE	$ 3.5 \times 10^{-1} $	$ 8.2 \times 10^{-2} $	$ 4.1 \times 10^{-2} $

| Show Table

DownLoad: CSV

Table 2. Recovery loss (MSE) of the STL-10 test set, of different methods and different blurring kernel sizes $ s $.

Blurring Kernel size $ s $	Diffusion	Proximal	EUnet (Ours)
1	4.3e-3	5.6e-3	1.3e-3
3	4.3e-2	3.0e-2	2.3e-2
5	1.9e-1	4.7e-2	4.2e-2
7	5.0e-1	6.3e-2	5.7e-2
9	9.5e-1	8.9e-2	7.1e-2

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	J. Adler and O. Öktem, Solving ill-posed inverse problems using iterative deep neural networks, Inverse Problems, 33 (2017), 124007, 24 pp. doi: 10.1088/1361-6420/aa9581.
[2]	A. Agarwal, A. Anandkumar, P. Jain, P. Netrapalli and R. Tandon, Learning sparsely used overcomplete dictionaries, Conference on Learning Theory, (2014), 123-137.
[3]	M. Aharon, M. Elad and A. Bruckstein, K-svd: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on signal processing, 54 (2006), 4311-4322. doi: 10.1109/TSP.2006.881199.
[4]	M. Aharon, M. Elad and A. M. Bruckstein, On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them, Linear Algebra and its Applications, 416 (2006), 48-67. doi: 10.1016/j.laa.2005.06.035.
[5]	A. M. Bruckstein, D. L. Donoho and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), 34-81. doi: 10.1137/060657704.
[6]	E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete data, IEEE Transactions on Information Theory, 4 (2006). doi: 10.1109/TIT.2005.862083.
[7]	S. S. Chen, D. L. Donoho and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM Review, 43 (2001), 129-159. doi: 10.1137/S003614450037906X.
[8]	T. Q. Chen, Y. Rubanova, J. Bettencourt and D. K. Duvenaud, Neural ordinary differential equations, Advances in Neural Information Processing Systems, (2018), 6571-6583.
[9]	H. Chung, J. Kim, M. T. Mccann, M. L. Klasky and J. C. Ye, Diffusion posterior sampling for general noisy inverse problems, arXiv preprint, (2022), arXiv: 2209.14687.
[10]	H. Chung, B. Sim, D. Ryu and J. C. Ye, Improving diffusion models for inverse problems using manifold constraints, (2022).
[11]	H. Chung, B. Sim and J. C. Ye, Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2022), 12413-12422. doi: 10.1109/CVPR52688.2022.01209.
[12]	A. Coates, A. Ng and H. Lee, An analysis of single-layer networks in unsupervised feature learning, Proceedings of the 14th AISTATS, (2011), 215-223.
[13]	F.-A. Croitoru, V. Hondru, R. T. Ionescu and M. Shah, Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, (2023). doi: 10.1109/TPAMI.2023.3261988.
[14]	M. H. DeGroot, Optimal Statistical Decisions, John Wiley & Sons, 2005. doi: 10.1002/0471729000.
[15]	M. Eliasof, E. Haber and E. Treister, Drip: Deep regularizers for inverse problems, Inverse Problems, 40 (2024), 015006, 19 pp. doi: 10.1088/1361-6420/ad0f3b.
[16]	E. Haber and L. Tenorio, Learing regularization functionals a suprvised training approach, Inverse Problems, 19 (2003), 611-626. doi: 10.1088/0266-5611/19/3/309.
[17]	E. Haber, K. Lensink, E. Triester and L. Ruthotto, Imexnet: A forward stable deep neural network, arXiv preprint, (2019), arXiv: 1903.02639.
[18]	E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2018), 014004, 22 pp. doi: 10.1088/1361-6420/aa9a90.
[19]	J. M. Harrison and M. I. Taksar, Instantaneous control of brownian motion, Mathematics of Operations Research, 8 (1983), 439-453. doi: 10.1287/moor.8.3.439.
[20]	L. Horesh and E. Haber, A second order discretization of Maxwell's equations in the quasi-static regime on octree grids, SIAM J. Sci. Comput, 33 (2011), 2805-2822. doi: 10.1137/100798508.
[21]	H. Huang, E. Haber, L. Horesh and J. K. Seo, Optimal estimation of l1 regularization prior from a regularized empirical bayesian risk standpoint, Inverse Problems and Imaging, (2013). doi: 10.3934/ipi.2012.6.447.
[22]	K. H. Jin, M. T. McCann, E. Froustey and M. Unser, Deep convolutional neural network for inverse problems in imaging, IEEE Transactions on Image Processing, 26 (2017), 4509-4522. doi: 10.1109/TIP.2017.2713099.
[23]	J. Kaipio and E. Somersalo., Statistical and Computational Inverse Problems, Applied Mathematical Sciences, 160. Springer-Verlag, New York, 2005. doi: 10.1007/b138659.
[24]	S. Kasiviswanathan, H. Wang, A. Banerjee and P. Melville, Online l1-dictionary learning with application to novel document detection, Advances in Neural Information Processing Systems, 25 (2012).
[25]	Y. Lecun and C. Cortes, The MNIST database of handwritten digits.,
[26]	M. Mardani, Q. Sun, D. Donoho, V. Papyan, H. Monajemi, S. Vasanawala and J. Pauly, Neural proximal gradient descent for compressive imaging, Advances in Neural Information Processing Systems, 31 (2018).
[27]	J. Martin, L. C. Wilcox, C. Burstedde and O. Ghattas, A stochastic newton mcmc method for large-scale statistical inverse problems with application to seismic inversion, SIAM Journal on Scientific Computing, 34 (2012), A1460-A1487. doi: 10.1137/110845598.
[28]	A. Mittelholz and C. L. Johnson, The martian crustal magnetic field, Frontiers in Astronomy and Space Sciences, 9 (2022), 895362. doi: 10.3389/fspas.2022.895362.
[29]	S. Mukherjee, C.-B. Schönlieb and M. Burger, Learning convex regularizers satisfying the variational source condition for inverse problems, (2021).
[30]	J. Nagy and P. C. Hansen, Deblurring Images, SIAM, Philadelphia, 2006.
[31]	J. Nocedal and S. Wright, Numerical Optimization, Springer, New York, 1999. doi: 10.1007/b98874.
[32]	G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis and R. Willett, Deep learning techniques for inverse problems in imaging, IEEE Journal on Selected Areas in Information Theory, 1 (2020), 39-56. doi: 10.1109/JSAIT.2020.2991563.
[33]	N. Parikh and S. Boyd, et al., Proximal algorithmsm, Foundations and trends in Optimization, 1 (2014), 127-239. doi: 10.1561/2400000003.
[34]	R. L. Parker, Geophysical Inverse Theory, Princeton University Press, Princeton NJ, 1994.
[35]	R. W. Pastor, Techniques and applications of langevin dynamics simulations, The Molecular Dynamics of Liquid Crystals, (1994), 85-138. doi: 10.1007/978-94-011-1168-3_5.
[36]	L. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Proceedings of the Eleventh Annual International Conference of the Center for Nonlinear Studies on Experimental Mathematics: Computational Issues in Nonlinear Science, (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F.
[37]	Y. Song, L. Shen, L. Xing and S. Ermon, Solving inverse problems in medical imaging with score-based generative models, (2022).
[38]	G. Strang, Calculus, Volume 1. SIAM, 1991.
[39]	A. Tarantola, Inverse Problem Theory, Elsevier, Amsterdam, 1987.
[40]	L. Tenorio, F. Andersson, M. de Hoop and P. Ma, Data analysis tools for uncertainty quantification of inverse problems, Inverse Problems, 27 (2011), 045001, 22 pp. doi: 10.1088/0266-5611/27/4/045001.
[41]	C. R. Vogel, Computational Methods for Inverse Problems, Volume 23. SIAM, Philadelphia, 2002. doi: 10.1137/1.9780898717570.
[42]	E. Weinan, A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[43]	L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, Y. Shao, W. Zhang, B. Cui and M.-H. Yang, Diffusion models: A comprehensive survey of methods and applications, arXiv preprint, (2022), arXiv: 2209.00796.
[44]	T. Zhang, Z. Yao, A. Gholami, K. Keutzer, J. Gonzalez, G. Biros and M. Mahoney, Anodev2: A coupled neural ode evolution framework, arXiv preprint, (2019), arXiv: 1906.04596.