Article Contents
Article Contents

# Learning nonlocal regularization operators

• * Corresponding author: Gernot Holler
• A learning approach for determining which operator from a class of nonlocal operators is optimal for the regularization of an inverse problem is investigated. The considered class of nonlocal operators is motivated by the use of squared fractional order Sobolev seminorms as regularization operators. First fundamental results from the theory of regularization with local operators are extended to the nonlocal case. Then a framework based on a bilevel optimization strategy is developed which allows to choose nonlocal regularization operators from a given class which i) are optimal with respect to a suitable performance measure on a training set, and ii) enjoy particularly favorable properties. Results from numerical experiments are also provided.

Mathematics Subject Classification: Primary: 49J20, 49N45, 35R11; Secondary 35J05.

 Citation:

• Figure 1.  Optimal weights for linear state equation in case (A) (first row) and case (B) (second row) using $\beta = 0$. The training set consisted of 512 data vectors given in a single batch

Figure 2.  Ground truth and reconstructed controls for one data vector from the validation set for the linear state equation in case (B) using $\beta = 0$. The training set consisted of 512 data vectors given in a single batch

Figure 3.  Optimal weights for $s = 0.1$ and various $L^1$ regularization parameters $\beta$ in case (B). The training and validation set both consisted of 512 data vectors given in a single batch. The average validation errors were $1.42 \times {10^{ - 2}}$ ($\beta = 10^{-2}$), $1.18\times {10^{ - 2}}$ ($\beta = 10^{-3}$), and $1.03\times {10^{ - 2}}$ ($\beta = 10^{-4}$)

Table 1.  Average training and validation error for optimal regularization parameter $\nu^*$ (second and third column) and optimal weight $\sigma^*$ (fourth and fifth column) in case (A) using $\beta = 0$. The training and validation set both consisted of 512 data vectors

 (a) s = 0:1 batchsize train error reg val error reg train error weight val error weight 8 1.81 × 10−2 2.19 × 10−2 1.42 × 10−2 2.04 × 10−2 64 1.88 × 10−2 2.11 × 10−2 1.61 × 10−2 1.86 × 10−2 512 1.90 × 10−2 2.09 × 10−2 1.65 × 10−2 1.82 × 10−2 (b) s = 0:9 8 1.53 × 10−2 1.98 × 10−2 1.51 × 10−2 1.97 × 10−2 64 1.63 × 10−2 1.85 × 10−2 1.61 × 10−2 1.84 × 10−2 512 1.65 × 10−2 1.82 × 10−2 1.64 × 10−2 1.80 × 10−2 (c) L2 and H1 regularization with optimal ν* batchsize train error L2 val error L2 train error H1 val error H1 8 1.93 × 10−2 2.30 × 10−2 1.51 × 10−2 1.97 × 10−2 64 1.99 × 10−2 2.22 × 10−2 1.61 × 10−2 1.84 × 10−2 512 2.01 × 10−2 2.20 × 10−2 1.64 × 10−2 1.80 × 10−2

Table 2.  Average training and validation error for optimal regularization parameter $\nu^*$ (second and third column) and optimal weight $\sigma^*$ (fourth and fifth column) in case (B) using $\beta = 0$. The training and validation set both consisted of 512 data vectors

 (a) s = 0.1 batchsize train error reg val error reg train error weight val error weight 8 2.03 × 10−2 2.12 × 10−2 1.02 × 10−2 1.09 × 10−2 64 2.05 × 10−2 2.09 × 10−2 1.02 × 10−2 1.04 × 10−2 512 2.06 × 10−2 2.08 × 10−2 1.14 × 10−2 1.15 × 10−2 (b) s = 0.9 8 1.37 × 10−2 1.44 × 10−2 1.33 × 10−2 1.41 × 10−2 64 1.38 × 10−2 1.42 × 10−2 1.35 × 10−2 1.38 × 10−2 512 1.39 × 10−2 1.41 × 10−2 1.35 × 10−2 1.38 × 10−2 (c) L2 and H1 regularization with optimal ν* 8 2.48 × 10−2 2.60 × 10−2 1.33 × 10−2 1.40 × 10−2 64 2.51 × 10−2 2.56 × 10−2 1.35 × 10−2 1.38 × 10−2 512 2.52 × 10−2 2.55 × 10−2 1.35 × 10−2 1.38 × 10−2
•  [1] R. A. Adams and J. J. F. Fournier, Sobolev Spaces, Pure and Applied Mathematics (Amsterdam), 140, Elsevier/Academic Press, Amsterdam, 2003. [2] B. Alali, K. Liu and M. Gunzburger, A generalized nonlocal vector calculus, Z. Angew. Math. Phys., 66 (2015), 2807-2828.  doi: 10.1007/s00033-015-0514-1. [3] H. Antil, E. Otárola and A. J. Salgado, Optimization with respect to order in a fractional diffusion model: Analysis, approximation and algorithmic aspects, J. Sci. Comput., 77 (2018), 204-224.  doi: 10.1007/s10915-018-0703-0. [4] H. Antil and C. N. Rautenberg, Sobolev spaces with non-Muckenhoupt weights, fractional elliptic operators, and applications, SIAM J. Math. Anal., 51 (2019), 2479-2503. doi: 10.1137/18M1224970. [5] K. Bredies and D. Lorenz, Mathematische Bildverarbeitung, Vieweg+Teubner Verlag, 2011. doi: 10.1007/978-3-8348-9814-2. [6] H. Brezis, Functional Analysis, Sobolev Spaces and Partial Differential Equations, Universitext, Springer, New York, 2011. doi: 10.1007/978-0-387-70914-7. [7] H. Cartan, Differential Calculus, Hermann, Paris; Houghton Mifflin Co., Boston, Mass., 1971. [8] E. Casas, M. Mateos and A. Rösch, Analysis of control problems of nonmontone semilinear elliptic equations, ESAIM: COCV, 26 (2020). doi: 10.1051/cocv/2020032. [9] E. Casas and F. Tröltzsch, Second order optimality conditions and their role in PDE control, Jahresber. Dtsch. Math.-Ver., 117 (2015), 3-44.  doi: 10.1365/s13291-014-0109-3. [10] J. Chung, M. Chung and D. P. O'Leary, Designing optimal spectral filters for inverse problems, SIAM J. Sci. Comput., 33 (2011), 3132-3152.  doi: 10.1137/100812938. [11] J. Chung and M. I. Español, Learning regularization parameters for general-form Tikhonov, Inverse Problems, 33 (2017), 21pp. doi: 10.1088/1361-6420/33/7/074004. [12] M. D'Elia, J. C. De Los Reyes and A. Miniguano Trujillo, Bilevel parameter optimization for nonlocal image denoising models, preprint, arXiv: 1912.02347v3. doi: 10.2172/1592945. [13] M. D'Elia and M. Gunzburger, The fractional Laplacian operator on bounded domains as a special case of the nonlocal diffusion operator, Comput. Math. Appl., 66 (2013), 1245-1260.  doi: 10.1016/j.camwa.2013.07.022. [14] J. C. De Los Reyes, C.-B. Schönlieb and T. Valkonen, The structure of optimal parameters for image restoration problems, J. Math. Anal. Appl., 434 (2016), 464-500.  doi: 10.1016/j.jmaa.2015.09.023. [15] J. C. De Los Reyes and C.-B. Schönlieb, Image denoising: Learning the noise model via nonsmooth PDE-constrained optimization, Inverse Probl. Imaging, 7 (2013), 1183-1214.  doi: 10.3934/ipi.2013.7.1183. [16] S. Dempe, Foundations of Bilevel Programming, Nonconvex Optimization and its Applications, 61, Kluwer Academic Publishers, Dordrecht, 2002. doi: 10.1007/b101970. [17] E. Di Nezza, G. Palatucci and E. Valdinoci, Hitchhiker's guide to the fractional Sobolev spaces, Bull. Sci. Math., 136 (2012), 521-573.  doi: 10.1016/j.bulsci.2011.12.004. [18] Q. Du, M. Gunzburger, R. B. Lehoucq and K. Zhou, Analysis and approximation of nonlocal diffusion problems with volume constraints, SIAM Rev., 54 (2012), 667-696. doi: 10.1137/110833294. [19] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, Studies in Advanced Mathematics, CRC Press, Boca Raton, FL, 1992. [20] G. B. Folland, Real Analysis. Modern Techniques and Their Applications, Pure and Applied Mathematics, John Wiley & Sons, Inc., New York, 1999. [21] P. Grisvard, Elliptic Problems in Nonsmooth Domains, Monographs and Studies in Mathematics, 24, Pitman (Advanced Publishing Program), Boston, MA, 1985. [22] M. Gunzburger and R. B. Lehoucq, A nonlocal vector calculus with application to nonlocal boundary value problems, Multiscale Model. Simul., 8 (2010), 1581-1598.  doi: 10.1137/090766607. [23] E. Haber and L. Tenorio, Learning regularization functionals-A supervised training approach, Inverse Problems, 19 (2003), 611-626.  doi: 10.1088/0266-5611/19/3/309. [24] M. Hintermüller and C. N. Rautenberg, Optimal selection of the regularization function in a weighted total variation model. Part I: Modelling and theory, J. Math. Imaging Vision, 59 (2017), 498-514.  doi: 10.1007/s10851-017-0744-2. [25] G. Holler, K. Kunisch and R. C. Barnard, A bilevel approach for parameter learning in inverse problems, Inverse Problems, 34 (2018), 28pp. doi: 10.1088/1361-6420/aade77. [26] K. Ito and K. Kunisch, Lagrange Multiplier Approach to Variational Problems and Applications, Advances in Design and Control, 15, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2008. doi: 10.1137/1.9780898718614. [27] K. Ito and K. Kunisch, The primal-dual active set method for nonlinear optimal control problems with bilateral constraints, SIAM J. Control Optim., 43 (2004), 357-376. doi: 10.1137/S0363012902411015. [28] K. Kunisch and T. Pock, A bilevel optimization approach for parameter learning in variational models, SIAM J. Imaging Sci., 6 (2013), 938-983.  doi: 10.1137/120882706. [29] P. Ochs, R. Ranftl, T. Brox and T. Pock, Bilevel optimization with nonsmooth lower level problems, in Scale Space and Variational Methods in Computer Vision, Lecture Notes in Comput. Sci., 9087, Springer, Cham, 2015,654-665. doi: 10.1007/978-3-319-18461-6_52. [30] M. Ulbrich and S. Ulbrich, Nichtlineare Optimierung, Mathematik Kompakt, Birkhäuser, Basel, 2012. doi: 10.1007/978-3-0346-0654-7. [31] J. Zowe and S. Kurcyusz, Regularity and stability for the mathematical programming problem in Banach spaces, Appl. Math. Optim., 5 (1979), 49-62.  doi: 10.1007/BF01442543.

Figures(3)

Tables(2)