December 2017, 11(6): 1047-1070. doi: 10.3934/ipi.2017048

Accelerated bregman operator splitting with backtracking

1. 

Department of Mathematics, University of Florida, Gainesville, FL 32611, USA

2. 

Department of Mathematical Sciences, Clemson University, Clemson, SC 29634, USA

3. 

Munitions Directorate, Air Force Research Laboratory, Eglin AFB, FL 32542-9998, USA

* Corresponding author: Xianqi Li

Received  November 2016 Revised  June 2017 Published  September 2017

Fund Project: The first author is supported by NSF grant DMS-1319050, NSF/DMS 1719932 and AFOSR/Eglin (FA8651-08-D-0108). The second author was partially supported by UFII fellowship and AFRL Mathematical Modeling and Optimization Institute.

This paper develops two accelerated Bregman Operator Splitting (BOS) algorithms with backtracking for solving regularized large-scale linear inverse problems, where the regularization term may not be smooth. The first algorithm improves the rate of convergence for BOSVS [5] in terms of the smooth component in the objective function by incorporating Nesterov's multi-step acceleration scheme under the assumption that the feasible set is bounded. The second algorithm is capable of dealing with the case where the feasible set is unbounded. Moreover, it allows more aggressive stepsize than that in the first scheme by properly selecting the penalty parameter and jointly updating the acceleration parameter and stepsize. Both algorithms exhibit better practical performance than BOSVS and AADMM [21], while preserve the same accelerated rate of convergence as that for AADMM. The numerical results on total-variation based image reconstruction problems indicate the effectiveness of the proposed algorithms.

Citation: Yunmei Chen, Xianqi Li, Yuyuan Ouyang, Eduardo Pasiliao. Accelerated bregman operator splitting with backtracking. Inverse Problems & Imaging, 2017, 11 (6) : 1047-1070. doi: 10.3934/ipi.2017048
References:
[1]

A. Beck and M. Teboulle, Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems, IEEE Transactions on Image Processing, 18 (2009), 2419-2434. doi: 10.1109/TIP.2009.2028250.

[2]

K. BlockM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint, Magnetic Resonance in Medicine, 57 (2007), 1086-1098. doi: 10.1002/mrm.21236.

[3]

S. BoydN. ParikhE. ChuB. Peleato and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends® in Machine Learning, 3 (2011), 1-122. doi: 10.1561/2200000016.

[4]

Y. ChenW. HagerF. HuangD. PhanX. Ye and W. Yin, Fast algorithms for image reconstruction with application to partially parallel MR imaging, SIAM Journal on Imaging Sciences, 5 (2012), 90-118. doi: 10.1137/100792688.

[5]

Y. ChenW. HagerM. Yashtini and X. Ye, Bregman operator splitting with variable stepsize for total variation image reconstruction, Computational Optimization and Applications, 54 (2013), 317-342. doi: 10.1007/s10589-012-9519-2.

[6]

W. Deng and W. Yin, On the global and linear convergence of the generalized alternating direction method of multipliers, Journal of Scientific Computing, 66 (2012), 1-28. doi: 10.1007/s10915-015-0048-x.

[7]

D. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006), 1289-1306. doi: 10.1109/TIT.2006.871582.

[8]

D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation, Computers & Mathematics with Applications, 2 (1976), 17-40. doi: 10.1016/0898-1221(76)90003-1.

[9]

R. Glowinski and A. Marroco, Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problémes de Dirichlet non linéaires, Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique, 9 (1975), 41-76.

[10]

D. GoldfarbS. Ma and K. Scheinberg, Fast multiple-splitting algorithms for convex optimization, SIAM Journal on Optimization, 22 (2012), 533-556. doi: 10.1137/090780705.

[11]

T. GoldsteinB. O'DonoghueS. Setzer and R. Baraniuk, Fast alternating direction optimization methods, SIAM Journal on Imaging Sciences, 7 (2014), 1588-1623. doi: 10.1137/120896219.

[12]

T. Goldstein and S. Osher, The split Bregman method for L1-regularized problems, SIAM Journal on Imaging Sciences, 2 (2009), 323-343. doi: 10.1137/080725891.

[13]

W. HagerC. NgoM. Yashtini and H. Zhang, An alternating direction approximate Newton algorithm for ill-conditioned inverse problems with application to parallel MRI, Name of the Journal, 3 (2015), 139-162. doi: 10.1007/s40305-015-0078-y.

[14]

W. HagerM. Yashtini and H. Zhang, An $\backslash$mathcalO(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm, SIAM Journal on Numerical Analysis, 54 (2016), 1535-1556. doi: 10.1137/15100401X.

[15]

M. Hong and Z. Luo, On the linear convergence of the alternating direction method of multipliers, Mathematical Programming, 162 (2017), 165-199, arXiv:1208.3922. doi: 10.1007/s10107-016-1034-2.

[16]

C. LiW. YinH. Jiang and Y. Zhang, An efficient augmented Lagrangian method with applications to total variation minimization, Computational Optimization and Applications, 56 (2013), 507-530. doi: 10.1007/s10589-013-9576-1.

[17]

Q. Liu, J. Luo, S. Wang M. Xiao and M. Ye, An augmented Lagrangian multi-scale dictionary learning algorithm EURASIP Journal on Advances in Signal Processing, 2011 (2011), p58. doi: 10.1186/1687-6180-2011-58.

[18]

M. LustigD. Donoho and J. Pauly, Sparse MRI: The application of compressed sensing for rapid MR imaging, Magnetic Resonance in Medicine, 58 (2007), 1182-1195. doi: 10.1002/mrm.21391.

[19]

R. Monteiro and B. Svaiter, Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers, SIAM Journal on Optimization, 23 (2013), 475-507. doi: 10.1137/110849468.

[20]

Y. Nesterov, Smooth minimization of non-smooth functions, Mathematical Programming, 103 (2005), 127-152. doi: 10.1007/s10107-004-0552-5.

[21]

Y. OuyangY. ChenG. Lan and E. Pasiliao, An accelerated linearized alternating direction method of multipliers, SIAM Journal on Imaging Sciences, 8 (2015), 644-681. doi: 10.1137/14095697X.

[22]

K. PruessmannM. WeigerP. Börnert and P. Boesiger, Advances in sensitivity encoding with arbitrary k-space trajectories, Magnetic Resonance in Medicine, 46 (2001), 638-651.

[23]

K. ScheinbergD. Goldfarb and X. Bai, Fast first-order methods for composite convex optimization with backtracking, Foundations of Computational Mathematics, 14 (2014), 389-417. doi: 10.1007/s10208-014-9189-9.

[24]

L. Shepp and B. Logan, The Fourier reconstruction of a head section, IEEE Transactions on Nuclear Science, 21 (1974), 21-43. doi: 10.1109/TNS.1974.6499235.

[25]

Y. WangJ. YangW. Yin and Y. Zhang A new alternating minimization algorithm for total variation image reconstruction, A new alternating minimization algorithm for total variation image reconstruction, SIAM Journal on Imaging Sciences, 1 (2008), 248-272. doi: 10.1137/080724265.

[26]

B. Wohlberg, Efficient Algorithms for Convolutional Sparse Representations, IEEE Transactions on Image Processing, 25 (2016), 301-315. doi: 10.1109/TIP.2015.2495260.

[27]

J. YangY. Zhang and W. Yin, A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data, IEEE Journal of Selected Topics in Signal Processing, 4 (2010), 288-297.

[28]

X. YeY. Chen and F. Huang, Computational acceleration for MR image reconstruction in partially parallel imaging, IEEE Transactions on Medical Imaging, 30 (2011), 1055-1063.

[29]

X. ZhangM. BurgerX. Bresson and S. Osher, Bregmanized nonlocal regularization for deconvolution and sparse reconstruction, SIAM Journal on Imaging Sciences, 3 (2010), 253-276. doi: 10.1137/090746379.

[30]

X. ZhangM. Burger and S. Osher, A unified primal-dual algorithm framework based on Bregman iteration, Journal of Scientific Computing, 46 (2011), 20-46. doi: 10.1007/s10915-010-9408-8.

[31]

M. Zhu and T. Chan, An efficient primal-dual hybrid gradient algorithm for total variation image restoration, UCLA CAM Report, (2008), 8-34.

[32]

M. ZhuS. Wright and T. Chan, Duality-based algorithms for total-variation-regularized image restoration, Computational Optimization and Applications, 47 (2010), 377-400. doi: 10.1007/s10589-008-9225-2.

show all references

References:
[1]

A. Beck and M. Teboulle, Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems, IEEE Transactions on Image Processing, 18 (2009), 2419-2434. doi: 10.1109/TIP.2009.2028250.

[2]

K. BlockM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint, Magnetic Resonance in Medicine, 57 (2007), 1086-1098. doi: 10.1002/mrm.21236.

[3]

S. BoydN. ParikhE. ChuB. Peleato and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends® in Machine Learning, 3 (2011), 1-122. doi: 10.1561/2200000016.

[4]

Y. ChenW. HagerF. HuangD. PhanX. Ye and W. Yin, Fast algorithms for image reconstruction with application to partially parallel MR imaging, SIAM Journal on Imaging Sciences, 5 (2012), 90-118. doi: 10.1137/100792688.

[5]

Y. ChenW. HagerM. Yashtini and X. Ye, Bregman operator splitting with variable stepsize for total variation image reconstruction, Computational Optimization and Applications, 54 (2013), 317-342. doi: 10.1007/s10589-012-9519-2.

[6]

W. Deng and W. Yin, On the global and linear convergence of the generalized alternating direction method of multipliers, Journal of Scientific Computing, 66 (2012), 1-28. doi: 10.1007/s10915-015-0048-x.

[7]

D. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006), 1289-1306. doi: 10.1109/TIT.2006.871582.

[8]

D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation, Computers & Mathematics with Applications, 2 (1976), 17-40. doi: 10.1016/0898-1221(76)90003-1.

[9]

R. Glowinski and A. Marroco, Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problémes de Dirichlet non linéaires, Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique, 9 (1975), 41-76.

[10]

D. GoldfarbS. Ma and K. Scheinberg, Fast multiple-splitting algorithms for convex optimization, SIAM Journal on Optimization, 22 (2012), 533-556. doi: 10.1137/090780705.

[11]

T. GoldsteinB. O'DonoghueS. Setzer and R. Baraniuk, Fast alternating direction optimization methods, SIAM Journal on Imaging Sciences, 7 (2014), 1588-1623. doi: 10.1137/120896219.

[12]

T. Goldstein and S. Osher, The split Bregman method for L1-regularized problems, SIAM Journal on Imaging Sciences, 2 (2009), 323-343. doi: 10.1137/080725891.

[13]

W. HagerC. NgoM. Yashtini and H. Zhang, An alternating direction approximate Newton algorithm for ill-conditioned inverse problems with application to parallel MRI, Name of the Journal, 3 (2015), 139-162. doi: 10.1007/s40305-015-0078-y.

[14]

W. HagerM. Yashtini and H. Zhang, An $\backslash$mathcalO(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm, SIAM Journal on Numerical Analysis, 54 (2016), 1535-1556. doi: 10.1137/15100401X.

[15]

M. Hong and Z. Luo, On the linear convergence of the alternating direction method of multipliers, Mathematical Programming, 162 (2017), 165-199, arXiv:1208.3922. doi: 10.1007/s10107-016-1034-2.

[16]

C. LiW. YinH. Jiang and Y. Zhang, An efficient augmented Lagrangian method with applications to total variation minimization, Computational Optimization and Applications, 56 (2013), 507-530. doi: 10.1007/s10589-013-9576-1.

[17]

Q. Liu, J. Luo, S. Wang M. Xiao and M. Ye, An augmented Lagrangian multi-scale dictionary learning algorithm EURASIP Journal on Advances in Signal Processing, 2011 (2011), p58. doi: 10.1186/1687-6180-2011-58.

[18]

M. LustigD. Donoho and J. Pauly, Sparse MRI: The application of compressed sensing for rapid MR imaging, Magnetic Resonance in Medicine, 58 (2007), 1182-1195. doi: 10.1002/mrm.21391.

[19]

R. Monteiro and B. Svaiter, Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers, SIAM Journal on Optimization, 23 (2013), 475-507. doi: 10.1137/110849468.

[20]

Y. Nesterov, Smooth minimization of non-smooth functions, Mathematical Programming, 103 (2005), 127-152. doi: 10.1007/s10107-004-0552-5.

[21]

Y. OuyangY. ChenG. Lan and E. Pasiliao, An accelerated linearized alternating direction method of multipliers, SIAM Journal on Imaging Sciences, 8 (2015), 644-681. doi: 10.1137/14095697X.

[22]

K. PruessmannM. WeigerP. Börnert and P. Boesiger, Advances in sensitivity encoding with arbitrary k-space trajectories, Magnetic Resonance in Medicine, 46 (2001), 638-651.

[23]

K. ScheinbergD. Goldfarb and X. Bai, Fast first-order methods for composite convex optimization with backtracking, Foundations of Computational Mathematics, 14 (2014), 389-417. doi: 10.1007/s10208-014-9189-9.

[24]

L. Shepp and B. Logan, The Fourier reconstruction of a head section, IEEE Transactions on Nuclear Science, 21 (1974), 21-43. doi: 10.1109/TNS.1974.6499235.

[25]

Y. WangJ. YangW. Yin and Y. Zhang A new alternating minimization algorithm for total variation image reconstruction, A new alternating minimization algorithm for total variation image reconstruction, SIAM Journal on Imaging Sciences, 1 (2008), 248-272. doi: 10.1137/080724265.

[26]

B. Wohlberg, Efficient Algorithms for Convolutional Sparse Representations, IEEE Transactions on Image Processing, 25 (2016), 301-315. doi: 10.1109/TIP.2015.2495260.

[27]

J. YangY. Zhang and W. Yin, A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data, IEEE Journal of Selected Topics in Signal Processing, 4 (2010), 288-297.

[28]

X. YeY. Chen and F. Huang, Computational acceleration for MR image reconstruction in partially parallel imaging, IEEE Transactions on Medical Imaging, 30 (2011), 1055-1063.

[29]

X. ZhangM. BurgerX. Bresson and S. Osher, Bregmanized nonlocal regularization for deconvolution and sparse reconstruction, SIAM Journal on Imaging Sciences, 3 (2010), 253-276. doi: 10.1137/090746379.

[30]

X. ZhangM. Burger and S. Osher, A unified primal-dual algorithm framework based on Bregman iteration, Journal of Scientific Computing, 46 (2011), 20-46. doi: 10.1007/s10915-010-9408-8.

[31]

M. Zhu and T. Chan, An efficient primal-dual hybrid gradient algorithm for total variation image restoration, UCLA CAM Report, (2008), 8-34.

[32]

M. ZhuS. Wright and T. Chan, Duality-based algorithms for total-variation-regularized image restoration, Computational Optimization and Applications, 47 (2010), 377-400. doi: 10.1007/s10589-008-9225-2.

Figure 1.  The objective function values and relative error vs. CPU time for first instance
Figure 2.  The objective function values and relative error vs.CPU time for second instance
Figure 3.  Sensitivity maps of data1
Figure 4.  Left: True image; Right: Cartesian mask for data 1
Figure 5.  Comparison of 400 iterations of BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ in partially parallel image reconstruction (Top) and their differences with true image (Bottom) using data1. From left to right: BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ
Figure 6.  Sensitivity maps of data2
Figure 7.  Left: True image; Right: Cartesian mask for data 2
Figure 8.  Comparison of 400 iterations of BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ in partially parallel image reconstruction (Top) and their differences with true image (Bottom) using data2. From left to right: BOSVS, TVAL3, ALADMML, and ABOSVS-Ⅱ.
Table 1.  Parameter settings for ABOSVS-I and ABOSVS-Ⅱ in (4.1). Note that $\rho$ and $\tau$ are only used in the first instance
ParametersασηminCβkρτ
Values102||A||2/101001/k11
ParametersασηminCβkρτ
Values102||A||2/101001/k11
Table 2.  Parameter settings for ABOSVS-Ⅱ in 4.2
ParametersασηminC
Values10-10n2||A||2/5100
ParametersασηminC
Values10-10n2||A||2/5100
Table 3.  Comparison of objective function value, relative error, and CPU time in seconds using data1
AlgorithmsObjective valueRelative errorCPU
BOSVS15.468410.05962528.0
TVAL356.72630.050228.6
ALADMML15.468100.049966.5
ABOSVS-Ⅱ15.445350.047949.7
AlgorithmsObjective valueRelative errorCPU
BOSVS15.468410.05962528.0
TVAL356.72630.050228.6
ALADMML15.468100.049966.5
ABOSVS-Ⅱ15.445350.047949.7
Table 4.  Comparison of objective function value, relative error, and CPU time in seconds using data2
AlgorithmsObjective valueRelative errorCPU
BOSVS5007e+20.06467448.4
TVAL31704e+30.050681.2
ALADMML5.006e+20.0499180.0
ABOSVS-Ⅱ4.970e+20.0479144.7
AlgorithmsObjective valueRelative errorCPU
BOSVS5007e+20.06467448.4
TVAL31704e+30.050681.2
ALADMML5.006e+20.0499180.0
ABOSVS-Ⅱ4.970e+20.0479144.7
[1]

Yunhai Xiao, Junfeng Yang, Xiaoming Yuan. Alternating algorithms for total variation image reconstruction from random projections. Inverse Problems & Imaging, 2012, 6 (3) : 547-563. doi: 10.3934/ipi.2012.6.547

[2]

Juan C. Moreno, V. B. Surya Prasath, João C. Neves. Color image processing by vectorial total variation with gradient channels coupling. Inverse Problems & Imaging, 2016, 10 (2) : 461-497. doi: 10.3934/ipi.2016008

[3]

Zhengmeng Jin, Chen Zhou, Michael K. Ng. A coupled total variation model with curvature driven for image colorization. Inverse Problems & Imaging, 2016, 10 (4) : 1037-1055. doi: 10.3934/ipi.2016031

[4]

Huan Gao, Yu-Hong Dai, Xiao-Jiao Tong. Barzilai-Borwein-like methods for the extreme eigenvalue problem. Journal of Industrial & Management Optimization, 2015, 11 (3) : 999-1019. doi: 10.3934/jimo.2015.11.999

[5]

Baoli Shi, Zhi-Feng Pang, Jing Xu. Image segmentation based on the hybrid total variation model and the K-means clustering strategy. Inverse Problems & Imaging, 2016, 10 (3) : 807-828. doi: 10.3934/ipi.2016022

[6]

Xavier Bresson, Tony F. Chan. Fast dual minimization of the vectorial total variation norm and applications to color image processing. Inverse Problems & Imaging, 2008, 2 (4) : 455-484. doi: 10.3934/ipi.2008.2.455

[7]

Ke Chen, Yiqiu Dong, Michael Hintermüller. A nonlinear multigrid solver with line Gauss-Seidel-semismooth-Newton smoother for the Fenchel pre-dual in total variation based image restoration. Inverse Problems & Imaging, 2011, 5 (2) : 323-339. doi: 10.3934/ipi.2011.5.323

[8]

Xiaoqun Zhang, Tony F. Chan. Wavelet inpainting by nonlocal total variation. Inverse Problems & Imaging, 2010, 4 (1) : 191-210. doi: 10.3934/ipi.2010.4.191

[9]

Gang Li, Xiaoqi Yang, Yuying Zhou. Stable strong and total parametrized dualities for DC optimization problems in locally convex spaces. Journal of Industrial & Management Optimization, 2013, 9 (3) : 671-687. doi: 10.3934/jimo.2013.9.671

[10]

Rinaldo M. Colombo, Francesca Monti. Solutions with large total variation to nonconservative hyperbolic systems. Communications on Pure & Applied Analysis, 2010, 9 (1) : 47-60. doi: 10.3934/cpaa.2010.9.47

[11]

Lacramioara Grecu, Constantin Popa. Constrained SART algorithm for inverse problems in image reconstruction. Inverse Problems & Imaging, 2013, 7 (1) : 199-216. doi: 10.3934/ipi.2013.7.199

[12]

Jianjun Zhang, Yunyi Hu, James G. Nagy. A scaled gradient method for digital tomographic image reconstruction. Inverse Problems & Imaging, 2018, 12 (1) : 239-259. doi: 10.3934/ipi.2018010

[13]

Liejune Shiau, Roland Glowinski. Operator splitting method for friction constrained dynamical systems. Conference Publications, 2005, 2005 (Special) : 806-815. doi: 10.3934/proc.2005.2005.806

[14]

Murat Adivar, Shu-Cherng Fang. Convex optimization on mixed domains. Journal of Industrial & Management Optimization, 2012, 8 (1) : 189-227. doi: 10.3934/jimo.2012.8.189

[15]

Liyan Ma, Lionel Moisan, Jian Yu, Tieyong Zeng. A stable method solving the total variation dictionary model with $L^\infty$ constraints. Inverse Problems & Imaging, 2014, 8 (2) : 507-535. doi: 10.3934/ipi.2014.8.507

[16]

Chunlin Wu, Juyong Zhang, Xue-Cheng Tai. Augmented Lagrangian method for total variation restoration with non-quadratic fidelity. Inverse Problems & Imaging, 2011, 5 (1) : 237-261. doi: 10.3934/ipi.2011.5.237

[17]

Florian Krügel. Some properties of minimizers of a variational problem involving the total variation functional. Communications on Pure & Applied Analysis, 2015, 14 (1) : 341-360. doi: 10.3934/cpaa.2015.14.341

[18]

Adriana González, Laurent Jacques, Christophe De Vleeschouwer, Philippe Antoine. Compressive optical deflectometric tomography: A constrained total-variation minimization approach. Inverse Problems & Imaging, 2014, 8 (2) : 421-457. doi: 10.3934/ipi.2014.8.421

[19]

Yuyuan Ouyang, Yunmei Chen, Ying Wu. Total variation and wavelet regularization of orientation distribution functions in diffusion MRI. Inverse Problems & Imaging, 2013, 7 (2) : 565-583. doi: 10.3934/ipi.2013.7.565

[20]

Johnathan M. Bardsley. An efficient computational method for total variation-penalized Poisson likelihood estimation. Inverse Problems & Imaging, 2008, 2 (2) : 167-185. doi: 10.3934/ipi.2008.2.167

2016 Impact Factor: 1.094

Metrics

  • PDF downloads (10)
  • HTML views (148)
  • Cited by (0)

[Back to Top]