Fast self-adaptive regularization iterative algorithm for solving split feasibility problem

Split feasibility problem (SFP) is to find a point which belongs to one convex set in one space, such that its image under a linear transformation belongs to another convex set in the image space. This paper deals with a unified regularized SFP for the convex case. We first construct a self-adaptive regularization iterative algorithm by using Armijo-like search for the SFP and show it converges at a subliner rate of \begin{document}$ O(1/k) $\end{document} , where \begin{document}$ k $\end{document} represents the number of iterations. More interestingly, inspired by the acceleration technique introduced by Nesterov[ 12 ], we present a fast Armijo-like regularization iterative algorithm and show it converges at rate of \begin{document}$ O(1/k^{2}) $\end{document} . The efficiency of the algorithms is demonstrated by some random data and image debluring problems.


Introduction. Split feasibility problem (SFP) is to find a point
x * ∈ C and Ax * ∈ Q, (1.1) where C and Q are nonempty, closed and convex subsets of N and M , respectively; A : N → M is a given bounded linear operator (here we denote its adjoint operator as A T ). This problem was first proposed by Censer and Elfving [4] to mode for inverse problems arising from medical image reconstruction and intensity modulated radiation therapy [1,3]. Various algorithms have been invented to solve it (see [5,14,17,19,20] and references therein). If the set Q takes the form of Q := {b ∈ M }, the SFP collapses to the well studied Linear Inverse Problem (LIP)(see [6,16] and the references therein). Hence, SFP is a natural generalization of LIP. CQ algorithm is one of the most popular solvers for SFP which was first introduced by Byrne [2]. Indeed, the CQ algorithm can be regarded as an application of the gradient-projection method to SFP [11], when we formulate (1.1) as a convex minimization problem. An important notice is that we often face the case where A is ill-posed in many applications such as image reconstruction. To deal with the ill-posed case, ones usually use the popular Tikhonov , regularization to promote stability of solutions. Such a regularization technique also has been applied to SFP in recent works [18,10,9]. However, with the increasing dimensionality of the data, many problems have some more special structures, such as (group) sparsity, lowrankness making the Tikhonov regularization not fitting for such structures. As a consequence, the numerical solutions obtained from the traditional model (1.1) often are not satisfactory, which also encourages us to consider a more general model of SFP embracing a generic regularization term. Mathematically, the regularized SFP takes the form of: where ϕ(x) is a convex function, which serves as the role of regularization. Obviously, when ϕ(x) := 0 and ϕ(x) := 1 2 x 2 , model (1.2) reduces to the unregularized and Tikhonov regularized SFP, respectively. If we take ϕ(x) := x 1 , it is the model considered in [10,9] for promoting sparse solutions of the SFP. Moreover, model (1.2) also covers regularized LIPs studied in [15].
Similar to the reformulation, we transfer the constraint Ax ∈ Q into the objective function of (1.2) and get the following minimization problem: where σ > 0 is a regularization parameter and where P Q (Ax) denotes the orthogonal projection of Ax onto Q, that is, In this paper, we only consider the case where the regularize function is convex for the purpose of algorithmic design and we name our algorithms as regularization iterative algorithm since we extend our problem to the general regularization optimization problem. We construct a self-adaptive regularization CQ algorithm by using Armijo-like technique in a new way for (1.3) and show it converges at a subliner rate of O(1/k), where k represents the number of iterations, which, as far as we know, is not discussed in the SFP literature. To further improve its convergence rate, we employ the acceleration technique introduced by Nesterov [12] to the CQ algorithm such that the resulting algorithm is also convergent with rate of O(1/k 2 ). All theoretical results are clearly demonstrated through preliminary numerical results with applications to random synthetic data and image deblurring problems.
The rest of the paper is organized as follows. Some basic definitions and lemmas are given in Section 2. Section 3 presents the self-adaptive regularization CQ algorithm (new version construction of CQ algorithm) and show its convergence in a new way. In Section 4, an accelerated self-adaptive regularization CQ algorithm is proposed and the convergence is proved under some suitable conditions. Section 5 is devoted to numerical applications to general split feasibility problem (random synthetic data) with no regularization and image deblurring problems. The conclusion appears in Section 6.
2. Preliminaries. In this section, we give some preliminaries which will be used in the sequel. Let N be an N −dimensional Euclidean space. For any two vectors x, y ∈ N , x, y denotes the standard inner product. Furthermore, we denote · 1 , · and · ∞ the standard l 1 -norm, l 2 -(Euclidean) norm, and l ∞ -norm, respectively.
Definition 2.1 [7]. Let f : N → be convex. The subdifferential of f at x is defined as An element of ∂f (x) is said to be a subgradient of f .
We know the following results .
Then, a k ≤ c for every k ≥ 1.
3. Self-adaptive regularization CQ algorithm and its convergence. Define the following minimization problem: where f (x) := 1 2 Ax − P Q (Ax) 2 , ϕ(x) is defined as in (1.2), σ > 0 is the regularization parameter. Note that f (x) defined as above is a convex and differentiable function. For any L > 0, consider the following quadratic approximation of F (x) at a given point y: Suppose p L (y) is the minimizer of the approximation function Q L (x, y) at point y on set C, that is, By ignoring constant terms in y, we can further simplify the above scheme and easily obtain Clearly, from (3.3), we know the basic step of regularization CQ (projection gradient) method for the problem (P) is where 1 L plays the role of a stepsize.
If we consider the standard SFP without regularization term, that is, ϕ(x) = 0, the previous algorithm reduces to the standard CQ algorithm introduced in [1], that is However, the standard CQ algorithm employs a fixed stepsize related to A 2 which adds amount of calculation, and the next iterate generated by the algorithm relies only on the current one, without previously computed iterates, which leads to slow convergence. Hence, in this paper, we will present a self-adaptive regularization iterative algorithm by adopting Armijo-like technique, which needs not to compute A 2 and makes a sufficient decrease of the objective function at each iterate, see the following algorithm 3.1. To further improve the convergence, we will purpose an accelerated self-adaptive iterative algorithm based on Algorithm 3.1, which computes the next iterate based not only on the previous one, but also on two or more previously computed iterates, more details can be found in Section 4.
Before proceeding with proposing our algorithms we recall the following lemma, which is the well-known fundamental property for a smooth function in the class C 1,1 [Lemma 3.2. [12], [13]]. It will be crucial for the convergence analyses of self-adaptive regularization CQ algorithm and the new accelerated self-adaptive regularization iterative method.
be a continuously differentiable function with Lipschitz continuous gradient and Lipschitz constant L f . Then, for any L ≥ L f , Note that from Lemma 3.1, if L ≥ L f , then for any y ∈ N , Now, we construct the self-adaptive regularization CQ Algorithm for model (3.1) as follows.
Therefore, Algorithm 3.1 is an extension of the classical self-adaptive CQ algorithm.
Remark 3.4. Note that the sequence of function values {F (x k )} produced by Algorithm 3.1 is nonincreasing. Indeed, for every k ≥ 1, Proof. Obviously, from the search rule (3.8), β ≤ L k for every k ≥ 1. Since inequality (3.8) and (3.9) are satisfied for some βη m ≥ A 2 , we know that L k /η must violate it, i.e., L k ≤ η A 2 for every k ≥ 1. Hence, β ≤ L k ≤ η A 2 , for every k ≥ 1. This completes the proof.
Before narrating the next lemma, we first exhibit the first-order optimality condition of (3.7): or equivalently, where ξ k ∈ ∂ϕ(x k ). It is a variational inequality problem. Accordingly, we can obtain the following result.

Lemma 3.3.
For any x ∈ C , we have Proof. From (3.11), we know Since f and ϕ are convex, we have where ξ k ∈ ∂ϕ(x k ). Adding the above two inequalities yields On the other hand, by setting x := x k , y := x k−1 in (3.2), we get (3.18) Therefore, substituting (3.17) and (3.18) into (3.16), we obtain From (3.14), we arrive at Hence, we obtain the results.
Theorem 3.1. Suppose the solution set of (3.1) is nonempty. Let z be an arbitrary solution of (3.1), and let {x k } be sequence generated by Algorithm 3.1. Then, for any k ≥ 1, From Lemma 3.2 and the fact that Summing the above inequality over h = 1, 2, · · · , k gives We note that Lemma 3.3 holds for any x ∈ C. Hence, taking x := x h , x k := x h and Multiplying the above inequality by (h − 1) and summing over h = 1, 2, · · · , k, we obtain

Adding (3.19) and (3.20) times
which can be simplified as Then, The assertion of this theorem is obtained.
Remark 3.5. The above theorem means that obtaining an ε-optimal solution, denoted byx, requires the number of iterations at most [ζ/ε] such that 4. Fast regularization CQ algorithm and its convergence. In this section, we will present a fast self-adaptive regularization CQ algorithm and show its convergence, which is simple but with improved convergence.
Similarly, the first-order optimality condition of (4.1) is also equivalent to a variational inequality: where ξ k ∈ ∂ϕ(x k ). Accordingly, we can obtain the following Lemma 4.1. then, for any x ∈ C, we have Proof. The process of proof is similar to that of Lemma 3.3 with setting x k−1 instead of y k .
Proof. It is easy to see from (4.6) that holds for arbitrary x ∈ C. Let x := x k and x := z in the above inequality, respectively, we have By the definition of u k , the above two inequalities can be rewritten as 2 and Multiplying by (t k+1 − 1) in (4.8) and adding it to (4.9) yields 2 L k+1 (4.10) Notice that (4.4) implies that t 2 k = t 2 k+1 − t k+1 . Thus, we multiply (4.10) by t k+1 and obtain 2 (4.11) Thus, let a := t 2 k+1 y k+1 , b := t k+1 x k+1 and c := (t k+1 −1)x k +z. By the Pythagoras . Thus, it follows from the above inequality and the notation of v k that 2 This completes the proof.
Proof. Let us define the quantities where t k , u k and v k are defined as in Lemma 4.2. Since t 1 = 1, using the definition of v k , we have Applying Lemma 3.3 to the points x = z, y = y 1 with L = L 1 , we get Consequently, 2 That is a 1 + b 1 ≤ c.
which combined with t k ≥ k+1 2 and using Lemma 3.2 yields The desire result follows.

5.
Numerical results. In this section, we focus on verifying the efficiency of the proposed fast iterative algorithm. We consider two problems: the first one is a general randomly generated SFP without regularization; and the last one is an l 1 -regularized wavelet-based image deblurring problem. All codes were written by Matlab 2014b on a laptop with an Intel quad-core i7 2.4GHz CPU with 8GB of RAM running on Vista operating system. 5.1. Random synthetic date. We first consider the general SFP(1.1) without regularization for random synthetic data. The problem is constructed randomly as follows: where the vector b is generated by using the following way: given a random N −dimensional negative vector (each component is negative) z ∈ C, r = z , taking b = Az. Find x ∈ C and Ax ∈ Q.
To exhibit the promising performance of the fast CQ algorithm (Algorithm 4.1), we compare it with the CQ algorithm (abbreviated by " CQ ") and Algorithm 3.1 (indeed, it is the Armijo-like algorithm introduced in [14]). Throughout, we set β = 4, η = 3, γ = 0.3 and take e 0 = (0, 0, · · · , 0) as the initial point in this example. Set Err as the termination rule for the algorithms. In our experiments, we test three scenarios with dimensions (N, M ) = (10,20), (20,40), (50, 50). The numerical results can be seen from Table 1. In this table, "Iter."," s" denote the number of iterations and cpu time in seconds, respectively.
Obviously, from Table 1, Algorithm 4.1 outperforms CQ algorithm and Algorithm 3.1 by reason of taking much less iterations and computing time. Hence, the results manifest that our algorithms are efficient for solving the SFP (1.1).

5.2.
Image deblurring. Now, we consider a wavelet-based image deblurring problem, which takes the form of wherex andx represent the clean and restored images, respectively. Obviously, a larger SNR value implies an image with higher quality. Furthermore, we use ≤ 10 −5 to be the stopping criterion. Below, we first report the number of iterations (Iter.), computing time in seconds (s), SNR values (SNR) of model (5.2) in Table 2.
Clearly, the results in Table 2 also show us that Algorithm 4.1 converges faster than Algorithm 3.1. Then, we plot evolution of the SNR with respect to iterations in Fig.1 and the objective values with respect to iterations in Fig.2, which graphically show that Algorithm 4.1 improves the convergence of Algorithm 3.1.
In Fig.3, the clean images and corrupted images are listed at the top row and the recovered images by Algorithm 3.1 and Algorithm 4.1 are listed at the bottom row in Subfigs (a), (b)and (c), respectively.
From above, we can see that our algorithms perform well on image debluring. Hence, all results support the effectiveness of the accelerated strategy. 6. Conclusions. We have consider a unified regularized SFP which is an extension of the timely regularized linear inverse problem. To handle this model, we have studied an application of the classical CQ algorithm to the resulting regularized version. Preliminary numerical results indicate that the iterative schemes of Algorithm 3.1