Quadratic optimization with two ball constraints

In this paper, the minimization of a general quadratic function subject to two ball constraints, called two ball trust-region subproblem (TBTRS), is studied. It is shown that the global optimal solution can be found by solving two extended trust-region subproblems. Strong duality conditions for two special cases are discussed. Finally, a comparison of results of the new algorithm with the other two recently proposed algorithms and CVX software are presented for several classes of randomly generated test problems.


1.
Introduction. This paper studies the minimization of a general quadratic function subject to two ball constraints, called two ball trust-region (TBTRS), which can be given as follows: ||x − c 2 || 2 ≤ δ 2 2 , where n ∈ N, x, c 1 , c 2 ∈ R n (c 1 = c 2 ), A ∈ R n×n is a symmetric matrix, and δ 1 , δ 2 ∈ R + . If A is positive semidefinite, then TBTRS is solvable in polynomial time. Therefore, throughout this paper it is assumed that A is indefinite and Slater condition holds i.e., there exists x ∈ R n such that ||x − c 1 || 2 < δ 2 1 , ||x − c 2 || 2 < δ 2 2 . The TBTRS arises in the numerical solution of parameter identification problems [3,22,30]. Consider the operator equation where the operator F is nonlinear and continuous, s ∈ R n and y ∈ R. Examples of ill-posed nonlinear problems include inverse problem [17,18], inverse scattering [19] and nonlinear Fredholm first kind integral equations [23] in which F is an integral operator of the form F(s)(t) = b a K (τ, t, s(τ )) dτ, a ≤ t ≤ b, s ∈ R n and K is nonlinear in s. To obtain reasonable approximate solutions, regularization is used, that replaces the ill-posed problem with a stabilized problem whose solution depends on a parameter, called the regularization parameter. These methods need to solve the following minimization problem: To solve (2), a sequence of trust-region subproblems in the form are solved, where s k k ∈ N is the current iterate satisfying ||s k || ≤ δ 2 , a k = F s (s k ) T (F(s k ) − y) and A k = F s (s k ) T F s (s k ) + H k , where H k is obtained using a structured Quasi-Newton method [10]. The sequence {s k } (s k+1 = s k + x k , x k is a solution of (2)) converges only locally under some standard conditions [14]. An appropriate initial point for convergence is not always available and one can solve the following TBTRS to generate a sequence of solution {s k } of (2) to guarantee global convergence: Further applications where TBTRS arises can be found in [13,31]. In [15], the author studied TBTRS and proposed a solution approach under the assumption that the objective function is convex. The TBTRS is a generalization of trust-region subproblems (TRS) and a special case of two trust-region subproblems (TTRS) [8,9]. Its necessary and sufficient optimality conditions are studied in [5,24]. In general, the semidefinite programming (SDP) relaxation is not exact for it. However, it is exact if and only if the Hessian of Lagrangian is positive semidefinite at global solution [21]: X xx T .
In this paper, it is proved that TBTRS can be solved by solving two extended trust-region subproblems in the following form: where b ∈ R n and α ∈ R. The extended trust-region subproblem (eTRS) is a wellstudied problem, specially in recent years [11,27,28]. Its classical SDP relaxation is as follows which is not exact in general [29]: First the authors in [29] proposed an exact SOCP/SDP 1 formulation for eTRS. Beck and Eldar in [4] have studied eTRS when dim(Ker(A − λ 1 I)) ≥ 2 which is equivalent to λ 1 = λ 2 (λ 1 and λ 2 are the two smallest eigenvalues of A and Ker(A) is the null space of matrix A). Under this condition, they have proved that the following conditions are necessary and sufficient optimality conditions: where ν and η are the Lagrange multipliers of eTRS. In 2013, Burer and Anstreicher in [6] rederived the SOCP/SDP formulation in [29] and later it was extended to the case where there are m non-intersecting linear constraints in the ball [7]. Jeyakumar and Li in [16] have shown that dim( Ker(A − λ 1 I n )) ≥ 2 and the Slater condition together ensure that a set of combined first and second-order Lagrange multiplier conditions are necessary and sufficient for the global optimality of eTRS and consequently for strong duality. In [28], the authors presented efficient algorithms to solve the problem using a diagonalization scheme that requires solving a simple convex minimization problem. Moreover in [11] the authors rederived the SOCP/SDP formulation by different analysis. Also, in [26,27] the authors for the first time were able to solve large-scale eTRS by the generalized eigenvalue approach. The rest of the paper is organized as follows. In Section 2, it is proved how the global optimal solution of TBTRS can be found by solving two eTRS and for two special cases strong duality are discussed. Numerical results for several classes of test problems are presented in Section 3 to show the efficiency of the new approach compared to Sakaue et. al algorithm [25], the recent Alternating Direction Method of Multipliers (ADMM) algorithm suggested in [2] and CVX software [12] for small, medium and large-scale problems.
2. Main results. In this section, it is shown that TBTRS can be solved by solving two eTRS. Let then the feasible region is singleton.
The proof of the second part is obvious.
The following assumption is made to avoid redundant constraint and empty feasible region.
Without loss of generality, we assume that From Theorem 2.3, it follows that we can solve TBTRS by solving the following two eTRS: and min 1 2 The eTRS is efficiently solvable using the recent generalized eigenvalue approach proposed in [27], thus we can solve TBTRS efficiently as outlined in the following algorithm. - Step 1: Check the feasibility of TBTRS. If ||c 1 − c 2 || > δ 1 + δ 2 , then TBTRS is infeasible, Stop; else go to Step 2.
Step 2: Solve the resulting TRS and Stop; else go to Step 3.
Step 4: The optimal solution is given by In the following, strong duality for two special cases of TBTRS are discussed. Let x ∈ R n be a local solution of TBTRS. Then, there exists a pair of Lagrange multipliers (γ, µ) ∈ R 2 + that satisfying the Karush-Kuhn-Tucker (KKT) conditions: where H(γ, µ) = A + µI n + γI n is the Hessian of Lagrangian that has at most two negative eigenvalues [24]. The following theorem gives necessary optimality conditions for TBTRS.
Proof. When the gradients of constraints are linearly independent at the optimal solution, the proof is similar to the proof of Theorem 4.3 in [24]. Now suppose that the gradients of constraints are linearly dependent at x * , the optimal solution of TBTRS. Then, According to (8), the constraints of TBTRS are as follow: Since A is indefinite, at least one of the constraints is active at the optimal solution. Therefore, one of the following cases may occur: Case 1: Consider that at the optimal solution both constraints are active: If α = δ1 δ2 , then ||c 1 − c 2 || = δ 1 − δ 2 . From Lemma 2.2 one of the constraints is redundant. Thus, Hessian of Lagrangian is positive semidefinite. If α = − δ1 δ2 , then ||c 1 − c 2 || = δ 1 + δ 2 and so the feasible region of TBTRS is singleton, which contradicts the Slater condition. Case 2: Let ||x * − c 1 || = δ 1 and ||x * − c 2 || < δ 2 . Therefore, the optimal solution of TBTRS is the local non-global minimum (LNGM) or the global optimal solution of TRS. If x * is the global optimal solution of TRS, then Hessian of Lagrangian is positive semidefinite and analogously, if x * is a LNGM of TRS, then Hessian of Lagrangian has n − 1 nonnegative eigenvalues [20]. Case 3: Let ||x * − c 1 || < δ 1 and ||x * − c 2 || = δ 2 . This case is similar to the Case 2.
In what follows, special cases of TBTRS where strong duality holds are discussed. Nesterov and Wolkowicz showed that if the Hessian of Lagrangian is positive semidefinite at a global solution, then strong duality holds ( [21], p.404). By considering their result, we have the following theorem.
If λ 1 + γ + µ < 0, then three smallest eigenvalues of H(γ, µ) are negative, which contradicts the second order necessary optimality condition of TBTRS. Thus, any local minimum of TBTRS is a global minimum.
3. Numerical experiments. In this section, the performance of the new algorithm for solving TBTRS is compared with two algorithms in the literature and CVX software on several classes of test problems. Comparison with the Sakaue et. al's algorithm [25] is performed for n ≤ 30 as the computational cost of their algorithm is too high when we increase the dimension. For test problems where strong duality holds, comparison is done with CVX 2.1 [12] which is solving the SDP relaxation (3) and finally, for all set of test problems, the results of the recent ADMM algorithm of [2] are reported. All computations are performed in MATLAB R2015a on a 2.50 GHz laptop with 8 GB of RAM. To solve the TRS within the algorithm, the algorithm of [1] is used. Also, the algorithm of [27] is used to solve eTRS.
• First class of test problems: In this class, test problems are generated based on the following lemma.
Lemma 3.1 (Lemma 2, [2]). Let A be a symmetric matrix and suppose that λ 1 < min{0, λ 2 }. Also, let eigenvector v 1 is associated with λ 1 . Then, there exists a linear term "a" for which the vector (v 1 + c i ) is the LNGM of P c i : To generate the desirable random instances of TBTRS, we proceed as follows. First a TRS instance of the form P c i is constructed having LNGM based on Lemma 3.1. Then, the constraint ||x − c j || 2 ≤ δ 2 j (j = i) is added to enforce that the global minimizer, (−v 1 + c i ), of TRS be infeasible but the LNGM, (v 1 + c i ), remains feasible ( Figure 1) for TBTRS. For 20% of the generated instances, the LNGM of the TRS P c j is also in the feasible region of TBTRS. Moreover, strong duality fails at 90% of the generated instances. Result are reported in Tables 2 and 3 for the average of 100 runs. It can be observed from Table 2 that, the new algorithm is much faster than ADMM algorithm [2] providing practically the same objective values (the difference is of O(10 −10 )). In Table 3, a comparison with the Sakaue et. al's algorithm [25] for small dimensions problems is presented. Numerical results show that the new algorithm is significantly faster, leading to very close solution results. • Second class of test problems: In this class, TBTRS instances were generated such that the global minimizers of P c 1 and P c 2 are all infeasible for TBTRS. Then, the vector "a" is generated such that it is orthogonal to some eigenvectors corresponding to the smallest eigenvalue of A. Thus, P c i (i = 1, 2) has no LNGM (Lemma 3.2, [20]). Moreover, in this class, strong duality holds for at least 90% of the generated instances. Results are summarized in Tables 4 and 5 for the average of 100 runs. In Table 4, the new algorithm has significant advantages over the ADMM algorithm [2] in terms of CPU time with very similar objective values. For smaller dimensions (n ≤ 30), the new algorithm is compared with the Sakaue et. al's algorithm [25]. The corresponding results are reported    in Table 5, showing the new algorithm is extremely faster and the objective values are quite close to each other. • Third class of test problems: In this class, TBTRS instances were generated such that strong duality holds and for that purpose the Theorem 2.5 was used. In this case, the SDP relaxation of TBTRS will be exact [21]. Therefore, matrix A is generated such that the multiplicity of its minimum eigenvalue is at least two. A comparison of the results obtained with the new algorithm, the CVX (that solves the SDP relaxation), the Sakaue et. al's algorithm and the ADMM algorithm is presented in terms of CPU time and objective function values. Results are  summarized in Tables 6 and 7 for the average of 100 runs. As for previous classes, results show that the new algorithm is significantly faster than the other algorithms. We were not able to provide results using CVX applied in problems with dimension higher than 2000, due to limited computational resources.

4.
Conclusions. In this paper, the problem of minimizing an indefinite quadratic function subject to two ball constraints is studied and an algorithm based on extended trust-region subproblems is proposed to find its global optimal solution. Comparison on several classes of randomly generated test problems with several algorithms showed that the proposed algorithm is significantly faster while giving almost equal objective values. Extending the proposed approach to include more than two balls constraints with additional linear constraints can be considered for future research.