ON THE GLOBAL CONVERGENCE OF A PARAMETER-ADJUSTING LEVENBERG-MARQUARDT

. The Levenberg-Marquardt (LM) method is a classical but popular method for solving nonlinear equations. Based on the trust region technique, we propose a parameter-adjusting LM (PALM) method, in which the LM parameter µ k is self-adjusted at each iteration based on the ratio between actual reduction and predicted reduction. Under the level-bounded condition, we prove the global convergence of PALM. We also propose a modiﬁed parameter-adjusting LM (MPALM) method. Numerical results show that the two methods are very eﬃcient.

1. Introduction. In this paper, we aim to solve the system of nonlinear equations where F : R n → R m is continuously differentiable. Define . It is obvious that solving system (1) is equivalent to find the stationary point of the unconstrained optimization problem, minimize x∈R n f (x). ( A classical but very popular method for solving nonlinear equations (1) is the Levenberg-Marquardt method [5,6]. We do not attempt to survey the literature on LM method, which is vast. For the k−th iterate point x k , denote At the k−th iteration, the LM method calculates the direction p k by solving the following linear equations where µ k > 0 is the LM parameter which is updated at each iteration. Note that, if B k is nonsingular and µ k = 0, the the LM direction p k is reduced to the traditional Newton step. Thus, the LM method has global convergence and local quadratic convergence rate if the LM parameter µ k is chosen suitably and the Jacobian B k is nonsingular at the solution. However, the nonsingularity condition of the Jacobian is too strong. Yamashita and Fukushima [10] proved that, under the local error bound condition, the LM method maintain the quadratic convergence. Fan [1] proposed a modified Levenberg-Marquardt method, which uses the addition of an LM step and an approximate LM step as the LM direction. At every iteration, the modified LM method first obtains an LM step d k by solving the linear equations then solves the linear equations to obtain the approximate LM stepd k , and the LM direction is p k = d k +d k . The author showed that, under the local error bound condition, the modified LM method achieves the cubic convergence. The modified LM method with line search was recently introduced in [2]. This paper focuses on the choice of the LM parameter µ k . It is well known that, if ∇f (x) is Lipschitz continuous and nonsingular at the solution, the LM method converges quadratically if µ k is chosen as O( ∇f k ). Fan and Yuan [4] chose µ k = F k δ with δ ∈ (0, 2] and showed that, under the local error bound condition, the LM method converges superlinearly when δ ∈ (0, 1) and quadratically when δ ∈ [1,2]. Moré [7] reviewed the relationship between Levenberg-Marquardt method and the trust region method. Inspired by the trust region method, we proposed a new LM method, in which the parameter µ k is self-adjusted based on the ratio between actual reduction and predicted reduction. A similar parameter self-adjusted LM method was studied in [3]. However, the Lipschitz of F (x) and J F (x) is assumed to obtain the global convergence. In this paper, based on the traditional trust region method we proved that, the parameter-adjusting LM method we construct is convergent globally if F (x) is continuously differentiable and f (x) is level bounded.
The main differences between the modified parameter-adjusting LM method we proposed and the existing Levenberg-Marquardt method can be stated as follows. First, the choice of LM parameter µ k is different. In most of the existing LM methods, the parameter µ k is chosen as F k δ with δ ∈ (0, 2]. However, when the iterative sequence is close to the solution set, the LM parameter µ k = F k δ may be too small to lose its role. In our new LM method, the parameter µ k is automatically updated according to the ratio between actual reduction and predicted reduction, which is inspired by the success of traditional trust region method. Second, the assumption to obtain global convergence is relaxed. Fan and Pan [3] assumed that both F (x) and J F (x) are Lipschitz continuous. We show in this paper that the global convergence can also be proved without this assumption, which will expand the applicability of LM method.
The paper is organized as follows. In Section 2, we proposed a parameteradjusting LM method and obtain its global convergence under some assumptions. A modified parameter-adjusting LM method is proposed in Section 3. Finally, some numerical results are presented in Section 4.

2.
A parameter-adjusting LM method. Consider the quadratic model for approximating f around x k defined as follows For p k , which will be calculated in the following method, define the ratio between actual reduction and predicted reduction We present the parameter-adjusting Levenberg-Marquardt (PALM) method as follows.
The following lemma, which gives a lower bound of the predicted reduction, is a key to the proof of global convergence theorem.
Proof. Noting that (p, λ) = (p k , µ k ) satisfies the following relations which is indeed the Karush-Kuhn-Tucker system of the following problem min m k (p) Since m k (·) is a convex function, we have that p k is the optimal solution to the problem (6). If we take the Cauchy step defined as in [9,Chapter 4] then the following inequality follows from [9, Lemma 4.3] The global convergence requires the following assumption. Assumption 1. The mapping F is continuously differentiable and the level set Next, we show that the sequence generated by PALM method with η = 0 converges to a stationary point of the merit function f (x).
Theorem 2.2. Let Assumption 1 be satisfied and η = 0 in Algorithm 1. Suppose that {x k } is generated by Algorithm 1, then we have Proof. We proceed the proof by contradiction. Suppose that there exist ε > 0 and an infinite set of positive integers N such that Note that From Taylor's theorem, we have that where In view of (5) in Lemma 2.1, we have for k ∈ N that Using (9) and (10), we obtain for k ∈ N that We have from (11) for k ∈ N with p k small enough that Therefore, ρ k ≥ 3/4 for k ∈ N with p k small enough, and by Algorithm 1, we have µ k+1 ≤ µ k when µ k ≥ µ. It follows that increase of µ k (by a factor 4) can only occur if µ k < µ. Therefore, we conclude that Suppose now that there exists an infinite index set K ⊂ N such that ρ k ≥ 1/4 for k ∈ K. If k ∈ K, we have from (4) and (5) that Since f is bounded from below, we have lim k∈K,k→∞ p k = 0, which implies µ k → ∞, k ∈ K, k → ∞, this contradicts (12). Hence no such infinite index set K can exist, and we must have ρ k < 1/4 for all k ∈ N large enough. In this case, µ k , k ∈ N will eventually be enlarged by a factor 4 at every iteration, and we have µ k → ∞, which again contradicts (12). Therefore, (8) is incorrect, giving .
Finally in this section, we show that the PALM method also globally converges to a stationary point when η ∈ (0, 1/4). If the entire sequence {x k } k≥m satisfies ∇f k ≥ ε, by using the reasoning in the proof of Theorem 2.2, we can show that this scenario does not occur. Therefore, we can choose the index l ≥ m such that x l+1 is the first iterate after x m with ∇f l+1 < ε. From Assumption 1, ∇f (x) is continuous, then there exists a positive scalar R > 0 such that From the framework of Algorithm 1, we can write where we have limited the sum to the iterations k for which x k = x k+1 , that is, those iterations on which a step was actually taken. Note that those iterations occur only under the condition ρ k ≥ η, by using (10) If p k ≤ ε/β for all k = m, m + 1, . . . , l, we have Otherwise, we have p k > ε/β for some k = m, m + 1, . . . , l, and so Since the sequence {f (x k )} ∞ k=0 is decreasing and bounded below, we have that Therefore, using (14) and (15), we can see which implies ∇f m → 0.

3.
A modified parameter-adjusting LM method. In this section, we proposed a modified parameter-adjusting LM (MPALM) method. One and only difference between PALM and MPALM is that the calculation of LM direction in MPALM is altered to The proof of global convergence for MPALM is almost the same as PALM.
The following key lemma can be proved as Lemma 2.1. Proof. The proof is very similar to Theorem 2.2. Define We proceed the proof by contradiction. Suppose that there exists ε > 0 and an infinite set of positive integers N such that As in the proof of Theorem 2.2, by using Lemma 3.1 we obtain for k ∈ N that Since F k ≥ ∇f k /β 1 , so we have where λ min (·) denotes the minimal eigenvalue of a matrix. Thus Let µ > µ N , where N is the first index in N , be sufficiently large such that when µ k ≥ µ, k ∈ N , p k is small enough satisfying We have from (19) for k ∈ N with p k small enough that The remainder is similar to the proof of Theorem 2.2.    We set µ 0 = 10 −3 F 0 , η = 0. The algorithms are terminated when the norm of derivative of f at x k is less than 10 −5 , i.e., ∇f k ≤ 10 −5 , or the number of the iterations exceeds 100 · n. The numerical experiments were carried out with MATLAB 2014b on a laptop with 2.3GHz CPU and 4GB memory . The numerical results for PALM and MPALM on those ten numerical problems are presented in Table 1. The second column (n, m) of the table denotes the dimensions of the problems. The third column shows that the initial point is x 0 , 10x 0 or 100x 0 , where x 0 is defined in the problems. "NF" , "NJ" and "IT" stand for the number of function calculations, Jacobian calculations and the number of iterations, respectively. Note that, for general nonlinear equations, the number of calculations of a Jacobian is usually n times of that of a function. The sign "-" denotes that the iterations had underflows or overflows. always obtain the solutions with different initial points. If we change the stopping criterion to f (x k ) = 1 2 F (x k ) 2 ≤ 10 −5 , the number of iterations will be fewer for most of the problems.

5.
Conclusions. Inspired by the traditional trust region method, we proposed a parameter-adjusting LM (PALM) method. The main difference between classical LM method and PALM is that, the LM parameter µ k in PALM is self-adjusted at each iteration according to the ratio between actual reduction and predicted reduction. An analogous method can be found in the literature. However, the global convergence of PALM is proved under a weaker condition.
Next, we will consider the local quadratic convergence of PALM. In the literature, a local error bound condition is necessary to prove the local quadratic convergence. A natural question is that, is it possible to relax the local error bound condition? Another question is, how to accelerate the PALM method to a higher order convergence rate. The papers [1,2] provide some insights for finding the answers.
Another very important question is about solving nonsmooth (or semismooth) nonlinear equations by Levernberg-Marquardt method. In the literature, almost all of the references on LM method focus on solving smooth nonlinear equations. It is well known that the nonsmooth nonlinear equations have various applications. Therefore, it is of significant importance for us to study the LM method for nonsmooth equations. Based on the existing results of nonsmooth Newton method, it is very likely to construct a LM method for solving nonsmooth equations and prove the corresponding convergence results.