EXISTENCE AND DECAY PROPERTY OF GROUND STATE SOLUTIONS FOR HAMILTONIAN ELLIPTIC SYSTEM

. In this paper we study the following nonlinear Hamiltonian elliptic system with gradient term where z = ( u,v ) ∈ R 2 . Under some suitable conditions on the potential and nonlinearity, we obtain the existence of ground state solutions in periodic case and asymptotically periodic case via variational methods, respectively. More-over, we also explore some properties of these ground state solutions, such as compactness of set of ground state solutions and exponential decay of ground state solutions.

1. Introduction and main results. We study the following nonlinear Hamiltonian elliptic systems with gradient term where z = (u, v) : R N → R 2 , N ≥ 3, b(x) = (b 1 (x), · · · , b N (x)) ∈ C 1 (R N , R N ), V ∈ C(R N , R) and f ∈ C(R N × R, R). Such a system arises when one is looking for stationary solutions to certain systems of optimal control (cf. Lions [16]) or systems of diffusion equations (cf. Itô [13] and Nagasawa [20]). In this paper we will prove the existence of ground state solutions of system (1.1) under suitable conditions on V and f , and show some properties of ground state solutions. System (1.1) and its variants were studied by a number of authors. But most of them focused on the case b(x) = 0 and V (x) = 0. For example, see [5,7,8,12,14] for the case of a bounded domain, and [3,4,10,19,24,33] for the case of the whole space R N .
When b(x) = 0, as we know, there are very few works in the context. In [32], Zhao and Ding first considered the following system of Hamiltonian type (1. 2) In this case, the appearance of the gradient term in this system will bring some difficulties, and the variational framework for the case b(x) = 0 cannot work any longer. Hence the authors handled (1.2) as a generalized Hamiltonian system, and established suitable variational framework through the study of the spectrum of Hamiltonian operator. Moreover, by using generalized linking theorem and reduction method, the existence and multiplicity of solutions are obtained. After that, Zhang et al. [34] studied the periodic superquadratic case and proved the existence of ground state solutions of system (1.2) by means of the linking and concentration compactness arguments. In [31], the authors considered the non-periodic superquadratic case for system (1.2) with constant vector b and V = 1. Since the problem is set in unbounded domain, the (C) c -condition does not hold in general.
With the aid of limit problem which is autonomous, the authors proved that the energy functional satisfies local (C) c -condition, and the least energy solution was obtained.
Recently, the paper [35] studied the Hamiltonian elliptic system with inverse square potential of the form and the ground state solutions was obtained by using non-Nehari manifold method developed by Tang [28,29]. Moreover, some asymptotic behaviors of ground state solutions, such as the monotonicity and convergence property of ground state energy, were also explored as µ → 0. In addition, the singularly perturbed problem has been considered in [36,37,39,40]. Here > 0 is a small parameter. More precisely, the authors proved the existence of semi-classical ground state solutions, and shown some new concentration phenomenons of these solutions on the minimum points of V or the maximum points of K.
In this paper, we are interested in the potential V and the nonlinearity f are asymptotically periodic at infinity. To the best of our knowledge, it seems that such a problem was not considered in literature before. More specifically, we will prove the existence of ground state solution of system (1.1) by applying variational methods and analyze some properties of ground state solutions by using some analysis techniques, such as compactness of set of ground state solutions and exponential decay of ground state solutions. As we all know, the associated limit problem plays an important role in studying the asymptotically periodic problem. Hence it is very necessary to study the limit problem of the asymptotically periodic problem. Noting that the limit form of asymptotic periodic system is periodic problem, and the other purpose of this paper is to study the system (1.1) under periodicity condition.
In order to precisely state our results we denote by H the class of functions h ∈ C(R N , R)∩L ∞ (R N , R) such that, for every > 0, the set x ∈ R N : |h(x)| ≥ has finite Lebesgue measure. Moreover, we assume that the potential V and the nonlinearity f satisfy the following conditions: , and there exist p ∈ (2, 2 * ) and c 0 > 0 such that We are now in a position to state the main results of this paper. On the periodic case we have the following results. Theorem 1.1. Suppose that (B1), (V1) and (A1)-(A5) are satisfied. Then system (1.1) has at least a ground state solution. If additionally | b| < 2, then (1) L is compact in H 2 (R N ), where L denotes the set of all ground state solutions of (1.1); (2) there exist constants c, C > 0 such that On the asymptotically periodic case we have the following results. Theorem 1.2. Suppose that (B1), (V2), (A1)-(A4) and (A6) are satisfied. Then system (1.1) has at least a ground state solution. If additionally | b| < 2, then (1) L is compact in H 2 (R N ), where L denotes the set of all ground state solutions of (1.1); (2) there exist constants c, C > 0 such that Remark 1. The role of condition (B1) is to ensure the self-adjointness of operator corresponding to system (1.1) (see [32]). Condition (V1) or (V2) implies that the potential function V (x) may change sign. Here we say that V (x) is sign-changing if V (x 1 ) < 0 < V (x 2 ) for some x 1 , x 2 ∈ R N . Remark 2. As an example of application of our main theorems we take where b i ∈ R for i = 1, 2, · · · , N and p ∈ (2, 2 * ). It is easy to verify that b satisfies (B1), V 0 and f 0 satisfy the periodic conditions, and V and f satisfy the asymptotically periodic conditions.
It is worth pointing out that although the paper [34] studied the existence of ground state solutions to system (1.1) under periodicity condition. But to the authors' knowledge, the main results obtained in this paper still are new. On the one hand, the conditions we assumed are different from those assumed in [34], and seem more natural than [34]. On the other hand, this paper carefully analyzes some properties of ground state solutions, but not in [34].
As a motivation we recall that there are many works devoted to the study on the nonlinear Schrödinger equation arising in the non-relativistic quantum mechanics See, for example [1,2,18,21,22,25,26,27,29,38,41] and the references therein. Especially, [1,2,18,26,27,29,38,41] considered the asymptotically periodic case, and obtained the existence results of solutions by means of various variational techniques. Motivated by the above facts, it is quite natural to ask if certain similar results can be forwarded to system (1.1). As we will see, the answer is affirmative. However, since the system (1.1) is of Hamiltonian type in R N , there are also some additional difficulties for system (1.1). One is that the energy functional associated with system (1.1) is strongly indefinite which has complicated geometric structure, and so the classical critical point theory cannot be applied directly; the other is that the energy functional does not satisfy the compactness condition. These new mathematical difficulties make the study of Hamiltonian elliptic system (1.1) particularly interesting and challenging. Now we give the main ideas for the proof of the main results. Our argument is variational, which can be outlined as follows. The solutions are obtained as critical points of the energy functional associated to system (1.1). Roughly speaking, because the energy functional of system (1.1) is strongly indefinite, we shall use generalized linking theorem [15,18] and the diagonal method [28,29] to construct a Cerami sequence for the energy functional at some level. After proving that the Cerami sequence is bounded, we show that its weak limit is a ground state solution of system (1.1). However, since we work in the whole space, the main difficulty when dealing with this problem is the lack of compactness of Sobolev embedding. It is natural to ask how to show this weak limit is nontrivial? To obtain our results, some arguments are in order. First, for the periodic case, we will take advantage of the invariance of the energy functional under translation and the concentration compactness principle [17] to solve this difficulty. Second, for the asymptotically periodic case, the energy functional loses the translation invariance, which bring new difficulty. For these reasons, many effective methods for periodic problem cannot be applied to asymptotically periodic one. With the help of the limit problem, we will take advantage of the asymptotic properties of the potential and nonlinearity to establish a global compactness result for bounded Cerami sequences in order to recover the Cerami compactness condition. Finally, by some analysis techniques, we prove some properties of ground state solutions for both periodic and asymptotically periodic case. Especially, for proving the exponential decay, the sub-solution estimates of |z| seems not work well since the presence of the gradient term in system (1.1), we give the sub-solution estimates of |z| 2 , and then describe the decay at infinity in a subtle way.
The remainder of this paper is organized as follows. In Section 2, we formulate the variational setting and introduce some useful preliminaries. In Section 3, we analyze the boundedness of (C) c -sequences. We complete the proofs of main results in Section 4 and 5, respectively.

2.
Variational setting and preliminaries. Below by | · | q we denote the usual L q -norm, (·, ·) 2 denote the usual L 2 inner product, c, c i or C i stand for different positive constants. For the sake of convenience, we need the following notations. Let Then system (1.1) can be rewritten as where H z (x, z) = f (x, |z|)z. In this way, system (1.1) can be regarded as a generalized Hamiltonian system (see [9]). Denote by σ(A) and σ e (A) the spectrum and the essential spectrum of the operator A, respectively. In order to establish a suitable variational framework for system (1.1), we need to analyze the properties of the spectrum of the associated Hamiltonian operator A. The proof can be seen [32], here so we omit the details.
It follows from Lemma 2.1 and Lemma 2.2 that the space L 2 := L 2 (R N , R 2 ) possesses the orthogonal decomposition such that A is negative definite (resp. positive definite) in L − (resp. L + ). Let |A| denote the absolute value of A and |A| 1/2 be the square root of |A|. Let E = D(|A| 1/2 ) be the Hilbert space with the inner product (z, w) = (|A| 1/2 z, |A| 1/2 w) 2 and norm z = (z, z) 1/2 . Moreover, it is clear that E possesses the following decomposition which is orthogonal with respect to the inner products (·, ·) 2 and (·, ·). Since σ(A) = R \ (−1, 1), one has and · is equivalent to the usual norm of H 1 (see [32]). Hence E embeds continuously into L q for all q ∈ [2, 2 * ] and compactly into L q loc for all q ∈ [1, 2 * ), then there exists constant Observe that in virtue of the assumptions (A1) and (A2), for any > 0, there exists positive constant C such that for all (x, s) ∈ R N × R + and p ∈ (2, 2 * ). By (A4) we obtain On E we define the energy functional Φ corresponding to system (1.1) by Lemma 2.2 implies that the energy functional Φ is strongly indefinite. Moreover, our hypotheses imply that Φ ∈ C 1 (E, R), and where the dot stands for the scalar product in R 2 , and a standard argument shows that critical points of Φ are solutions of system (1.1) (see [6,30]). In order to seek for the ground state solutions of system (1.1), we consider the following set which is introduced in Pankov [21] M := {z ∈ E\E − : Φ (z)z = 0 and Φ (z)w = 0 for any w ∈ E − }.
Following from Szulkin and Weth [25], we will call the set M the generalized Nehari manifold. Obviously, the set M is a natural constraint and it contains all nontrivial critical points of Φ. Let If m is attained by z 0 ∈ M, then z 0 is a critical point of Φ. Since m is the lowest level for Φ, z 0 be called a ground state solution of system (1.1).
Recall that for a functional Φ ∈ C 1 (E, R), Φ is said to be weakly sequentially lower semi-continuous if for any u n u in E one has Φ(u) ≤ lim inf n→∞ Φ(u n ), and Φ is said to be weakly sequentially continuous if lim We say that Φ satisfy the (C) c -condition if any (C) c -sequence has a convergent subsequence in E.
To prove the main results, we need the following generalized linking theorem due to [15,18].
Suppose that the following assumptions are satisfied: is bounded from below and weakly sequentially lower semicontinuous; (Ψ 2 ) Ψ is weakly sequentially continuous; (Ψ 3 ) there exist R > ρ > 0 and e ∈ X + with e = 1 such that Then there exist a constant c ∈ [κ, sup Φ(Q)] and a sequence {u n } ⊂ X satisfying For the sake of convenience, we write Employing a standard argument, one can check easily the following lemma (see [6]).
Then Ψ is nonnegative, weakly sequentially lower semi-continuous, and Ψ is weakly sequentially continuous. .
From Lemma 2.5, we have the following lemma.
In particular, we write E(z) := E − ⊕ R + z for z ∈ E\E − . Let z ∈ M, then Lemma 2.6 implies that z is the global maximum of Φ| E(z) . Hence, applying Lemma 2.6, we can prove the following results.
(ii) For z ∈ M, by (2.1) and (2.4) we have Lemma 2.8. Suppose that (V1) (or (V2)) and (A1)-(A4) are satisfied. Then for any e ∈ E + with e = 1, sup Φ(E − ⊕ R + e) < ∞, and there is R e > 0 such that In particular, there is a R 0 > ρ such that If not, then there are constant M > 0 and a sequence {z n } ⊂ E − ⊕ R + e with z n → +∞ such that Φ(z n ) > −M for all n. Denote w n = z n / z n = s n e + w − n , where s n ≥ 0, w − n ∈ E − , then 1 = w n 2 = s 2 a.e. on R N , and s n → s ≥ 0. Observe that, by (2.1) and (2.4) we get
Let z k = z k,n k , k ∈ N. Then, going if necessary to a subsequence, we have In a similar way to [25, Lemma 2.6], we have Lemma 2.11. Suppose that (V1) (or (V2)) and (A1)-(A4) are satisfied. Then for any z ∈ E \ E − , M ∩ E(z) = ∅, i.e., there exist t > 0 and w ∈ E − such that tz + w ∈ M.

3.
Behaviour of (C) c -sequence. for some M 0 > 0. Suppose to the contrary that z n → ∞ as n → ∞. Setting w n = z n / z n , then w n = 1. After passing to a subsequence, we may assume that w n w in E, w n w in L p loc for 2 ≤ p < 2 * , and w n (x) → w(x) a.e. on R N . Let If δ = 0, by the vanishing lemma (see [17,30]), then w + n → 0 in L p for any 2 < p < 2 * . It follows from (2.3) that for any s > 0, By virtue of (2.5) and (3.1), we have Let t n = s/ z n , then by (3.2) we get Observe that by (2.1) and (2.4), we obtain which implies that w + n 2 ≥ c 0 for some c 0 > 0. Hence, (3.3) yields a contradiction if s is large enough. Then δ > 0. Going if necessary to a subsequence, we may assume the existence of {k n } ⊂ Z N such that Let us definew n (x) = w n (x + k n ) so that Therefore passing to a subsequence,w + n →w + in L 2 loc andw + = 0. Note that if w = 0, then |z n (x + k n )| = |w n (x)| z n → ∞. Hence, it follows from (A3) and Fatou's lemma that since w n is bounded. Thus we get a contradiction and the desired conclusion holds true.
4. The periodic case. In this section we give the proof of the main result for the periodic case. Throughout this section, we always assume that (V1) and (A5) are satisfied. Let K := {z ∈ E\{0} : Φ (z) = 0} denote the set of all critical points of Φ. To describe some properties of ground state solutions, by using the standard bootstrap argument (see, e.g., [11] for the iterative steps) we can obtain the following regularity result.
Let L be the set of all ground state solutions of Φ. If z ∈ L then Φ(z) = m, a standard argument shows that L is bounded in E, hence, |z| 2 ≤ C 2 for all z ∈ L and some C 2 > 0. Therefore, as a consequence of Lemma 4.1 we see that, for each q ∈ [2, +∞), there is C q such that This, together with the Sobolev embedding theorem, implies that there is C ∞ > 0 with |z| ∞ ≤ C ∞ for all z ∈ L . Assume that (A1) and | b| < 2 hold, then there is C 0 > 0 independent of x and z ∈ L such that Proof. It follows from Lemma 4.1 that z ∈ H 2,q for any z = (u, v) ∈ L . Recall that |z| 2 = u 2 + v 2 . Then for i = 1, 2, · · · , N . This yields that where ∇z = (∇u, ∇v). Since z = (u, v) is a solution of system (1.1), then ∆|z| 2 = 2(|∇z| 2 + z∆z) since | b| < 2. From (A1) and (4.1), there exists constant c 1 > 0 such that Proof of Theorem 1.1. The proof will be carried out in several steps.
Step 1. Existence of ground state solutions. Applying Lemma 2.10, we deduce that there exists a (C)c-sequence {z n } of Φ such that Φ(z n ) →c ≤ m and Φ (z n ) (1 + z n ) → 0.
It follows from Lemma 3.1 that {z n } is bounded. If In view of the vanishing lemma (see [17,30]), then z n → 0 in L p for any 2 < p < 2 * . Moreover, by (2.3) we get and consequentlỹ which is a contradiction. Thus δ > 0. Going if necessary to a subsequence, we may assume the existence of k n ∈ Z N such that Let us defineũ n (x) = u n (x + k n ) so that (4.5) By virtue of (A5), we have z n = z n and Φ(z n ) →c ≤ m and (1 + z n )Φ (z n ) → 0. (4.6) Passing to a subsequence, we assume thatz n z in E,z n →z in L p loc for 2 ≤ p < 2 * , andz n (x) →z(x) a.e. on R N . Hence it follows from (4.5) and (4.6) thatz = 0 and Φ (z) = 0. This shows thatz ∈ M and Φ(z) ≥ m. On the other hand, By (2.4), (4.6) and Fatou's lemma, we have which implies that Φ(z) ≤ m. Hence Φ(z) = m = inf u∈M Φ andz is a ground state solution of system (1.1).
Step 2. L is compact in H 2 (R N ) (Compactness). It follows from Step 1 that L = ∅. Let {z n } ⊂ L , then z n ∈ M, Φ(z n ) = m and Φ (z n ) = 0. Thus {z n } is a (C) m -sequence. By Lemma 3.1, {z n } is bounded. Hence we can deduce that, passing to a subsequence, there exists z such that z n z in E. It is clear that z ∈ K and z n → z in L q loc for q ∈ [2, 2 * ). We claim that z = 0. Suppose to the contrary that z = 0, then z n → 0 in L q loc and (4.2) implies z n → 0 in C 0 loc . This contradicts |z n | ∞ ≥ > 0 since 0 is an isolated critical point of Φ. Then z = 0 and hence Φ(z) ≥ m . Since there is no nonzero critical value of Φ less than m and z = 0, it is standard to show that Φ(z n − z) → m − Φ(z) and Φ (z n − z) → 0, and hence z n − z → 0 (see [6,Lemma 5.7]). Observe that Since z n and z are solutions of system (1.1), we have We estimate each term appeared in the right hand side of the above equality. By (4.1) and the facts that z n → z in E and the decay of integral of z, there holds Therefore, we get |A(z n − z)| 2 = o(1), which implies that z n → z in H 2 (R N ).
Step 3. Exponential decay of ground state solutions. Since L is compact in H 2 (R N ), then |z(x)| → 0 as |x| → ∞ uniformly in z ∈ L . In fact, if not, then by (4.2) there exist c 0 > 0, z j ∈ L and x j ∈ R N with |x j | → ∞ such that We may assume that z j → z ∈ L in H 2 (R N ) by the compactness of L , then we get which implies a contradiction. Therefore, it follows from (A2) and (4.3) that there exist R > 0 and τ > 0 such that ∆|z| 2 ≥ τ |z| 2 , for |x| ≥ R, z ∈ L .

5.
The asymptotically periodic case. In this section we give the proof of the main result for the asymptotically periodic case. Throughout this section, we always assume that (V2) and (A6) are satisfied. In order to prove the global compactness result, we need to introduce a technical result due to [26,27,41].
Now we estimate each term the above appeared. In fact, since h ∈ L ∞ and q ∈ [2, 2 * ), by Hölder inequality we have Since ε is arbitrary, we know that the first conclusion holds. Below we show that the second conclusion. Indeed, by the same way we can prove that Therefore, The proof is completed.
In order to overcome the lack of compactness, we recall some known facts about the following limit problem where V 0 and f 0 are 1-periodic in x i for i = 1, · · · , N , and satisfy the conditions given in (V2) and (A6). We define the energy functional of limit problem (5.1) and the generalized Nehari manifold M 0 := {z ∈ E\E − : Φ 0 (z)z = 0 and Φ 0 (z)w = 0 for any w ∈ E − }.
By Theorem 1.1, it is easy to see that problem (5.1) has a ground state solution z 0 satisfying Next we use Lemma 5.1 to establish a global compactness result for bounded (C) c -sequences.
Lemma 5.2. Let {z n } be a bounded (C) c -sequences of Φ at level c ≥ 0. Then there exist z ∈ E such that Φ (z) = 0, and there exist a number k ∈ N ∪ {0}, nontrivial critical points z 1 , · · · , z k of Φ 0 and k sequences of points {x i n } ⊂ Z N , 1 ≤ i ≤ k, such that |x i n | → +∞, |x i n − x j n | → +∞, if i = j, i, j = 1, 2, · · · , k, Proof. Let {z n } be a bounded (C) c -sequences of Φ at level c ≥ 0. Then, passing to a subsequence, we may assume that z n z in E, z n → z in L 2 loc , z n (x) → z(x) a.e. on R N . It follows from Lemma 2.4 that Φ (z) = 0. Denote w n = z n − z. Then w n 0 in E, w + n 0 in E + and w − n 0 in E − . Hence we can obtain Indeed, it is clear that the first conclusion holds by Brezis-Lieb lemma. Observe that |z|)) .

(5.4)
On the one hand, since z n → z in L p loc for 2 ≤ p < 2 * , by a standard argument we get On the other hand, using (2.3), the Hölder inequality, the mean value theorem and the decay integral of z, we obtain, for R enough large f (x, |z n + θ n z|)(z n + θ n z) · z − F (x, |z|) = o(1). Moreover, by some similar arguments (see also [6]) we can get In view of the vanishing lemma (see [17,30]), then w n → 0 in L p for any 2 < p < 2 * . Since the orthogonal projection of E on E + and E − is continuous in L p , then w + n → 0 and w − n → 0 in L p for any 2 < p < 2 * . Thus it follows from (2.3) that R N f (x, |w n |)w n · (w + n − w − n ) = o(1). (5.9) Observe that {z n } is bounded (C) c -sequences, then Φ (z n ) = o(1), and it follows from (5.8) that Φ (w n ) = o(1). Using this fact and (2.1), we obtain Therefore, from (5.9), we deduce that w n → 0 in E, and so z n → z in E and k = 0. If {w n } is nonvanishing, then there exist δ > 0, > 1 and {y n } ⊂ Z N such that lim inf n→∞ B(yn, ) |w n | 2 ≥ δ. (5.10) It is clear that {y n } is unbounded. Passing to a subsequence we may assume that |y n | → ∞. Letw n = w n (x+y n ) and note that by (5.10) we find z 1 = 0 such that up to a subsequence,w n z 1 in E,w n → z 1 in L p loc for 2 ≤ p < 2 * andw n (x) → z 1 (x) a.e. on R N . Next we claim that z 1 is a nontrivial critical point of Φ 0 . Indeed, for any ϕ ∈ E and denote ϕ n = ϕ(· − y n ), by Lemma 5.1 and the periodicity of V 0 and f 0 we have f (x, |w n |)w n · ϕ n = (w + n , ϕ + n ) − (w − n , ϕ − n ) + R N V 0 (x)w n · ϕ n − R N f 0 (x, |w n |)w n · ϕ n + o(1) which implies that Φ 0 (z 1 )ϕ = 0 by Lemma 2.4 and z 1 is a nontrivial critical point of Φ 0 . Now denote w 1 n = z n − z − z 1 (· − y n ). Then by direct computation we have w 1+ f (x, |z n |)|z n | 2 − F (x, |z n |) which implies that Φ(z) ≤ m. Hence Φ(z) = m = inf z∈M Φ and z is a ground state solution of system (1.1). Next we claim that z = 0. Indeed, by Theorem 1.1, the limit problem (5.1) has a ground state solution z 0 ∈ M 0 such that Φ 0 (z 0 ) = m 0 . It follows from Lemma 2.11 that there exist t 0 > 0 and w 0 ∈ E − such that t 0 z 0 +w 0 ∈ M, and Φ(t 0 z 0 +w 0 ) ≥ m. Therefore by (V2), (A6)-(i) and Lemma 2.6, we have then by Lemma 5.2 we get k = 0 and z n → z in E, and so z = 0. The proof is completed.
Step 2: L is compact in H 2 (R N ) (Compactness). It follows from Step 1 that L = ∅. Let {z n } ⊂ L , then z n ∈ M, Φ(z n ) = m and Φ (z n ) = 0. Thus {z n } is a (C) m -sequence. By Lemma 3.1, {z n } is bounded. Similar to the proof of Step 1, passing to a subsequence, we can deduce that there exists z such that z n → z in E and z ∈ L . The remaining proof is the similar as the proof of Theorem 1.1, here we omit the details.
Step 3. Exponential decay of ground state solutions. The proof is the same as the proof of Theorem 1.1, here we also omit the details.