GROUND STATES OF NONLINEAR SCHR¨ODINGER SYSTEMS WITH PERIODIC OR NON-PERIODIC POTENTIALS

. In this paper we study a class of weakly coupled Schr¨odinger system arising in several branches of sciences, such as nonlinear optics and Bose- Einstein condensates. Instead of the well known super-quadratic condition that lim | z |→∞ F ( x,z ) | z | 2 = ∞ uniformly in x , we introduce a new local super-quadratic condition that allows the nonlinearity F to be super-quadratic at some x ∈ R N and asymptotically quadratic at other x ∈ R N . Employing some analytical skills and using the variational method, we prove some results about the existence of ground states for the system with periodic or non-periodic potentials. In particular, any nontrivial solutions are continuous and decay to zero exponentially as | x | → ∞ . Our main results extend and improve some recent ones in the literature.

= ∞ uniformly in x, we introduce a new local superquadratic condition that allows the nonlinearity F to be super-quadratic at some x ∈ R N and asymptotically quadratic at other x ∈ R N . Employing some analytical skills and using the variational method, we prove some results about the existence of ground states for the system with periodic or non-periodic potentials. In particular, any nontrivial solutions are continuous and decay to zero exponentially as |x| → ∞. Our main results extend and improve some recent ones in the literature.
gap solitons in photonic crystals [29] and the Hartree-Fock theory for the double condensate [16]. System (1.1) is also important for industrial applications in fiber communications systems [17] and all-optical switching devices [18]. Depending on various assumptions on the potential V i , i = 1, 2 and the nonlinearity F , there are many results of existence and multiplicity solutions to problem (1.1). When V 1 = V 2 and u = v, (1.1) becomes the scalar Schrödinger equation, which has been widely investigated in the literature, see, e.g., [19,34,35,40,42]. For the case of a bounded domain, system like or similar to (1.1) has been studied by many authors, see [11,7,21,25] and the references therein.
Very recently, many authors focused their attention on system (1.1) for the case of unbounded domain or the whole space case. The main difficulty of such type problem is the lack of compactness of the Sobolev embedding. By assuming symmetry property on the potential and working on the radially symmetric function space, one can recover the compactness of embedding, see, e.g., [23,45]. Another usual way to regain the compactness is by imposing coercive assumption on the potential, see, for instance, [3,8,10,35]. The concentration compactness argument is also well employed to deal with the whole space case provided that the potential and nonlinearity are periodic in the variable x, we refer readers to [6,9,26,31] and the references therein. By using the constrained minimization method [5] and the Nehari manifold method [35,24], and applying the bootstrap argument [9] and some skills related to ordinary differential system [23,45], many authors studied system (1.1) for case that the potentials are nonnegative constants and obtained certain of results on the existence, regularity and uniqueness of ground state solutions. Here, a ground state(least energy) solution means a nontrivial solution z 0 = (u, v) ∈ E with minimum energy, which can be formulated as follows, where Φ is the energy functional defined later by (2.7) and E the working space given by (2.4). In [1], Ambrosetti, Cerami and Ruiz considered (1.1) for pure powers nonlinear terms and obtained bound state solutions by using linking together with the barycenter function restricted on the Nehari manifold. In recent paper [33], system (1.1) with non-autonomous and non-homogeneous nonlinearity was studied by Qin, Chen and Tang. Due to the geometrical hypotheses imposed on the potential and nonlinearity, it is not allowed to use the concentration compactness argument or work on the radially symmetric function space. To conquer the difficulties, a Nehari-Pohozaev manifold which is a combination of the usual Nehari manifold and the Pohozaev's identity was introduced there. By applying a linking argument and minimizing the functional Φ on an appropriate subset of the manifold, bound states solutions were obtained there.
When V i , i = 1, 2, are positive and coercive, existence of nontrivial solutions of (1.1) was established by Costa [10] under a condition which was called nonquadratic at infinity or the following classic condition introduced by Ambrosetti and Rabinowitz [2], (AR) there exists a constant µ > 2 such that where the dot denotes the inner product in R 2 , by virtue of which the mountain-pass geometry and the Palais-Smale condition were checked there. For the periodic case, i.e., (V) V 1 , V 2 ∈ C R N are 1-periodic in x j , j = 1, 2, · · · , N , and Chen and Ma [6] studied (1.1) for super-quadratic or asymptotically quadratic nonlinearity F and obtained the existence of least energy solutions by using a generalized weak linking theorem [38]. More precisely, following super-quadratic condition (SQ) introduced by Liu-Wang [22] and technical condition (DL) introduced by Ding-Lee [13] were used there. Later, by employing the non-Nehari manifold method introduced by Tang [41], these results were improved by Qin, He and Tang [31,32] and generalized to the asymptotically periodic case where the potentials V i (x) and the nonlinearity are allowed to be asymptotically periodic in x. See [26] for related results and [8] for nonexistence result.
, and there exist constants r 0 > 0, c 1 > 0 and κ > max{1, N/2} such that Since (SQ) is weaker than (AR), it is commonly used in the literature and it plays a crucial role in establishing the mountain-pass(or linking) geometry and in verifying the boundedness of the Palais-Smale(or Cerami) sequence for the energy functional Φ. Indeed, it is essential to prove the existence of nontrivial solutions for periodic problem like or similar to (1.1) in all literature. Motivated by above works, we continue to study system (1.1) and our purpose in this paper is twofold. First we introduce a local super-quadratic condition which allows the nonlinearity to be super-quadratic at some x ∈ R N and asymptotically quadratic at other x ∈ R N . Therefore it weakens condition (SQ) properly, while it brings some new difficulties in the verification of linking geometry and boundedness of Cerami sequences for Φ. Employing some new techniques and using a technical condition similar to (DL), we obtain the existence and exponential decay estimate of ground state solution for periodic system (1.1). Such an existence result, to the best of our knowledge, is up to date. Even under (SQ), the result partially extends and, in fact, complements the above mentioned results of [6,13,26,31]. Secondly we consider the non-periodic case where the potentials V i and the nonlinearity F are non-periodic with respect to x. Clearly, the concentration compactness argument is no more applicable, moreover, the potential V i considered here is allowed to be sign-changing, which is different from [1,5,10,9,24,33]. Under the local superquadratic condition, we obtain a continuous ground state solution of (1.1) which decays to zero exponentially, the result extends and complements related ones in [10,8]. More precisely, we will prove Theorems 1.1 and 1.2 below by using following local super-quadratic condition instead of (SQ), (S2) there exists a domain Ω ⊂ R N such that lim |z|→∞ F (x,z) |z| 2 = ∞ a.e. x ∈ Ω. Before introducing our technical condition, let us define and Λ 0 > 0 by (V).
Theorem 1.1. Let (V), (S0), (S1), (S2) and (S3) be satisfied. If F (x, z) is 1periodic in each of x 1 , x 2 , · · · , x N , then problem (1.1) has a continuous ground state solution z 0 , moreover, there exist τ, C > 0 such that Next, we consider the non-periodic case and make use of following assumption: where meas(·) denotes the Lebesgue measure in R N (S4)F (x, z) ≥ 0, and there exist constants C 0 , R 0 > 0 and σ ∈ (0, 1) such that the inequality . Now, we are ready to state our main results for the non-periodic case. Remark 1. Condition (V ) has a potential well which is widely used in the literature(see [3,8]), and implies, as a consequence of Molchanov's result [28], that the embedding H : Clearly, potentials V i are allowed to be sign-changing by (V ), thus problem (1.1) considered here is indefinite, while it is strongly indefinite under (V).
Before proceeding to the proof of main results, we give two nonlinear examples to illustrate the assumptions (S2), (S3) and (S4).
Let E = E + ⊕ E 0 ⊕ E − be the working space with the norm defined later by (2.9), as we shall see in Section 2, the functional corresponding to (1.1) is To prove Theorem 1.1 and Theorem 1.2, we have to conquer two main difficulties due to the lack of the usual super-quadratic condition (SQ), which is embodied in checking the linking geometry of Φ and verifying the boundedness of Cerami sequences. More specifically, first we need to find an e ∈ E + \ {(0, 0)} and an r > 0 such that sup Φ(∂Q) ≤ 0, where To achieve the result with reduction to absurdity, it seems to be necessary to prove that This can be deduced from (S1), (SQ) and Fatou's lemma if Ω = R N , since w + se| R N = (0, 0) for any s > 0 and w ∈ E − ⊕ E 0 . However, if we replace (SQ) by (S2), the above equation becomes difficult to verify since it cannot be determined whether w + se| Ω = (0, 0) for any s > 0 and w ∈ E − ⊕ E 0 . Secondly, for any Cerami sequence {z n } ⊂ E satisfying z n / z n w = (0, 0), it follows from (S1), (SQ) and Fatou's lemma that Similarly, we cannot determine whether w| Ω = (0, 0), which makes it difficult to certify the above equation when (SQ) is replaced by (S2). Therefore, some new techniques are looked forward to being introduced to surmount the above mentioned two difficulties, which is the right issue this paper intends to address. We point out that such a local super-quadratic condition can be used to study existence of nontrivial solutions for other indefinite problems, as well as multiple solutions for elliptic problems.
Remaining of this paper is organized as follows. In Section 2, we present the variational setting and give some preliminaries. In Section 3, the linking geometry for the periodic case and non-periodic case is established, respectively. Applying the generalized linking theorem [19,20], we find the Cerami sequences for Φ. Boundedness of Cerami sequences are verified in Section 4, by virtue of which we prove Theorem 1.1 and Theorem 1.2.
Throughout this paper, we use c i and C i (i = 1, 2, . . . ,) to denote different positive constants.
2. Variational setting and preliminaries. We establish a unitary variational framework for the periodic or non-periodic elliptic system similar to or like (1.1). [15,Theorem 4.26]). Let {E i (λ) : −∞ ≤ λ ≤ +∞} and |A i | be the spectral family and the absolute value of A i , respectively, and Then U i commutes with A i , |A i | and |A i | 1/2 , and A i = U i |A i | is the polar decomposition of A i (see [14,Theorem IV 3.3]). Let H i = D(|A i | 1/2 ) and For any u i ∈ H i , fixing i = 1 or i = 2, it is easy to see that 2) For fixed i taking 1 or 2, define an inner product and the corresponding norm where (·, ·) L 2 denotes the inner product of L 2 (R N ), · L s stands for the usual which are orthogonal with respect to both (·, ·) L 2 and (·, ·) Hi . Then and the corresponding norm (1.2) and (2.5) yield that z 2 ≥ Λ 0 z 2 2 for any z ∈ E, where · s stands for the usual L s (R N , R 2 ) norm, 1 ≤ s < ∞. Moreover, by Remark 1.4 we have following lemma.
Proof. Employing a standard argument, one can checks easily the lemma, see, for example, [19] and [46].
3. Linking structure for the periodic or non-periodic system. In this section, we assume (S0) and (S1) are satisfied without mentioning. Without loss of generality, we may assume that Ω ⊂ R N is a bounded domain.
The proof is standard, so we omit it. We first consider the periodic case, i.e. (V) holds. Note that E 0 = ∅ in this case and E − is infinite dimensional, which is the so called strongly indefinite problem. Such type of problems have appeared extensively in the study of differential equations via critical point theory, see, for example, [12,19,40] and the references therein. Choose (υ, ν) this together with (2.10) implies that (υ + 1 , ν + 2 ) = e + = (0, 0). To obtain the linking structure of Φ, we need to establish following lemma.

4.
Boundedness of Cerami sequences. In this section, we show the boundedness of Cerami sequences for the periodic or non-periodic system (1.1) by using some new tricks. Proofs of main results will be given at the end of the section. To find Cerami sequences for the functional Φ, we introduce following generalized linking theorem established in [19]. . Let X be a Hilbert space with X = X − ⊕X + and X − ⊥ X + , and let I ∈ C 1 (X, R) of the form Suppose that the following assumptions are satisfied: (I1) η ∈ C 1 (X, R) is bounded from below and weakly sequentially lower semicontinuous; (I2) η is weakly sequentially continuous; (I3) there exist r > ρ > 0 and e ∈ X + with e = 1 such that Then for some c ≥ α, there exists a sequence {u n } ⊂ X satisfying Applying Lemmas 2.2, 3.1, 4.1 and Lemma 3.2 or Lemma 3.4, we obtain directly following result. In the following two lemmas, we certify the boundedness of Cerami sequences obtained in Lemma 4.2 for the periodic case and non-periodic case, respectively, with the aid of the technical conditions (S3) and (S4).
Cases 1)-2) yield the existence of ground state solution of (1.1). Note that (V ) implies that the point 0 is not contained in σ ess (− + V i ), using the same argument as in the proof of Theorem 1.1 and applying Theorem C.3.3 of [39], we certify the continuity and exponential decay estimate for any nontrivial solution of (1.1), see [30,Theorem 5]