POSITIVE SOLUTIONS OF A NONLINEAR SCHR¨ODINGER SYSTEM WITH NONCONSTANT POTENTIALS

. Existence of a solution of the nonlinear Schr¨odinger system ) , where N = 1 , 2 , 3, and V j ,µ j ,β are continuous functions of x ∈ R N , is proved provided that either V j ,µ j ,β are invariant under the action of a ﬁnite subgroup of O ( N ) or there is no such invariance assumption. In either case the result is obtained both for β small and for β large in terms of V j and µ

1. Introduction and main results. In recent years, much attention has been paid to the following 2-coupled system of nonlinear Schrödinger equations where N = 1, 2, 3, and λ j , µ j , β are constants. This type of system appears when one focuses on standing wave solutions of the time-dependent nonlinear Schödinger system which stems from many physical problems, especially in the Hartree-Fock theory for a double condensate and nonlinear optics. See [2,18,19,20,36] and the references therein for more information on physical background of this system.
A solution of (1) is said to be nontrivial if both of its components are nonzero and is called semitrivial if one component is zero and the other is nonzero. A solution of (1) is said to be positive if both of its components are positive. Note that (1) has infinitely many semitrivial solutions (w n1 , 0) and (0, w n2 ), where w nj are nonzero solutions of the equation Because of physical meaning, only nontrivial solutions of (1) are of interest. The main difficulty in studying nontrivial solutions of (1) lies in distinguishing them from semitrivial solutions. The first attempt is the work of Lin and Wei [24] for small interaction coefficient β. Following the work [24], a large amount of papers have been devoted to the study of existence and multiplicity of nontrivial solutions of (1) in various different regimes of the nonlinear coupling coefficient β. See, for instance, [4,5,9,10,24,34] for existence of a ground state or a bound state nontrivial solution, and [8,14,27,28,30,33,35,37,38] for multiplicity of nontrivial solutions.
In all the papers mentioned above the coefficients in (1) are constants. (1) with nonconstant coefficients has not been much studied, and we are only aware of a few papers in this direction ( [29,32,41]). Recently, in the case where either the potentials V j , µ j and β are periodic or V j are well-shaped and µ j and β are anti-wellshaped, existence of a positive ground state of the nonlinear Schrödinger system where N = 1, 2, 3, was proved in [26] by the authors both for β small and for β large in terms of V j and µ j . In this paper, we continue the study on existence of positive solutions of (3) both for β large and for β small in terms of V j and µ j . In contrast to [26], neither periodicity nor well-shaped structure is imposed on V j , µ j , β, except in Theorem 1.6 where V j are well-shaped and µ j are anti-well-shaped. Assume that V j , µ j , β ∈ C(R N , R). Two cases will be considered in this paper: (i) V j , µ j , β are invariant under the action of a finite subgroup G of O(N ) and (ii) there is no such invariance assumption.
In the first case, a function Q is said to be G-invariant if Q(gx) = Q(x) for all x ∈ R N and g ∈ G. Assume, for any x ∈ S N −1 = x ∈ R N | |x| = 1 , there exists g ∈ G such that gx = x. Set m = min where #{gx | g ∈ G} denotes the cardinal number of the set {gx | g ∈ G}. Let x 0 ∈ S N −1 be such that m = #{gx 0 | g ∈ G} and denote {e 1 , e 2 , · · · , e m } = {gx 0 | g ∈ G}, σ 0 = min i =j |e i − e j | ∈ (0, 2].
The following assumptions will be made use of.
and β ∞ − β(x) ≤ Ce −σ|x| . Taking σ in (H 3 ) smaller if necessary, it can be assumed that σ ∈ (σ 0 , 4). Without loss of generality, it can also be assumed that σ = 2. Let · ∞ be the usual norm in L ∞ (R N ). The first main result in this paper is as follows and is for the case where (3) is G-invariant and β is large.
As a direct consequence of Theorem 1.1, the following corollary follows. Under the assumptions (H 1 ), (H 2 ), and (H 3 ), the single nonlinear Schrödinger equation − ∆u + V j (x)u = µ j (x)u 3 in R N , u ∈ H 1 (R N ) (5) has a G-invariant positive solution (see Theorem 4.1 below), denoted by w jG , for j = 1, 2, such that The following theorem is the second main result which is for the case where (3) is G-invariant and β is small. where It is easy to see that (6) is satisfied if β ∞ is sufficiently small. Moreover, in the special case where V 1 = V 2 , the two numbers S 1G S 2G and S 2G S 1G satisfy Then, since the Hölder inequality implies η 2 , min This means the following is a direct corollary of Theorem 1.3.
Now we consider the second case where the coefficients V j , µ j , β may not have any symmetry. In this case, the following assumption for β large will be used. (H 4 ) Either of the following is satisfied.
The next theorem is the third main result which is for the case where (3) may not have any group invariance and β is large. A more general result Theorem 5.6 is stated at the end of Section 5. Theorem 1.5. Let (H 1 ) and (H 4 ) be satisfied. Assume where τ = 4 if (H 4.1 ) holds and τ = max Denote u − = min{u, 0}. To state the last main result which is for the case where (3) may not have any group invariance and β is small, it will be assumed that (H 5 ) there exists ν > 0 such that, for all x ∈ R N , Note that (H 5 ) implies µ j ≥ µ j∞ for j = 1, 2. Then, under the assumptions (H 1 ) and (H 5 ), (5) has a positive ground state solution ( [16,25,40]), denoted by w j , for j = 1, 2, such that Theorem 1.6. Assume (H 1 ) and (H 5 ) hold. If where η = R N βw 2 1 w 2 2 1/2 , and β ∞ < min{µ 1∞ , µ 2∞ }, then (3) has a positive ground state solution.
The above theorems assert existence of positive solutions of (3) only for β large and for β small. It can not be expected to have existence of positive solutions of (3) for all bounded and positive β, as proved in [9] in the case of constant coefficients.
The assumptions (H 1 ) and (H 3 ) were first used in [6,7] to investigate existence of positive solutions of a single semilinear elliptic equation. These assumptions were used in [1,21] to study existence of positive solutions of a single equation invariant under the action of a finite group. Since we are concerned with system and are interested in solutions with both components being nonzero, new difficulties arise. First of all, compactness of Palais-Smale sequences of the functional I associated with (3) has to be taken into account. Second, one has to compare variant infimum values of I with those of I ∞ associated with the limit system to find Palais-Smale sequences at appropriate levels so that compactness holds, and this turns out to be more difficult than the case of a single equation. This is the case as showed in the proof of Lemma 4.6, where two fractions both having positive terms and negative terms in the numerators as well as in the denominators should be compared. Finally, special attention has to be paid to distinguishing nontrivial solutions from semitrivial solutions.
The solutions obtained in Theorems 1.1, 1.3 and 1.6 are minimizers of the infimum values c G ,c G andc defined in (17), (30) and (45) below, respectively. The assumptions (6) and (9) are from [26] where such an assumption was also used to obtain a positive solution of (3) for β relatively small. Theorem 1.6 generalizes [26, Theorem 1.1] where β was assumed to satisfy β(x) ≥ β ∞ for all x ∈ R N . Once c defined in (39) is not achieved, the solution in Theorem 1.5 is a critical point corresponding to the critical value c 1 defined in (42) and is obtained in a similar way as in [3,11], but the argument will be more involved here due to the character of system. If c is achieved then the solution in Theorem 1.5 is a positive ground state solution.
It is in the same way as below to extend the above results to the more general system where N ≥ 1, 2 < p < 2 * , 2 * = 2N/(N − 2) for N > 2 and 2 * = +∞ for N = 1, 2.
The paper is organized as follows. In section 2, we give a global compactness result, which is applied in the proof of the main results. Sections 3, 4, 5 and 6 are devoted to the proof of Theorems 1.1, 1.3, 1.5 and 1.6 respectively.

2.
A compactness result. Let the Sobolev space H 1 (R N ) be endowed with the two equivalent norms If V j are replaced with 1 then · j become the usual norm · in H 1 (R N ) and (·, ·) * becomes the usual norm (·, ·) in H. It is known that solutions of (3) correspond to critical points of the functional I : H → R defined by Under (H 1 ), the limit system and its corresponding functional play important roles in searching solutions of (3).
Let H −1 (R N ) be the dual space of H 1 (R N ) and H * be the dual space of H. A solution of (3) or (11) is said to be nonzero if at least one of its components is nonzero. Thus nonzero solutions are either nontrivial solutions or semitrivial solutions. The main result in this section is the following theorem. Theorem 2.1. Let (H 1 ) hold and assume {(u n , v n )} ⊂ H to be a (P S) c sequence for I, that is Then, replacing {(u n , v n )} by a subsequence if necessary, there exist a nonnegative integer k, a solution (u 0 , v 0 ) of (3), nonzero solutions (u 1 , v 1 ), · · · , (u k , v k ) of the limit system (11) and k sequences {y j n } ⊂ R N such that, as n → ∞, |y j n | → ∞, |y j n − y j n | → ∞, j = j , In Theorem 2.1 the superscript j in (u j , v j ) means an index, not a power, while it stands for a power in the expression of I(u, v). This will not cause any confusion since the exact meaning of superscripts will be clear from the context. Such a result for a single equation is due to Benci and Cerami [11]. The result stated here is in the form of [40,Theorem 8.4], and for completeness of the paper a proof will be given. For that purpose some lemmas are needed, and the first one is the following lemma which is an analog of the Brézis-Lieb lemma in [13]. Proof. The proof is similar to the argument in [13]. Firstly, Fatou's lemma yields that u ∈ L rp (Ω) and v ∈ L sq (Ω). Given any ε > 0, there exists C ε > 0 such that for all x, y ∈ R. Taking x = b a and y = d c yields, for a, b, c, d ∈ R, ||a + b| r |c + d| s − |a| r |c| s | ≤ ε|a| r |c| s + C ε (|b| r |c| s + |a| r |d| s + |b| r |d| s ) ≤ ε(|a| r |c| s + |a| rp + |c| sq ) + C ε (|b| rp + |d| sq + |b| r |d| s ).
The next lemma is a special case of [40,Lemma 8.1]. Proof for all ϕ ∈ C ∞ 0 (R N ). Since u n → u and v n → v in L 4 loc (R N ), for R given above and for n large, Combining (14) with (15) completes the proof.
Note that Lemmas 2.2-2.4 also hold if some weight function Q with Q ∈ L ∞ (R N ) is appropriately added. We are now ready to prove Theorem 2.1.
Making use of assumption (H 1 ), the Brézis-Lieb lemma and Lemma 2.2 gives

Using (H 1 ) again together with Lemmas 2.3 and 2.4 leads to
where B 1 (y) is the unit ball centered at y ∈ R N . If δ 1 = δ 2 = 0, then the P. L.
In the following, we consider the case where δ 1 > 0 or δ 2 > 0. Assume δ 1 > 0 without loss of generality. Then, passing to a subsequence if necessary, there exists {y 1 n } ⊂ R N such that, for all n, and the weak convergence u 1 The Brézis-Lieb lemma and Lemma 2.2 imply

Moreover, it follows from Lemmas 2.3 and 2.4 that
. Iterating the above procedure, suppose k nonzero solutions (u j , v j ) of (11) and k sequences {y j n } ⊂ R N with |y j n | → ∞, j = 1, 2, · · · , k, have been constructed such that, for (u k+1 n , v k+1 n ) defined by u k+1 where S is the optimal constant defined by Therefore, the iterating process must terminate in some finite, say k, steps. That is, the two numbers To see that |y j n − y j n | → ∞ for j = j , assume for induction that |y j n − y j n | → ∞ holds for all j < j ≤ l and for some l ≤ k − 1. Now, let j < j = l + 1 and assume, without loss of generality, B1(y l+1 n ) |u l+1 n | 2 ≥ δ for some δ > 0. Since it follows that Since u j n (x + y j n ) u j in H 1 (R N ) and |y j n − y i n | → ∞ for i = j + 1, · · · , l, it follows that |y l+1 n − y j n | → ∞. Therefore, |y j n − y j n | → ∞ for all j = j . It remains to prove the last sentence of the conclusions of the theorem. In fact, if u n ≥ 0, v n ≥ 0 a.e. in R N for all n, then the functionals can be used to obtain the conclusion.
3. Proof of Theorem 1.1. Set By the principle of symmetric criticality due to Palais [31], any critical point of the restriction functional I| H G is a critical point of I. Therefore, finding G-invariant solutions of (3) is reduced to seeking critical points of I| H G . Define the Nehari manifolds

Consider the two infimums
and Clearly, c G > 0 and c ∞ > 0. It will be showed that c G < mc ∞ , and for this purpose some lemmas are needed.
Proof. The first inequality is from [23, Lemma 3.3] and the second one follows directly from the first one.
The next lemma is a well known result and its proof can be found, say, in [17,22].
has a unique positive radial solution w ∈ C ∞ (R N ) which satisfies, for some A > 0, Moreover, every positive solution of (18) has the form w(· − y) for some y ∈ R N .
(iii) For any R > 0, there exists C R > 0 and s R > 0 such that for all i, j = 1, 2, · · · , m with i = j and s ≥ s R .
Proof. Recall that w is the unique positive radial solution of (18). Set Since [9]). Thus For simplicity of notations, for s > 0, denote and Then, it is easy to see that and thus c G can be estimated as According to (20) and (21), to conclude it suffices to prove that for s sufficiently large. Then F 1 (s) and F 2 (s) are to be estimated. Setting where the fact that w is a solution of (18) has been used. By Lemma 3.3 (i), where the fact that k 2 1 µ 1∞ + 2k 1 k 2 β ∞ + k 2 2 µ 2∞ = k 1 + k 2 has been used. Using Lemma 3.3 (i) yields To estimate R N U , fix an R > 0, take s R as in Lemma 3.3 (iii) and enlarge it so that B R (se i ) ∩ B R (se j ) = ∅ for i = j and s ≥ s R . Decompose R N into ∪ m n=1 B R (se n ) and R N \ ∪ m n=1 B R (se n ). For the integral on ∪ m n=1 B R (se n ), take K = {w(x) | |x| ≤ R} and use Lemma 3.1 (ii) to see that w(x − se j ).
Using Lemma 3.1 (i) for the integral on R N \ ∪ m n=1 B R (se n ) yields Putting the last two inequalities into (24), it then follows that which combined with Lemma 3.3 (iii) leads to Note that it is in the proof of the last inequality where the assumption m ≥ 2 is used. Then, for s large,
Proof. It is similar to the proof of [40,Theorem 4.3].
Proof of Theorem 1.1. By the Ekeland variational principle, there exist a sequence {λ n } of real numbers and {(u n , v n )} ⊂ N G such that (1)). Then λ n = o(1) and ∇I| H G (u n , v n ) → 0 in H G . Since I is G-invariant and (u n , v n ) ∈ H G , ∇I| H G (u n , v n ) = ∇I(u n , v n ) (see the proof of [40, Theorem 1.28]). Therefore, ∇I(u n , v n ) → 0 in H.
Let (u 0 , v 0 ), (u j , v j ) and {y j n }, j = 1, · · · , k, be as in Theorem 2.1. To show that {(u n , v n )} converges, it is necessary to prove k = 0. Assume by contradiction that k ≥ 1. Then k ≥ m. Indeed, assume, passing to a subsequence if needed, y j n /|y j n | → z j ∈ S N −1 . For any l, since #{gz l | g ∈ G} ≥ m, there exists g 1 , g 2 , · · · , g m ∈ G such that for any R > 0 if n is large enough then, for i = j, B(g i y l n , R) ∩ B(g j y l n , R) = ∅. Using Theorem 2.1 and the notations in its proof gives Letting R → ∞ yields k j=1 u j 2 ≥ m u l 2 , l = 1, 2, · · · , k. Similarly, This implies k ≥ m. Then which contradicts the conclusion of Lemma 3.4. Thus Observing we may assume u 0 ≥ 0 and v 0 ≥ 0. Lemma 3.5 and the principle of symmetric criticality imply that (u 0 , v 0 ) is a nonnegative nonzero solution of (3). It suffices to prove that u 0 = 0 and v 0 = 0. Assume, by contradiction, that v 0 = 0. Then the maximum principle implies u 0 (x) > 0 for all x ∈ R N . Since For t > 0 defined by Then it follows from (28) that which is a contradiction. Therefore v 0 = 0. Similarly, u 0 = 0. The maximum principle implies (u 0 , v 0 ) is a G-invariant positive solution of (3).

4.
Proof of Theorem 1.3. The proof of Theorem 1.3, which is more delicate than the proof of Theorem 1.1, will be given in this section. First, existence of w jG needed in the statement of Theorem 1.3 is proved in the following Theorem 4.1, which is closely related to the results in [1,21]. The proof of Theorem 4.1 will be only sketched since it is similar to the proofs in [1,21]. Let G and σ 0 be as in Section 1.
and there exist C > 0 and σ > σ 0 such that, for all x ∈ R N , Then the equation has a G-invariant positive solution, denoted by w G , such that w G is a minimizer of the minimization problem Obviously, Without loss of generality, assume V ∞ = 1. In the proof of Theorem 4.1, the following infimum will be used: If V , µ and µ ∞ are replaced with V j , µ j and µ j∞ , the corresponding γ G and γ ∞ will be denoted by γ jG and γ j∞ , respectively. The proof of Theorem 4.1 relies on the following lemma. Proof. A modification of the proof of Lemma 3.4 works. In the present case, Recall that W = W (s) = m i=1 w(· − se i ) for s > 0. If the two functions F 1 and F 2 in the proof of Lemma 3.4 are redefined as respectively, then and it suffices to prove that for s sufficiently large. This last inequality can be proved in the same way as in the proof of Lemma 3.4.
Proof of Theorem 4.1. Let {u n } ⊂ M G be a minimizing sequence for γ G . Replacing u n with |u n |, we may assume u n ≥ 0. Using the Ekeland variational principle, it can be assumed that φ (u n ) → 0 in H −1 (R N ). By [40,Theorem 8.4], replacing {u n } by a subsequence if necessary, there exist a nonnegative integer k, a nonnegative solution u 0 of (29), nonzero solutions u 1 , · · · , u k of the limit equation and k sequences {y j n } ⊂ R N with |y j n | → ∞, |y j n − y j n | → ∞, j = j , n → ∞, such that Since u n are G-invariant, if k = 0 then k ≥ m which implies γ G ≥ mγ ∞ , a contradiction to Lemma 4.2. Therefore, u n − u 0 → 0, φ(u 0 ) = γ G , and u 0 is a G-invariant positive solution of (29).
Denote J 1 (u, v) = I (u, v), (u, 0) and J 2 (u, v) = I (u, v), (0, v) . In order to prove Theorem 1.3, definec where N G is the generalized Nehari manifold with two constraints. To prove Theorem 1.3, it is necessary to comparec G with m multiple of the constantc ) . The following two lemmas are quoted from [26] and their proofs will be sketched just for convenience. .
Proof. Let w jG be the solution of (5) as in Section 1. Then (u, v) can be chosen as ( √ s 0 w 1G , √ t 0 w 2G ) for suitable s 0 > 0 and t 0 > 0.
, the definition of S jG in Section 1 and the Hölder inequality imply From these three inequalities it is easy to prove the result.
To prove Theorem 1.3,c G should be compared with γ 1G + γ 2G and mc ∞ . This will be done in the following two lemmas.
Proof. Clearly, the second and the third inequalities are consequences of Lemma 4.2. Using (6), the fact that R N µ j w 4 jG = S 2 jG and the Hölder inequality gives Thus the first inequality in (31) (1), and W 2 j = m w 2 + o(1) as s → ∞ and since β ∞ < min{µ 1∞ , µ 2∞ }, the linear system has a unique solution (l 1 , l 2 ) with each component l j = l j (s) > 0 for s sufficiently large, which is given by Let k 1 , k 2 be as in the proof of Lemma 3.4. Since β ∞ < min{µ 1∞ , µ 2∞ },c ∞ has the expression ( [9]) .
Therefore to prove (32), it suffices to prove for s sufficiently large.

HAIDONG LIU AND ZHAOLI LIU
Recall that Then the Hölder inequality implies that To prove (33), each quantity containing W should be estimated. Rewrite Using (H 1 ) and the fact that Here, o(1) means a quantity which tends to 0 as s → ∞. In the same way, To estimate the term W 2 1 W 2 2 R N βW 4 , rewrite it as Then, using (H 1 ) and the fact that Using (H 1 ) once more yields Inserting (34)-(37) into (33) and rearranging the terms, the left hand side of (33) is written in the form HAIDONG LIU AND ZHAOLI LIU Since β ∞ < min{µ 1∞ , µ 2∞ }, the coefficients of 2E − R N U and of the other five integrals on the right hand side are all positive if s is sufficiently large. Using (H 3 ) yields According to the proof of Lemma 3.4, for a fixed R > 0 there exists C R > 0 such that Then, combining the last two inequalities, Invoking Lemma 3.3, we arrive at for s sufficiently large.
We are now ready to prove Theorem 1.3.
Proof of Theorem 1.3. By the Ekeland variational principle, there exists a sequence {(u n , v n )} ⊂ N G such that We may assume that u n ≥ 0 and v n ≥ 0. Then {(u n , v n )} is bounded in H and there exist {s n }, {t n } ⊂ R such that Testing the left side of (38) with (u n , 0) and (0, v n ) yields By Lemma 4.3, we may assume that, for all n, .

POSITIVE SOLUTIONS OF A NONLINEAR SCHRÖDINGER SYSTEM 1453
According to Lemma 4.4, Therefore, s n → 0 and t n → 0 as n → ∞. From (38) it can be deduced that ∇I(u n , v n ) → 0 as n → ∞. Then, by Theorem 2.1, replacing {(u n , v n )} by a subsequence if necessary, there exist a nonnegative integer k, a solution (u 0 , v 0 ) with u 0 ≥ 0 and v 0 ≥ 0 of (3), nonzero solutions (u 1 , v 1 ), · · · , (u k , v k ) of the limit system (11) and k sequences {y j n } ⊂ R N such that, as n → ∞, If k = 0 then u n − u 0 → 0 and v n − v 0 → 0, and as a consequence of Lemma 4.4, Then the maximum principle implies that (u 0 , v 0 ) is a G-invariant positive solution of (3) and I(u 0 , v 0 ) =c G . Assume by contradiction that k ≥ 1. Then there are four cases.
Case 2. v j = 0 for all j = 1, 2, · · · , k and u l = 0 for some l ≥ 1. The argument is the same as in Case 1 and a contradiction is encountered.

HAIDONG LIU AND ZHAOLI LIU
which is a contradiction to the result of Lemma 4.5.
Case 4. u l = 0 and v l = 0 for some l ≥ 1. Then using (26) and (27) gives which is a contradiction to the result of Lemma 4.6.

5.
Proof of Theorem 1.5. This section is devoted to the proof of Theorem 1.5. where which is a closed subset of the Nehari manifold Note that N + does not have interior in N .
Proof. It is clear that c > 0. To prove c ≤ c ∞ , choose {y n } ⊂ R N such that |y n | → ∞. For any (u, v) ∈ N ∞ , define u n = |u(· − y n )| and v n = |v(· − y n )|. Then for t n > 0 defined by , (t n u n , t n v n ) ∈ N + . Since |y n | → ∞, using (H 1 ) we see that as n → ∞. Thus c ≤ c ∞ . It should be noted that each possibility may happen. To see this, first consider the case where, for any x ∈ R N , β ∞ > max{µ 1∞ , µ 2∞ }, and at least one of the five functions V j , µ j , β is not a constant. Then c ∞ is achieved at some (u, v) ∈ N ∞ and it may be assumed that u(x) > 0 and v(x) > 0 for all x ∈ R N since β ∞ > max{µ 1∞ , µ 2∞ } (see [9]). Let t > 0 be such that (tu, tv) ∈ N + . Then and therefore (i) occurs as showed by Of course (ii) happens if the five functions V j , µ j , β are all constants and β ∞ > max{µ 1∞ , µ 2∞ }. At last, consider the case in which, for any x ∈ R N , β ∞ > max{µ 1∞ , µ 2∞ }, and at least one of the five functions V j , µ j , β is not a constant. In this case, for any (u, v) ∈ N + , let t 1 > 0 be such that (t 1 u, t 1 v) ∈ N ∞ . Then Combining this with Lemma 5.1 yields c = c ∞ . Now suppose, by contradiction, that c is attained at (u, v) ∈ N + . Then c ∞ is achieved at (t 1 u, t 1 v). Since β ∞ > max{µ 1∞ , µ 2∞ }, u(x) > 0 and v(x) > 0 for all x. Then t 1 < 1 and c ∞ = I ∞ (t 1 u, t 1 v) < I(u, v) = c. This yields a contradiction with Lemma 5.1.
From this and Lemma 5.3, the following lemma can be proved as in [11].
Fix R > 0 such that Lemma 5.4 holds and define where The reason why N + is used in the definition of F instead of N is that a (P S) c1 sequence {(u n , v n )} with u n ≥ 0 and v n ≥ 0 has to exist if a positive solution of (3) at the c 1 level is to be found.

HAIDONG LIU AND ZHAOLI LIU
Denote a j = min x∈R N V j (x) and b j = µ j ∞ , j = 1, 2. Then and then Therefore, once a nonzero solution of (3) with energy less than 2c ∞ is found, then it is a nontrivial solution of (3).
Recall that there are three cases stated in Remark 5.2. In Case (i), any minimizing sequence for c is relatively compact according to Theorem 2.1 and thus it is standard to come to the conclusion. In Case (ii), the result is obvious. It remains to consider Case (iii). Since the functions in F take images in N + , in order to obtain a (P S) c1 sequence {(u n , v n )} ⊂ N + , N + should be proved an invariant set for the descending flow ϕ t defined by For (u, v) ∈ N , since I(u, v) = 1 4 (u, v) 2 * , the gradient ∇I| N (u, v) can be expressed as where π 1 , π 2 : N → R defined by are positive C 1 functions. Since, by the maximum principle, for u ≥ 0 and v ≥ 0, it can be seen that N + is an invariant set for the flow ϕ t according to [12] (see also [15]). With N + being invariant for the flow ϕ t , from the definition of c 1 in (42) and the fact that max y∈∂B R (0) I(h(y)) < c 1 , a standard argument leads to existence of {(u n , v n )} ⊂ N + such that Then ∇I(u n , v n ) → 0 in H. Let (u 0 , v 0 ), (u j , v j ) and {y j n }, j = 1, · · · , k, be as in Theorem 2.1. We claim k = 0. Otherwise, since c 1 < 2c ∞ , k = 1 and there holds If (u 0 , v 0 ) = (0, 0), then c 1 ≥ c + c ∞ = 2c ∞ which is a contradiction. If (u 0 , v 0 ) = (0, 0), then, since u n ≥ 0 and v n ≥ 0, u 1 ≥ 0 and v 1 ≥ 0. According to [39], (u 1 , v 1 ) must be, up to translation, one of the three solutions ( √ k 1 w, √ k 2 w), ( 1 √ µ1∞ w, 0), and (0, 1 √ µ2∞ w). Therefore, either c 1 = c ∞ , or and in any case there is a contradiction. Thus k = 0 and (u n , v n ) − (u 0 , v 0 ) → 0. Now, I(u 0 , v 0 ) = c 1 , I (u 0 , v 0 ) = 0, and (u 0 , v 0 ) is a nonnegative nonzero solution of (3). Since I(u 0 , v 0 ) = c 1 < 2c ∞ < γ j , there must be that u 0 = 0 and v 0 = 0. Moreover, the maximum principle implies (u 0 , v 0 ) is a positive solution of (3).
The discussion above in this section has in fact proved the following theorem which is a generalization of Theorem 1.5.
Theorem 5.6. Let (H 1 ) be satisfied. Assume for all x ∈ R N and Then where N is the generalized Nehari manifold with two constraints, J 1 (u, v) = I (u, v), (u, 0) and J 2 (u, v) = I (u, v), (0, v) . The following two lemmas from [26] will be used in the proof of Theorem 1.6.
Proof. The first inequality is proved in the same way as in the proof of Lemma 4.5 and the other two inequalities follow from the fact that γ j ≤ γ j∞ for j = 1, 2.
Lemma 6.4. Under the assumptions of Theorem 1.6, if at least one of the four functions V j , µ j is not a constant, thenc <c ∞ .
Proof of Theorem 1.6. If the four functions V j , µ j are all constants, then (H 5 ) implies that β(x) ≥ β ∞ for all x ∈ R N . Thus (3) has a positive ground state according to [26,Theorem 1.1]. In what follows we always assume that at least one of the four functions V j , µ j is not a constant. The argument in the proof of Theorem 1.3 can be used to find a sequence {(u n , v n )} ⊂ N with u n ≥ 0, v n ≥ 0 such that I(u n , v n ) →c, I (u n , v n ) → 0 in H * .
Then, by Theorem 2.1, replacing {(u n , v n )} by a subsequence if necessary, there exist a nonnegative integer k, a solution (u 0 , v 0 ) with u 0 ≥ 0 and v 0 ≥ 0 of (3), nonzero solutions (u 1 , v 1 ), · · · , (u k , v k ) of the limit system (11) and k sequences {y j n } ⊂ R N such that, as n → ∞, |y j n | → ∞, |y j n − y j n | → ∞, j = j , If k = 0 then u n − u 0 → 0 and v n − v 0 → 0, and as a consequence of Lemma 6.2, Then the maximum principle implies that (u 0 , v 0 ) is a positive solution of (3) and I(u 0 , v 0 ) =c. Assume by contradiction that k ≥ 1. Then there are four cases. Case 2. v j = 0 for all j = 1, 2, · · · , k and u l = 0 for some l ≥ 1. The argument is the same as in Case 1.