ON EXISTENCE AND NONEXISTENCE OF POSITIVE SOLUTIONS OF AN ELLIPTIC SYSTEM WITH COUPLED TERMS

. This paper is concerned with the elliptic system (cid:40) where n ≥ 3, p,q > 0 and max { p,q } ≥ 1. We discuss the nonexistence of positive solutions in subcritical case and stable solutions in supercritical case, the necessary and suﬃcient conditions of classiﬁcation in the critical case, and the Joseph-Lundgren-type condition for existence of local stable solutions.

1. Introduction. In this paper, we study the following elliptic system with coupled terms − u = (q + 1)u q v p+1 , u > 0 in R n , − v = (p + 1)v p u q+1 , v > 0 in R n , (1.1) where n ≥ 3, p, q > 0 and max{p, q} ≥ 1. This elliptic system and the corresponding parabolic problem appear in the study of static Schrödinger theory and Bose Einstein condensate with two components. They can also be used to describe competition of biological population. In 1996, Bidaut-Véron and Raoux [1] investigated comprehensively the mathematical theory. In addition, such an elliptic system is a representative example in the class of systems with homogeneous right hand side terms because of its variational structure. The corresponding functional is As a consequence of Theorem 1.4(i) in [21], it follows that any classical solution (u, v) of (1.3) satisfies u ≥ v or u ≤ v. In the special case of w 1 = w 2 = w, the system is reduced to the following Lane-Emden equation (1.4) Eq. (1.4) is an important equation in the study of conformal geometry. It has been the object of a huge literature and has been studied extensively. Some foundational work has been done by Lane, Emden, Fowler, Serrin, and Chandrasekar from 1898 to 1967 (cf. [22] and the references therein). In 1981, Serrin, Ni proved that (1.4) has no solution when p + q + 1 ≤ n n−2 (cf. [18,19]). Gidas, Spruck [14] pointed out that (1.4) has no classical solution when n n−2 < p + q + 1 < n+2 n−2 . In critical case p+q +1 = n+2 n−2 , all classical solutions of (1.4) are given by (cf. [2,13,5]) where c, t > 0, x * ∈ R n . The case of p + q + 1 > n+2 n−2 is not completely understood. All radial solutions w(x) of (1.4) can be showed by (cf. [14,23]) where µ = w Not all classical solutions are radial. In fact, (1.5) solves (1.4) in R n+1 . Now it is a cylindrical solution and does not decay along some direction.
Let w be a classical solution of (1.4). It is called a stable solution of (1.4), if for If there exists a compact subset K such that the above inequality holds for any ϕ ∈ C ∞ 0 (R n \ K), then w is called a finite Morse index solution. Introduce a Joseph-Lundgren exponent (cf. [15]) We introduce an integral system which is helpful to study PDE system (1.1).
In this paper, we study existence (nonexistence) of positive solutions of (1.1) and (1.7). Here, the Serrin exponent n n−2 , the Sobolev exponent n+2 n−2 and the Joseph-Lundgren exponent come into play.
As a corollary of Proposition 1.1 and Theorem 1.2, we know that (1.1) has no positive solution if 1 < p + q + 1 ≤ n n−2 . In fact, Theorem 1.2 in [21] shows that u ≡ v and hence (1.1) is reduced to (1.4) in the case of 1 < p + q + 1 ≤ n n−2 . By the nonexistence result of the Lane-Emden equation (cf. [18]), we also see this Liouville theorem of (1.1).
Finally, we consider a problem on the bounded domain where Ω ⊂ R n is a bounded domain. By using the Liouville theorem of (1.1), we will estimate boundary blowing-up rate. To obtain the estimate, we quote the following doubling lemma in [20] by Polacik, Quittner and Souplet. Based on this lemma, we first establish the equivalence between the Liouville theorem of (1.1) and the estimate of boundary blowing-up rate for solutions of (1.8) (cf. Theorem 2.1). Combining with the nonexistence of (1.1) we can obtain that if Here (u, v) solves (1.8).
Here, we will search for other equivalent conditions of the classification result.
Then the following items are equivalent (i) u, v ∈ L ∞ loc (R n ) and p + q + 1 = n+2 n−2 ; (ii) u(x) ≡ v(x) and they are both given by the radial form (1.5); According to Proposition 1.1, the result above holds for (1.3). And hence for (1.1), the corresponding result is still true.
1.3. Supercritical case: stable solutions. (1.10) If there exists some compact subset K of R n such that (1.10) holds for any ψ ∈ C ∞ 0 (R n \ K), (u, v) is called a pair of local u-stable solution. If (1.10) is replaced by for any ψ ∈ C ∞ 0 (R n ) and ψ ∈ C ∞ 0 (R n \ K) respectively, we call (u, v) v-stable solution and local v-stable solution correspondingly.
The definition of these stable solutions is a little 'partial'. In fact, E(u, v) is a functional of two-variables. Stability often comes from nonnegative definite of hessian matrix of two order Frechet derivatives. However, this nonnegative definite property is too strong to calculate easily those 'total' derivatives. Here, Definition 1.8 can be employed to deduce a sufficient condition analogous as the Joseph-Lundgren type, because (1.10) is just like to calculate the 'partial' derivatives of E(u, v). Other definitions of stable solutions on the semilinear elliptic system can be found in [4] and [11]. Lemma 1.9. Let u,v be the positive solutions of (1.1) in R n and set Then both lim r→∞ u(r) and lim r→∞ v(r) exist. We denote them by a and b respectively. Moreover, ab = 0.
Remark 1.12. (i) In view of Lemma 1.9, it is natural to suppose a = 0 or b = 0 in Theorems 1.10 and 1.11. If a = b = 0, then Lemma 3.1 in [21] shows (1.11) It has been studied by Farina [12].
(ii) Since the local stable solution is not necessarily stable, it is not verified are critical for nonexistence of stable solutions and local stable solutions. When p + q + 1 > p jl , we can find a stable solution (cf. Remark 4.2). Thus, there is a gap between ) for the existence of stable solutions.

Liouville theorems in subcritical case.
Proof of Theorem 1.2. The idea comes from [3] and [17]. Without loss of generality, suppose p ≥ q. Assume that (u, v) is a pair of solution of the integral system (1.7). Then, Take the power p for (2.2) and multiply by u q+1 . Integrating on B R , we get

YAYUN LI AND YUTIAN LEI
When p + q + 1 < n n−2 , letting R → ∞ and noting p ≥ 1, we can obtain a contradiction. When Take the power q for (2.1) and multiply by v p+1 . Integrating on (2.8) By taking the limit as R → ∞ in (2.8) and using (2.6), we obtain u ≡ v ≡ 0 at last. Theorem 1.2 is proved.
Proof of Proposition 1.4. By an analogous derivation of (1.3), we can assume Therefore, integrating by parts yields dy. Differentiate both sides with respect to λ and let λ = 1. Then, Similarly, we have Multiplying (2.12) by u q v p+1 and integrating, we obtain (2.13) Using the Fubini theorem and (1.7), we compute that the right hand side of (2.13) Integrating by parts, and using (2.9), we deduce that Combining with (2.13), we obtain Similarly, from (2.11) we also get Summing (2.14) and (2.15) together implies that Inserting (2.10) into this result, we see Therefore, p + q + 1 = n+2 n−2 is derived finally. Proposition 1.4 is complete.
Proof of Corollary 1.5. As a corollary of Proposition 1.1 and Theorem 1.2, we know that (1.1) has no positive solution if p + q + 1 ≤ n n−2 . In fact, Theorem 1.2 in [21] shows that u ≡ v and (1.1) is reduced to (1.4) in the case of p + q + 1 ≤ n n−2 . By the nonexistence result of the Lane-Emden equation (cf. [18]), we also see this Liouville theorem of (1.1).
When n n−2 < p + q + 1 < n+2 n−2 , (1.1) maybe has a pair of solution (u, v) which satisfies that one decays to zero and the other converges to a positive constant. This is different from the Liouville theorem of (1.4).
Next, we consider the problem on bounded domain where Ω ⊂ R n is a bounded domain. We will verify (1.9) in the case of 1 < p+q+1 ≤ Proof. We claim that, if (1.1) does not admit any bounded solution in R n . Then there exists C = C(n, p, q) > 0 (independent of Ω and (u, v)) such that any solution (u, v) of (2.18) satisfies (2.19). Assume that the above conclusion fails. Then, there exists sequences Ω k , (u k , v k ), y k ∈ Ω k , such that (u k , v k ) solves (1.1) on Ω k , and satisfies M k (y k ) > 2k dist −1 (y k , ∂Ω k ). By the Doubling Lemma (Lemma 1.6), it follows that there exists x k ∈ Ω k such that (2.20) Clearly, u k , v k are also solutions of system (1.1) for |y| ≤ k, and they also satisfy By using elliptic L q estimates and standard imbedding theorem, we deduce that some subsequence of ( u k , v k ) converges in C 1 loc (R n ) to a classical solution ( u, v) of (1.1) in R n . Moreover [ u p+q 2 + v p+q 2 ](0) = 1 by (2.21), hence ( u, v) is nontrivial, and moreover, u, v are bounded due to (2.22). This contradicts the assumption of Theorem 2.1.
On the contrary, if the classical positive solution (u, v) of (1.1) satisfies the estimate (2.19), by letting Ω → R n , we deduce u = v = 0 in R n . This shows the nonexistence of positive solution.
By the same way of argument above, we can also obtain v(x) ≤ C|x| − 2 p+q . More asymptotic results about the exterior domain problem can be seen in [1].
3. Critical case: equivalent conditions of classification. In this section, we will give several necessary and sufficient conditions of the classification of positive solutions of integral system (1.7).
Similar to the derivation of (1.3), we can obtain another corresponding system of (1.7). Without loss of generality, in this section we always omit those constants C 1 , C 2 in (1.7). Proof. It is a direct corollary of Theorem 5 in [6]. Proof. Clearly, this radial function w is bounded and w(x) ≤ c|x| 2−n as |x| > R for some R > 0. Therefore, by p + q + 1 > n n−2 , there holds Lemma 3.2 is proved. (1.7), then u, v ∈ L t (R n ) for all t > n n−2 . On the other hand, u, v ∈ L t (R n ) for all t ≤ n n−2 .
Proof. Write w = u + v. Then w ∈ L n(p+q) 2 (R n ). From (1.7), it follows that w satisfies Therefore, w solves the operator equation f = T f + F. By the Hardy-Littlewood-Sobolev inequality, we get Thus, T is a contraction map from L s (R n ) to itself for all s > n n−2 . In view of n(p+q) 2 > n n−2 (Otherwise, (1.7) has no solution, which is implied by Theorem 1.2), T is also a contraction map from L n(p+q)/2 (R n ) to itself. By the regularity lifting lemma of Theorem 3.3.1 in [7], we obtain w ∈ L s (R n ) for s > n n−2 . Thus, Proof.
Step 1. We claim that u, v are bounded. For r > 0, write In view of (3.2) and p + q + 1 > n n−2 , we obtain w ∈ L p+q+1 (R n ). Thus K 2 is bounded. On the other hand, by the Hölder inequality and Lemma 3.3, we can see that K 1 is also bounded.
When R → ∞, L 2 := (R n \B R (0))\B(x, |x| 2 ) u q (y)v p+1 (y) |x| n−2 |x−y| n−2 dy → 0. Here, we notice |x − y| ≥ |x| 2 as y ∈ (R n \ B R (0)) \ B(x, |x| 2 ). To estimate L 3 := B(x, |x| 2 ) u q (y)v p+1 (y) |x| n−2 |x−y| n−2 dy, we should observe that u, v are radially symmetric and decreasing about some x * ∈ R n . In fact, the argument in section 2 of [24] still works here via the method of moving planes in integral forms established by Chen-Li-Ou [9]. Thus, we can view the maximum point x * as the origin since |x| is sufficiently large. Write r = |x|, and define u(r) = u(x), v(r) = v(x). Thus, On the other hand, the integrability result (3.2) shows that u, v ∈ L t (R n ) with Here > 0 is sufficiently small. This integrability result, together with the decreasing property of u, implies u t (y)dy ≤ C.
Combining all the estimates for L 1 , L 2 and L 3 , we complete the proof.
From now on, we shall use the Pohozaev identity to prove (iv) =⇒ (i).
By an analogous argument of sections 5 and 6 in [16], we can obtain that the positive solutions u, v of (1.7) belong to C 1 (R n ) as long as u, v ∈ L ∞ (R n ).

Supercritical case: stable solutions and Joseph-Lundgren condition.
This section is devoted to the proofs of Lemma 1.9, Theorem 1.10 and Theorem 1.11. For convenience, we denote p + q + 1 by µ.
Proof of Lemma 1.9. Here we only discuss the function u(x). The corresponding result about v(x) can be obtained analogously.
Integrating from 0 to r, we obtain u (r) < 0. It means that u is a monotone decreasing function, and hence lim r→∞ u(r) exists.
(ii) Assume w = min{u, v} ≥ min{a, b} > 0, then we have − u ≥ (w) µ > C 0 for some positive constant C 0 . Similar to the argument in (i), we get r 1−n (r n−1 u ) < −C 0 . Integrating twice, we have Thus, there must be a suitable r > 0, such that the right side of the above inequality equals 0. Therefore, u(r) < 0. This is a contradiction. So there must be a = 0 or b = 0.
Hereafter, we only discuss existence/nonexistence of the u-stable solutions and the local u-stable solutions. Thus, we just need a = 0 and q > 1 in the proofs of Theorems 1.10 and 1.11. The argument of the v-stable solutions and the local v-stable solutions is analogous.
Before verifying Theorem 1.10, we need the following result.
Step 1. We claim that for any ϕ ∈ C ∞ 0 (R n ), there holds In fact, multiplying both sides of the first equation in (1.1) by u γ ϕ 2 and integrating by parts, we get The left hand side is equal to Identity (4.3) then follows by combining two results above.
Remark 4.2. When p + q + 1 > p jl , by the analogous argument above, we see that (u α , v α ) is also a u-stable solution by using (4.15).