Positive ground state solutions for fractional Laplacian system with one critical exponent and one subcritical exponent

In this paper, we consider the following fractional Laplacian system with one critical exponent and one subcritical exponent \begin{equation*} \begin{cases} (-\Delta)^{s}u+\mu u=|u|^{p-1}u+\lambda v&x\in \ \mathbb{R}^{N}, (-\Delta)^{s}v+\nu v = |v|^{2^{\ast}-2}v+\lambda u&x\in \ \mathbb{R}^{N},\\ \end{cases} \end{equation*} where $(-\Delta)^{s}$ is the fractional Laplacian, $0<s<1,\ N>2s, \ \lambda<\sqrt{\mu\nu },\ 1<p<2^{\ast}-1~ and~\ 2^{\ast}=\frac{2N}{N-2s}$~ is the Sobolev critical exponent. By using the Nehari\ manifold, we show that there exists a $\mu_{0}\in(0,1)$, such that when $0<\mu\leq\mu_{0}$, the system has a positive ground state solution. When $\mu>\mu_{0}$, there exists a $\lambda_{\mu,\nu}\in[\sqrt{(\mu-\mu_{0})\nu},\sqrt{\mu\nu})$ such that if $\lambda>\lambda_{\mu,\nu}$, the system has a positive ground state solution, if $\lambda<\lambda_{\mu,\nu}$, the system has no ground state solution.


Introduction
In the past decades, the Laplacian equation or system has been widely investigated and there are many results about ground state solutions, multiple positive solutions, signchanging solutions, etc(see [30,31,11,13,14,12,15] and references therein).
Compared to the Laplacian problem, the fractional Laplacian problem is non-local and more challenging. Recently, a great attention has been focused on the study of fractional and non-local operators of elliptic type, both for the pure mathematical research and in view of concrete real-world applications(see [4,16,28,8,34,7,36,21] and references therein). This type of operator arises in a quite natural way in many different contexts, such as, the thin obstacle problem, finance, phase transitions, anomalous diffusion, flame propagation and many others(see [2,18,29,37] and references therein).
For the case of fractional Laplacian equation, the existence and nonexistence of solutions has been studied by many researchers. For example (−∆) s u + u = f (u) in R N (1.1) has been studied by many authors under various hypotheses on the nonlinearity f . Such as, Wang and Zhou [38] obtained the existence of a radial sign-changing solution for equation (1.1) by using variational method and Brouwer degree theory. When the nonlinearity f satisfies the general hypotheses introduced by Berestycki and Lions [6], Chang and Wang [10] proved the existence of a radially symmetric ground state solution with the help of the Pohozǎev identity for (1.1). However, in all these works, they only consider the existence and nonexistence solutions, but there are few results about the uniqueness of solution for fractional Laplacian equation. In the remarkable papers [25] [26], for the subcritical case, when f (u) = |u| p−2 u, p ∈ (2, 2 * ), R.L. Frank and E. Lenzmann [25] showed the uniqueness of non-linear ground states solutions to the equation (1.1) for one dimension case and R.L. Frank, E. Lenzmann and L. Silvestre [26] showed the general unique ground state solution to the equation (1.1) for dimension great than one. It is also nature to study the coupled system. For the following fractional Laplacian system, has been investigated by many authors under various hypotheses on the nonlinearity F (u, v) and G(u, v). For example, when F (u, v) = f (u) + λv − u, G(u, v) = g(u) + λu − v, D.F. Lü, S.J. Peng [24] showed that under suitable condition of f, g, it has a vector ground state solution for λ ∈ (0, 1). When F (u, v) = µ 1 |u| 2 * −2 u+ αγ Zhen, J. C. He and H. Y. Xu [40] showed that the existence and nonexistence of ground state solutions under suitable condition of α, β, γ, s, N and Z. Guo, S. Luo and W. Zou [21] showed that under suitable condition of α, β, s, N the system has a positive ground state solution for all γ > 0. When F (u, v) = (|u| 2p + b|u| p−1 |v| p+1 )u− u, G(u, v) = (|v| 2p + b|v| p−1 |u| p+1 )v − ω 2α v, Q. Guo and X. He [20] proved the existence of a least energy solution via Nehari manifold method and showed that if b is large enough, it has a positive least energy solution. Note that in all these works, they only consider subcritical case or critical case. As far as we know, there are few results for the fractional Laplacian system with one subcritical equation and one critical equation. In this paper, we consider the following fractional Laplacian system with one critical exponent and one subcritical exponent on R N . In the case of Laplacian system, the problem has been investigated by Z. Chen, W. Zou in [12].
The system we consider is the following Let D s (R N ) be the Hilbert space defined as the completion of C ∞ 0 (R N ) with the scalar product and norm Let H s (R N ) be the Hilbert space of function in R N endowed with the standard scalar product and norm and let S s be the sharp imbedding constant of D s (R N ) ֒→ L 2 * (R N ) , with the norm given by . The energy functional associated with (1.2) is given by Define the Nehari manifold We say that (u, v) is a nontrivial solution of (1.2) if u = 0, v = 0 and (u, v) solves (1.2). Any nontrivial solution of (1.2) is in M. Due to the fact that if we take ϕ, ψ ∈ C ∞ 0 (R N ) with ϕ, ψ ≡ 0 and supp(ϕ) supp(ψ) = ∅, then there exist t 1 , t 2 > 0 such that (t 1 ϕ, t 2 ψ) ∈ M, so M = ∅. Let Our main result is: (1) If 0 < µ ≤ µ 0 , then the system (1.2) has a positive ground state solution.
We sketch our idea of the proof. It is well known that the Sobolev embedding H s (R N ) ֒→ L p (R N ) are not compact for 2 ≤ p ≤ 2 * . Hence, the associated functional of problem (1.2) does not satisfy the Palais-Smale condition. In order to overcome the lack of compactness, we first set our work space in H s : ϕ is radial}. By properties of symmetric radial decreasing rearrangement, we know C p+1 is achieved by radial functions in H s (R N ) and S s is achieved by radial functions in D s (R N ). By principle of symmetric criticality (Theorem 1.28 in [39]), the solutions for (1.  there are no general results for regularity(of the solutions) higher than the one derived from the Sobolev imbedding). The method we use here is different from the one used in [12], where they use a limiting argument to deal with the problem in Laplacian case and the C 2 regularity of the solutions are needed.
The paper is organized as follows. In section 2, we introduce some preliminaries that will be used to prove Theorem 1.1. In section 3, we prove Theorem 1.1.
For any ǫ > 0, we can take a (u, v) ≡ (0, 0) such that We take a two dimensional space S in H r contain (u, v) and (u 0 , v 0 ) and we choose a large We claim {(u n , v n )} is bounded in H r . For n large enough, we have since λ < √ µν, we can take a small τ > 0 such that λ 2 = (µ − τ )(ν − τ ), then by Sobolev imbedding it is easy to see that t n → 1 as n → ∞. So Consequently, A µ,ν,λ = A µ,ν,λ = A µ,ν,λ . This completes the proof of Lemma 2.2.
Remark 2.1. By properties of symmetric radial decreasing rearrangement, it is easy to show that the value A µ,ν,λ = A µ,ν,λ = A µ,ν,λ defined in H r is the same as the value defined in H. So the ground state solution in H r is also the ground state solution in H.
In order to prove Theorem 1.1, we need the following lemma.
s . Assume by contradiction that Thus, (w µ 0 , 0) is a ground state solution of (1.2). Since λ > 0, if (w µ 0 , 0) is a ground state solution of (1.2), then w µ 0 ≡ 0. This contract with s . (2) In order to prove second part of Lemma 2.6, we divide into two steps.
Thus, it is easily seen that S s is also the sharp constant of s . (2.14) On the other hand, assume 0 < λ ≤ (µ − µ 0 )ν, then µ − λ 2 ν ≥ µ 0 . By the same arguments as Lemma 2.2, we have s .
Combining this fact with (2.12), we see that (i) holds. This completes the proof. Step 1. Prove the existence of ground state solutions for system (1.2).
This contradict with Lemma 2.6.
Step 2. We claim that there exist a positive ground state solution. (|u n | p+1 + |v n | 2 * )dx + R N 2λ|u n ||v n |dx, this implies that there exists t n ∈ (0, 1] such that (t n |u n |, t n |v n |) ∈ M r . Hence, we can choose a minimizing sequence (u n , v n ) = (t n |u n |, t n |v n |) and the weak limit (u, v) is nonnegative. By Strong maximum principle for fractional Laplacian( see, Proposition 2.17 in [37]), we have u and v are both positive.