Least energy solutions for coupled hartree system with hardy-littlewood-sobolev critical exponents

In this paper we are interested in the following critical coupled Hartree system \begin{document}$ \left\{\begin{array}{l} (-\Delta)^{s} \tilde{u}+\lambda_{1}\tilde{u} = \alpha_{1}\int_{\Omega}\frac{|\tilde{u}(z)|^{2_{\mu}^{\ast}} }{|x-z|^{\mu}}dz|\tilde{u}|^{2_{\mu}^{\ast}-2}\tilde{u}\\ \quad +\beta\int_{\Omega}\frac{|\tilde{v}(z)|^{2_{\mu}^{\ast}}} {|x-z|^{\mu}}dz|\tilde{u}|^{2_{\mu}^{\ast}-2}\tilde{u}, \ \ &&\mbox{in} \ \Omega,\\ (-\Delta)^{s} \tilde{v}+\lambda_{2}\tilde{v} = \alpha_{2}\int_{\Omega}\frac{|\tilde{v}(z)|^{2_{\mu}^{\ast}} }{|x-z|^{\mu}}dz|\tilde{v}|^{2_{\mu}^{\ast}-2}\tilde{v}\\ \quad +\beta\int_{\Omega}\frac{|\tilde{u}(z)|^{2_{\mu}^{\ast}}} {|x-z|^{\mu}}dz|\tilde{v}|^{2_{\mu}^{\ast}-2}\tilde{v}, \ \ &&\mbox{in} \ \Omega,\\ \tilde{u} = \tilde{v} = 0, \ \ && \mbox{on} \ \partial \Omega, \end{array} \right. $\end{document} where \begin{document}$ 0 , \begin{document}$ \alpha_{1}, \alpha_{2}>0 $\end{document} , \begin{document}$ \beta\neq0 $\end{document} , \begin{document}$ 4s , \begin{document}$ 2_{\mu}^{\ast} = (2N-\mu)/(N-2s) $\end{document} , \begin{document}$ \Omega\subset\mathbb{R}^N(N\geq3) $\end{document} is a smooth bounded domain, \begin{document}$ -\lambda_{1}(\Omega) with \begin{document}$ \lambda_{1}(\Omega) $\end{document} the first eigenvalue of \begin{document}$ (-\Delta)^{s} $\end{document} under the Dirichlet boundary condition. Assume that the nonlinearity and the coupling terms are both of the upper critical growth due to the Hardy–Littlewood–Sobolev inequality, by applying the Dirichlet-to-Neumann map, we are able to obtain the existence of the ground state solution of the critical coupled Hartree system.


1.
Introduction. The following two-component system with Hartree type nonlinearities describes the boson stars in mean-field theory [20,27], which has attracted a great deal of attention in theoretical and numerical astrophysics over the past years.
where Ψ i : R × R 3 → C, i = 1, 2, m is the mass of the particles, is the planck constant, the real positive constant β is the scattering length, W i (x), i = 1, 2 are the external potentials and K(x) is the response function which possesses information on the mutual interaction between the particles. If the response function is a delta function K(x) = δ(x), the nonlinear response is local in fact and has been widely considered in recent years. In this case system (1) appears especially in nonlinear optics [37,38]. Physically, the solution Ψ i denotes the i-th component of the beam in Kerr-like photorefractive media( [2]). System (1) also arises in the Hartree-Fock theory to describe a binary mixture of Bose-Einstein condensates in two different hyperfine states. Set Ψ 1 (x, t) = u(x)e −iE1t and Ψ 2 (x, t) = v(x)e −iE2t , then system (1) can be transformed into a coupled elliptic system given by where 2m . The existence of solitary waves of the coupled Schrödinger system (2) has been investigated recently by many authors , see [5,8,19,24,29,30,31,32,34,36,39,44,47,51]. The critical growth case was studied in [15,16,17,23,42,43]. Among them, Chen and Zou studied the nonlinear Schrödinger system in Ω, u, v ≥ 0, in Ω, where Ω ⊂ R N , µ 1 , µ 2 > 0 and β = 0 is a coupling constant. For the special critical case p = 2 and N = 4, the authors also investigated the existence and properties of least energy solutions in the higher dimensional case 2p = 2N N −2 and N ≥ 5 in [17]. But nonlocal nonlinearities have attracted considerable interest as a means of eliminating collapse and stabilizing multidimensional solitary waves. It appears naturally in optical systems [33] and is known to influence the propagation of electromagnetic waves in plasmas [9]. Non-locality also plays an important role in the theory of Bose-Einstein condensation, where it accounts for the finite-range manybody interaction [18]. In [53] the authors applied the variational methods to study a Schrödinger system with non-local coupling nonlinearities of Hartree type Under suitable assumptions on the potential, the authors proved the existence of purely vector ground state solutions for the Schrödinger system if the parameter ε is small and β is large. In [10] the author considered a class of fractional Schrödinger systems of Choquard type, by using the concentration compactness techniques they were able to prove existence and stability results of the standing waves. The nonlocal Hartree system was also studied in [50], there the authors obtained the existence of standing waves. The aim of the present paper is to consider the following critical coupled Hartree system with Hardy-Littlewood-Sobolev critical exponents that is in Ω, where 0 < s < 1, α 1 , α 2 > 0, β = 0, 4s < µ < N , 2 * µ = (2N − µ)/(N − 2s), Ω ⊂ R N (N ≥ 3) is a smooth bounded domain, −λ 1 (Ω) < λ 1 , λ 2 < 0 with λ 1 (Ω) the first eigenvalue of (−∆) s under the Dirichlet boundary condition.
Before stating the main results, we need to recall some basic results about the fractional Laplacian operator (−∆) s . Consider the nonlocal operator (−∆) s in R N is defined on the Schwartz class of functions g ∈ S through the Fourier transform [(−∆) s g]ξ = (2π|ξ|) 2s g(ξ), (4) or via the Riesz potential (see, for example, [25,48]). There is another way of defining this operator. In the remarkable paper [14] by Caffarelli and Silvestre, the authors introduced the s−harmonic extension technique. Ifũ is a regular function in R N , we say that u = E s (ũ) is its s-harmonic extension to the upper half-space, R N +1 + , if u the solution to the following problem −div(y 1−2s ∇u) = 0, in R N +1 + , u(x, 0) =ũ, on R N = ∂R N +1 As shown in [14], (−∆) s can also be characterized by where κ s = (2 1−2s Γ(1 − s)/Γ(s)). Thus the appropriate function spaces to work with are X 2s (R N +1 + ) andḢ s (R N ), defined as the completion of C ∞ 0 (R N +1 + ) and C ∞ 0 (R N ) respectively, under the norms u 2 X 2s = R N +1 + y 1−2s |∇u(x, y)| 2 dxdy and ũ 2Ḣ s = R N |2πξ| 2s |û(ξ)| 2 dξ. The extension operator is well defined for smooth functions through a Poisson kernel, whose explicit expression is given in [12]. It can also be defined in the spaceḢ s (R N ), and, in fact, u X 2s = C ũ Ḣs , for allũ ∈Ḣ s (R N ), (6) where C = √ κ s (see for [6]). On the other hand, for a function u ∈ X 2s (R N +1 + ) we shall denote its trace on R N as tr R N (u). This trace operator is also well defined and it satisfies tr R N (u) Ḣs ≤ C u X 2s (7)

YU ZHENG, CARLOS A. SANTOS, ZIFEI SHEN AND MINBO YANG
The Sobolev trace embedding implies that the trace also belongs to L 2 s (R N ), where 2 s = 2N N −2s . Thus, for all u ∈ X 2s (R N +1 To treat the nonlocal nonlinearities, we need to recall the Hardy-Littlewood-Sobolev inequality, see for example [28]. Proposition 1 (Hardy-Littlewood-Sobolev inequality). Let t, r > 1 and 0 < µ < N with 1/t + µ/N + 1/r = 2, f ∈ L t (R N ) and h ∈ L r (R N ). There exists a sharp constant C(t, N, µ, r), independent of f, h, such that where | · | q for the In this case there is equality in (9) if and only if f ≡ (const.)h and Denote by D 1,2 (R N +1 + ) the closure of the set of smooth functions compactly supported in R N +1 + with respect to the norm of u D 1,2 (R N +1 From the Hardy-Littlewood-Sobolev inequality, we know where C(N, µ) is defined as in the Proposition 1 and 2 * s = 2N N −2s . Consequently, Thus, we can define the best constant S C by . By [7] we know that S 0 is achieved by the extremal functions where ε > 0 is arbitrary. Follow [21,54], by the Hardy-Littlewood-Sobolev inequality and the definition of S C , we have LEAST ENERGY SOLUTIONS FOR COUPLED HARTREE SYSTEM...

333
is the unique minimizer for S C and satisfies Moreover, Firstly we are interested in the existence of the least energy solution of the problem in the whole space (15) By using the harmonic extension technique, we will study the following boundary value problem in a half space: Thus if (u, v) satisfies (16), then the trace (ũ,ṽ) on R N of the function (u, v) will be a solution of problem (15). Clearly (16) has semi-trivial solutions (α 2s−N 2(N +2s−µ) 1 U ε (x, y), 0) and (0, α 2s−N 2(N +2s−µ) 2 U ε (x, y)). Here, we are only interested in nontrivial solutions of (16).
By using the harmonic extension, we can reformulate (3) as The critical problem with critical exponent and the the nonlocal Hartree equation driven by fractional Laplacian were studied in [7,26,41,45,54]. In [54], the authors studied the existence of least energy solution for the nonlocal critical problem where −λ 1 (Ω) < λ 1 , λ 2 < 0. Consequently, (3) has semi-trivial solutions (ũ α1 , 0) and (0,ũ α2 ). We are concerned with the existence of real nontrivial solutions of (25). Define H := X 2s 0 (C) × X 2s 0 (C) and introduce the energy functional We see that (u, v) is a weak solution of (25) if and only if (u, v) is a critical point of the functional E.

YU ZHENG, CARLOS A. SANTOS, ZIFEI SHEN AND MINBO YANG
We define a Nehari type manifold Then any nontrivial solutions of (25) belong to M. Define the least energy First we will study the symmetric case −λ 1 (Ω) < λ 1 = λ 2 = λ < 0. Recall the Brezis-Nirenberg problem which has a least energy solution w with energy Moreover, for any solution u of (29), we know The existence results for the the symmetric case can be stated in the following theorems.
where w is a least energy solution of the Brezis-Nirenberg problem (29).
We denote constants by C(possibly different in different places). An outline of the paper is as follows: In Section 2 is proved by following some ideas of [17,30]. In Section 3, we prove the idea of Theorem 1.2 come from [47]. In Section 4, we use the implicit function theorem to prove Theorem 1.3. In Section 5, we use Nehari manifold approach and Ekeland variational principle to prove Theorem 1.4 for the case β < 0, and use mountain pass argument to prove Theorem 1.4 for the case β > 0.
2. Proof of Theorem 1.1. We begin this section with the the result about the existence of solution for problem (16).
By the semigroup property of the Riesz potential and the Hölder inequality, we get

YU ZHENG, CARLOS A. SANTOS, ZIFEI SHEN AND MINBO YANG
For any (u, v) ∈ N , by (10) and (39), we have Then there exists two Lagrange multipliers L 1 , L 2 ∈ R such that Testing this equation with (u, 0) and (0, v) respectively, we obtain a contradiction. From this we have L 1 = L 2 = 0 and I (u, v) = 0.

339
Let Then By the Hardy-Littlewood-Sobolev inequality, the Riesz potential defines a linear continuous map Then for R > 0 sufficiently large, the following proof is inspired by [15]. We see that On the other hand, for any (u, v) ∈ N , we see from (11) and β < 0 that , so we can get , similarly, we also have Combining these with (17), then we Now, assume that A is attained by some (u, v) ∈ N and I(u, v) = A. By Lemma 2.5, we get that (u, v) is a nontrivial solution of (16) and β < 0. We have Therefore, it is easy to see that which is a contradiction. This completes the proof.
Proof of (2) in Theorem 1. (16) and By (11) and (39), we have This means Notice that c n , d n are uniformly bounded. Passing to a subsequence, we may assume that c n → c and d n → d as n → ∞. Then by (43) Hence, without loss of generality, we assume that c = 0. If d = 0, then (45) . By (46) and Lemma 2.3 we get 2N −µ C , then by (45)-(47) we see that (k, l) satisfies (38). By Lemma 2.4, we see that is a least energy solution of (16). For any 0 < β < N −2s N −µ+2s max{α 1 , α 2 }, next we can prove (16) has a least energy solution. The following proof works for all β > 0. Therefore, we assume that β > 0 and define A := inf . On the other hand, for any (u, v) ∈ N (R 1 ), we define Let (u n , v n ) ∈ N be a minimizing sequence of A . We may assume that u n , v n ∈ X 2s 0 (B N +1 Rn ) for R n > 0. Then (u n , v n ) ∈ N (R n ) and A = lim N −2s , we may let u i be a least energy solution of Then The follow proof is inspired by [1]. For any m ∈ R, there exists a unique t(m) > 0 such Therefore, as |m| > 0 small enough, due to 1 Recalling w αi in the proof of Theorem 1.1-(1), similarly as Lemma 2.7, we have Theorem 2.8. For any 0 < ε < N −µ+2s N −2s , (50) has a classical least energy solution (u ε , v ε ), and u ε , v ε are both radially symmetric and decreasing.
Completion of the proof (2) in Theorem 1.1, we need to establish a Pohožaev type formula of (48). We adapt the method of Brezis and Kato [11] as in [40] and obtain the regularity of the weak solutions by using the Morse iteration technique, we can ) . We skip the details.
satisfies the Pohožaev type identity:
Thus, by (48), we know that in B N +1 Integrating the above equation over B N +1 1 , we see Form Proposition 6.2 of [21], we get This implies that Consequently, we get Similarly, Then using (48), we have

5.
Proof of Theorem 1.4. In this section, without loss of generality, we assume that −λ 1 (Ω) < λ 1 ≤ λ 2 < 0. Since it is standard to see that B > 0. The Brezis-Nirenberg problem has a least energy solution u αi with energy The next lemma is very important, where we need the assumption λ 1 , λ 2 < 0.
Lemma 5.1. Let β < 0, then Since u α1 ∈ C(C) and u α1 ≡ 0 on ∂Ω, without loss of generality, we can denote that B + ρ = {(x, y)|(x, y) < ρ and y > 0} such that Given any ρ, let B ρ be the ball of radius ρ centered at the origin in R N and B + ρ be the half ball of radius ρ in R N +1 + satisfying B + ρ ⊂ C ∪ Ω. Choose a smooth cutoff function ψ ∈ C ∞ (C), and for small fixed ρ, Thus define v ε = ψU ε ∈ X 2s 0 (C), where U ε is defined in (13) and (14). Next it is known that S C is achieved by the extremal functions U ε . We see that

YU ZHENG, CARLOS A. SANTOS, ZIFEI SHEN AND MINBO YANG
By a similar argument, we can also prove that C . By (41) and (80), we have This completes the proof.
(112) Passing to a subsequence, we may assume that Then by (110) and (111) we have I(w n , σ n ) = N −µ+2s Case 1. u = 0, v = 0. By Lemma 5.2 and (113), we have b 1 = 0 and b 2 = 0, then we can assume that w n = 0 and σ n = 0 for n large. Then by (110) and (111), as n large, we get Then by a similar argument as Lemma 5.1, as n large, there exists (t n w n , m n σ n ) ∈ N . Up to a subsequence, we claim that lim n→+∞ (|t n − 1| + |m n − 1|) = 0. Denote Passing to a subsequence, we may assume that C n,1 → c 1 < +∞, C n,2 → c 2 < +∞ and D n → d < +∞. By (110) and (111) we have This implies that Assume that, up to a subsequence, t n → +∞ as n → ∞, then by (116) t 2·2 * µ n C n,1 − t 2 n B n,1 = m 2·2 * µ n C n,2 − m 2 n B n,2 , we also have m n → +∞. Then a contradiction. Therefore, t n , m n are uniformly bounded. Passing to a subsequence, by (117) we may assume that t n → t 0 ≥ b1 that is, h is increasing. If t 0 < 1, then m . Note that u is a nontrivial solution of (29), we have from (80) that E(u, 0) ≥ B α1 , by (113) we have a contradiction with Lemma 5.1, Therefore, Case 2 is impossible. Case 3. u = 0, v = 0. Since (u, v) ⊂ M, by (113) we have E(u, v) = B, by Lemma 2.5, (u, v) is a solution of (25), Therefore, (u, v) is a least energy solution of (25). This completes the proof.
Proof of Theorems 1.4 for the case β > 0. Assume that β > 0. Since the functional E has a mountain pass structure, by the mountain pass theorem there exists {(u n , v n )} ⊂ H such that lim n→∞ E(u n , v n ) = B, lim n→∞ E (u n , v n ) = 0. It is easy to see that {(u n , v n )} is bounded in H, and so we may assume that (u n , v n ) (u, v) weakly in H. Set w n = u n − u and σ n = v n − v, similarly as in the proof of Theorem 1.4 for the case β < 0, we know that E (u, v) = 0 and (110)-(112) are also hold. Moreover, Case 1. u = 0, v = 0. By (125), we have b 1 + b 2 > 0. Then we may assume that (w n , σ n ) = (0, 0) for n large. By (111), there exists t n > 0 such that (t n u n , t n v n ) ⊂ N and t n → 1 as n → ∞. Then by (60) (71), there exists a Lagrange multiplier γ ∈ R such that E (u, v) − γG (u, v) = 0. Since E (u, v)(u, v) = G(u, v) = 0 and G (u, v)(u, v) = 0, we get γ = 0, then E (u, v) = 0. This means that (u, v) is a least energy solution of (25).