LEAST ENERGY SOLUTIONS FOR NONLINEAR SCHR¨ODINGER EQUATION INVOLVING THE FRACTIONAL LAPLACIAN AND CRITICAL GROWTH

. In this paper, we study a class of nonlinear Schr¨odinger equations involving the fractional Laplacian and the nonlinearity term with critical Sobolev exponent. We assume that the potential of the equations includes a parameter λ . Moreover, the potential behaves like a potential well when the parameter λ is large. Using variational methods, combining Nehari methods, we prove that the equation has a least energy solution which, as the parameter λ large, localizes near the bottom of the potential well. Moreover, if the zero set int V − 1 (0) of V ( x ) includes more than one isolated component, then u λ ( x ) will be trapped around all the isolated components. However, in Laplacian case when s = 1, for λ large, the corresponding least energy solution will be trapped around only one isolated component and will become arbitrary small in other components of int V − 1 (0). This is the essential diﬀerence with the Laplacian problems since the operator ( − ∆) s is nonlocal.

In recent years, much attention has been devoted to the study of the fractional Laplacian. The fractional powers of the Laplacian, which are called fractional Laplacians and correspond to Lévy stable processes, appear in anomalous diffusion phenomena in physics, biology as well as other areas. They occur in flame propagation, chemical reaction in liquids, population dynamics. Lévy diffusion processes have discontinuous sample paths and heavy tails, while Brownian motion has continuous sample paths and exponential decaying tails. These processes have been applied to American options in mathematical finance for modeling the jump processes of the financial derivatives such as futures, forwards, options, and swaps; see [2] and references therein. Moreover, they play important roles in the study of the quasigeostrophic equations in geophysical fluid dynamics.
There are many results which are concerned with the problems involving the fractional Laplacian. Firstly, we refer the readers to the work by Caffarelli and Silvestre [13], in which a new formulation of the fractional Laplacians through Dirichlet-Neumann maps was introduced. By this formulation, they transferred the nonlocal problem to a local problem defined in a higher half space. After their pioneering work, there have been many investigations of the fractional Laplacian problem by using variational methods. For example, using variational methods, Cabré and Tan [11] established the existence of positive solutions for fractional problems in a bounded domain with power-type nonlinearities in the subcritical case.
Recently, the nonlinear nonlocal elliptic equations, which are denoted by have been widely studied. We first introduce the fractional Brezis-Nirenberg problems on bounded domains (−∆) s u = |u| 2 s −2 u + µu in Ω, u = 0 on ∂Ω.
Problem (1.3) was first studied by Tan [26] for s = 1/2, where he obtained the existence of a positive solution for µ > 0 and he also considered the nonexistence of positive solutions to (1.3) for star-shaped domains when µ = 0. After that, Tan [27] also obtained the similar results for the general cases 1/2 < s < 1. For more general nonlinearity N −2s + µu q with s < min{N/2, 1}, µ ∈ R and q ∈ (0, N +2s N −2s ), see also the work by Barrios et al. [3]. We also want to mention the paper by Choi, Kim and Lee [15], where they investigated the asymptotic behavior of solutions to (1.3). For the case Ω = R N of (1.2), Felmer, Quaas and Tan [17] have obtained the existence of positive solutions.
For the following related fractional Schrödinger equations with 0 < s < 1 and V : R N → R is an external potential function, there have been also many investigations; see also [7,8,10,12,14,16,17,18,19,21,22,24,25,28,30,32] and references therein. For the Laplacian, the following analog problems to (1.1) for different kinds of nonlinearities f , have been the main subject of investigation in large amount of works in recent two decades. A lot of papers studied the existence of one-bump or multibump solutions of the problems related to (1.5), where f (x, u) = |u| p−2 u; see [1,5,6] and references therein. Now we are ready to present our main assumptions on V (x) and µ, we firstly assume that: and Ω := int V −1 (0) is nonempty with smooth boundary and Ω = V −1 (0);

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3965
In section 2, we define µ 1 to be the first eigenvalue of A s = (−∆) s in trE 0 . We give the following further assumption on µ: We give some remarks for the operator (−∆) s − µ defined in trE 0 . Note that the operator (−∆) s : trE 0 → (trE 0 ) * is defined by on Ω c × {0} and for every ξ ∈ trE 0 , where ξ is the s-harmonic extension of ξ. We take ξ = u, then Under assumption (V 3 ), for any u ∈ trE 0 , u = 0, it holds that for some δ 0 > 0, which implies that the operator (−∆) s − µ is positively definite in trE 0 . In this paper, we consider the fractional Schrödinger equation (1.1) involving critical growth. We focus on the existence of least energy solutions, which localize near the potential well int V −1 (0) for λ large. For similar investigations involving Laplacian and critical growth, we refer the reader to the second author's paper [29].
Before stating our main results, we firstly give some notations and remarks.
To treat the nonlocal problem (1.1), we will study a corresponding extension problem in one more dimension, which allows us to investigate problem (1.1) by studying a local problem via classical nonlinear variational methods.
The homogeneous fractional Sobolev space D s (R N ) (0 < s < 1) is given by whereû denotes the Fourier transform of u. Note that D s (R N ) is a Hilbert space equipped with an inner product We also define a fractional Laplacian operator on the whole space, where F −1 denotes the inverse Fourier transform. We see for u, v ∈ D s (R N ), and assuming additionally u ∈ D 2s (R N ) v ∈ L 2 (R N ), we can integrate by parts: Finally, the notation H s (R N ) denotes the standard fractional Sobolev space defined as Similarly, it holds by taking trace that ∞}. Now we introduce the concept of s−harmonic extension of a function u ∈ D s (R N ), which provides a way to represent fractional Laplacian operators as a form of Dirichlet-to-Neumann map.
By works of Caffarelli-Silvestre [13] (for R N ), it is known that there is one unique function U ∈ H(t 1−2s , R N +1 respectively in the distributional sense. Moreover, if u is compactly supported and smooth, then the following limits are well defined and one must have then E is the Hilbert space under the inner product is Then by the definition of E, we have We take We can study problem (1.1) by variational methods for a local problem. More precisely, we will study the following boundary value problem in a half space: .

MIAOMIAO NIU AND ZHONGWEI TANG
We define the Nehari manifold and let be the infimum of J λ on the Nehari manifold M λ . For λ large, the problem (1.10) is some kind of limit problem of (1.1). We shall prove that there exists a least energy solution of (1.1) converging, for λ → ∞, to a least energy solution of (1.10). Similarly, to consider the problem (1.10), we will study the following mixed boundary value problem in a half space: . If U satisfies (1.11), then the trace u on R N of the function U is a solution of (1.10).
To consider problem (1.11), we define a subspace E 0 of E as follows: The energy functional associated with (1.11) is defined by Comparing with the Nehari manifold M λ , we define the Nehari manifold and let c 0 := inf{I(U ) : U ∈ N 0 } be the infimum of I on the Nehari manifold N 0 .
Definition 1.1. We say that a function u λ (x) =U λ (x, 0) is a least energy solution of (1.1) if c λ is achieved by some U λ ∈ M λ which is a critical point of J λ . Similarly, we say that a function u(x) = U (x, 0) is a least energy solution of (1.10) if c 0 is achieved by some U ∈ N 0 which is a critical point of I.
Our main results are the following:

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3969
Then for λ large, problem (1.1) has a least energy solution u λ . Furthermore, any sequence λ n (λ n → ∞ as n → ∞) has a subsequence such that u λn converges in H s (R N ) along the subsequence to a least energy solution of (1.10).
As in the case of the least energy solution of (1.1), any solutions of (1.1) converges, for λ → ∞, to a solution of (1.10). More generally, we have the following result.
Then u n (x) = U n (x, 0) converges strongly along the subsequence in H s (R N ) to a solution of (1.10).
Our paper is organized as follows: In Section 2, we present some results about the eigenvalues and eigenfunctions for the operators involving the fractional Laplacian. In Section 3, we give the Mountain Pass Geometry. Section 4 is devoted to the existence of the least energy solution to the limit problem. In Section 5, we prove the existence of the least energy solution. Section 6 contains the asymptotic behavior of the least energy solution and in Section 7 we give the proof of our main results.
We will use the same C to denote various generic positive constants and we will use o(1) to denote quantities that tend to 0 as λ ( or n) tends to ∞.

Eigenvalues and eigenfunctions.
In this section, we present some results about the eigenvalues and eigenfunctions for the operators involving the fractional Laplacian. We consider the following boundary value problems: To consider the above problem, we study the following boundary value problem in a half space: . If U satisfies (2.16), then the trace u on R N of the function U is a solution of (2.15). Firstly, we define on Ω × {0}, and ϕ 1 (x) = U 0 (x, 0) satisfies the following problem: x ∈ R N \ Ω. Moreover, we call µ 1 > 0 the first eigenvalue of (−∆) s in trE 0 and ϕ 1 (x) = U 0 (x, 0) the first eigenfunction corresponding to µ 1 in trE 0 . Now we are going to show that µ 1 can be achieved by some U 0 ∈ M 1 . To show that, we firstly give an imbedding lemma which is standard.
Proof. Note that tr Ω E 0 ⊂ H s (Ω) and the fact that the embedding It is easy to see that U k is bounded in E 0 . Since E 0 is reflexive and tr Ω E 0 is compactly embedded in L p (Ω) by lemma 2.1, we conclude that there exists a subsequence (we still denote it by U k ) and a function U 0 ∈ E 0 such that in Ω.

It follows that
We have U 0 ∈ M 1 and Consequently, U 0 is indeed a minimizer which achieves µ 1 .
Now we consider the eigenvalue problems for the operator We define and U (x, t) is the s-harmonic extension of u(x) with U (x, 0) = u(x). We will show that µ λ 1 is indeed an eigenvalue of the operator L λ . For this we only need to show µ λ 1 is a discrete spectrum of L λ for λ large. Indeed we have the following stronger result.

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3971
Remark 2. In fact, we can choose some ϕ ∈ tr R N E λ which satisfies suppϕ ⊆ Ω and For such fixed ϕ, whereφ(x, t) is the s-harmonic extension of ϕ. Thus the above lemma immediately indicates that µ λ 1 is a discrete spectrum of L λ for λ large and thus is also an eigenvalue of L λ , we denote the corresponding eigenfunction as ψ λ 1 (x). Now we give the proof of Lemma 2.3. A similar proof can be found in the paper by Bartsch, Pankov and Wang [4]. For the reader's convenience, we give the details here.
Proof. We set Thus it is easy to see that where σ(L) denotes the spectrum of an operator L. Let us denote by we only need to show that the operator of multiplication by W − λ is a relatively compact perturbation of H λ . Thus by the classical Weyl theorem ( see [23][XIII.4, p. 117]), we have Here σ ess (L) denotes the essential spectrum of the operator L.
Thus to complete the proof of this lemma, we only need to show that is compact. Indeed we will show that is compact. Thus the boundedness of the following embedding immediately implies that (2.17). We set then by assumptions (V 1 ) and (V 2 ), we have A 0 ⊂ B R (0) for some R > 0. Here B R (0) denotes the ball center at origin with radius R. Thus by the definition of Thus using the fact that H s (R N ) → L 2 (B R (0)) is compact immediately implies that S is precompact in L 2 (R N ) and thus the proof of this Lemma is completed. Now we give an asymptotic behavior result for µ λ 1 and U λ 0 , where U λ 0 is the sharmonic extension of ψ λ 1 (x). The proof follows the similar arguments as that in [29].
Proof. We only need to show that for any sequence {λ n }(λ n → ∞ as n → ∞), µ λn 1 → µ 1 and U λn 0 → U 0 as n → ∞. First of all, by the definition of µ λn 1 , it is easy to see that µ λn It is easy to see that {U λn 0 } is bounded in E and, without loss of generality, we assume that there exists a subsequence (we still denote U λn 0 ) and a function U 0 ∈ E such that U λn Indeed to show this claim, let us set C m =:
To prove this, we only need to show that for any ε > 0, In fact, by the assumption (V 2 ), we can take R > 0 large enough such that where B R (0) is the ball centered at the origin with radius R. Thus Thus for any ε > 0 small, there exists N 0 > 0 such that for all n ≥ N 0 we have Since U λn 0 (·, 0) → U 0 (·, 0) strongly in L 2 (B R (0)), by (2.18) and Claim 1, we have Thus we proved Claim 2.
3. Mountain Pass Geometry. Let X be a Hilbert space and ϕ ∈ C 1 (X, R).
We call a sequence {u n } ⊂ X a (Palais-Smale) c sequence ((P.S.) c sequence for shortness) of ϕ if it satisfies: where X * is the dual space of X.
To obtain a (P.S.) c sequence, we apply the well known Mountain Pass Lemma (see also [31]). More precisely, we show that the functional J λ has a Mountain Pass Geometry for λ large. As a result, J λ has a (P.S.) c sequence for some c ∈ R. Proof. We divide the proof into three steps.
Step 1: It is easy to verify that J λ ∈ C 1 (E λ , R).

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3975
By Lemma 3.1, we obtain that J λ has a Mountain Pass Geometry and thus by Mountain Pass Lemma, J λ possesses a (P.S.) sequence {U n } ⊂ E λ such that 4. Existence of the least energy solution to the limit problem. In this section, we study the existence of the least energy solution of the limit problem (1.10). Namely, the following problem (4.20) We study the following mixed boundary value problem in a half space: To begin with the proof of Proposition 1, we firstly present some notations and lemmas which are the main ingredients of the arguments. We denote S 0 := inf U ∈H(t 1−2s ,R N +1 At first, we have the following Lemma 4.1, the similar proof can be found in the paper by M. Gonzalez, J. Qing [20]. Lemma 4.1. Let µ ∈ (0, µ 1 ). Then we have Proof. It is easy to see that S 0 ≤ S, then it suffices to show that S µ < S 0 . It is known from [21] that S 0 is achieved by the extremal functions where ε > 0 is arbitrary and On the other hand, multiplying U ε on both sides of equation (4.27) and integrating, we get Given any ρ > 0, let B ρ be the ball of radius ρ centered at the origin in R N +1 and B + ρ be the half ball of radius ρ in R N +1 . Choose a smooth cutoff function η, 0 ≤ η ≤ 1 and for small fixed ρ, For ε << ρ, we choose a test function for the functional Q µ (U ), which we recall is given by Step 1. Computation of the energy in B + 2ρ \ B + ρ . At first, we note that on the half-annulus, (4.29)

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3977
One the other hand we have (1) and where we have used the fact (see [20]) that when N > 5 + 2s. Thus from formula (4.29) we may estimate Step 2. Conclusion. By a direct computation, we obtain that where ω N is the surface area of unit sphere in R N . We have Hence we have Since U ε are minimizers for S 0 and (4.28), we have that On the other hand,we have for all ε << ρ where c 1 , c 2 , c 3 and c 5 are positive constants.
Using the above estimates, we have In the case N > 5 + 2s, it holds that .
if we take ε > 0 small enough. Thus the proof of Lemma 4.1 is completed.
According to the definition of S, we know that .

MIAOMIAO NIU AND ZHONGWEI TANG
Now we are ready to give the proof of Proposition 1.
Then we have However, on the other hand This contradiction shows that the least energy solution U (x, y) of (4.21) satisfies U (x, 0) ≥ 0 and U (x, 0) = 0 both in Ω 1 and in Ω 2 . The phenomenon is totally different from the local operator Lapacian since in Laplacian case, u = 0 in Ω immediately indicates that ∆u = 0 in Ω for any domain Ω. For fractional Laplacian case, it is not the case.

5.
Existence of the least energy solution. After the above preliminaries, in this section, we are going to study the existence of the least energy solution of problem (1.1). Firstly, we show that any (P.S.) sequence of J λ is bounded. Proof. Assume {U n } is a (P.S.) sequence of J λ , setting ε n := ∇J λ (U n ) it follows from (5.40) that Thus it is easy to see that there exists a constant C which is independent of λ and n such that U n λ ≤ C.
Now we show the compactness of the functional J λ under certain level set. More precisely, we have the following lemma. where S 0 is the best Sobolev constant. Then there exists a subsequence of {U n } which converges strongly in E λ to a solution U λ of (1.8) such that Proof. By Lemma 5.1, we know that {U n } is bounded in E λ . Then there exists a function U λ ∈ E λ such that up to a subsequence, It is easy to check J λ (U λ ) = 0, J λ (U λ ) ≥ 0.
Let W n = U n − U λ , then Again using Brezis-Lieb's Lemma, we can prove that {W n } is also a (P.S.) sequence of J λ satisfying J λ (W n ) → 0, and lim 0 . Now we prove that W n → 0 strongly in E λ . Indeed since J λ (W n ) → 0, we only need to show that W + n (·, 0) → 0 in L 2 s (R N ). We prove it by a contradiction argument and suppose on the contrary that W + n (·, 0) → 0 in L 2 s (R N ). Without loss of generality, up to a subsequence, we assume that On the other hand, since lim inf Thus we obtain that and the fact that E 0 ⊆ E λ , we know that c λ ≤ c 0 , thus to complete the proof, we only need to show that c 0 < s N S N 2s 0 .

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3983
Let U be the one in the proof of Proposition 1. Then the function S N −2s 4s µ U ∈ N 0 , and thus, Since it has been proved in Lemma 4.1 that S µ < S 0 , we have that c 0 < s N S N 2s 0 .
6. Asymptotic behavior of the least energy solutions. In this section, we study the asymptotic behavior of the least energy solutions of (1.1) as λ tends to infinity. Firstly, we give the asymptotic behavior for c λ as λ → ∞.
Proof. By the definition of c λ and c 0 , it is easy to see that c λ ≤ c 0 for all λ ≥ 0. It is easy to show that c λ is monotone increasing of λ. We suppose on the contrary that there is a sequence {λ n } with λ n → ∞ as n → ∞ such that lim n→∞ c λn = k < c 0 .
Thus k > 0 and we take U n ∈ M λn as the least energy solution of problem (1.8) with λ being replaced by λ n satisfying By a standard argument, it is easy to see that the norms U n λn in E λn is bounded, which implies {U n } is bounded in E. As a consequence, up to a subsequence (we still denote U n ), there exists U ∈ E such that U n U weakly in E, U n (·, 0) U (·, 0) in L 2 s (R N ), U n (·, 0) → U (·, 0) in L 2 loc (R N ), U n (·, 0) → U (·, 0) a.e. in R N .

SOLUTIONS INVOLVING FRACTIONAL LAPLACIAN AND CRITICAL GROWTH 3985
Furthermore, This implies k ≥ c 0 , this is a contradiction. Hence we proved that lim λ→∞ c λ = c 0 and the proof of Lemma 6.1 is completed.
7. Proof of the main results. Now we give the proof of our main results.
Proof of Theorem 1.2. Combining Lemma 5.2 and Lemma 5.3, we have proved that for λ large, (1.8) has a least energy solution U λ corresponding to c λ and hence (1.1) has a least energy solution u λ = U λ (x, 0). For any sequence {λ n }(λ n → ∞ as n → ∞), we denote by U n ∈ E λn the corresponding solution of (1.8) such that J λn (U n ) = c λn . We will show that U n converges ( or along a subsequence when necessary) to a least energy solution U of (1.11) in E. Firstly, it is easy to see that {U n } is bounded in E. Therefore, we may assume that subject to a subsequence, U n U in E, U n (·, 0) U (·, 0) in L 2 s (R N ), U n (·, 0) → U (·, 0) a.e. in R N . As is done in the proof of Lemma 6.1, we can obtain that U (·, 0)| Ω c = 0, which implies that U ∈ E 0 . On the other hand, with a similar argument as in the proof of Lemma 5.2, we also can prove that U n (·, 0) → U (·, 0) strongly in L 2 s (R N ). Then it suffices to show that In both cases, we can get that Then there exists α ∈ (0, 1) such that αU ∈ N 0 , that is Furthermore, This is a contradiction. Then we have proved that the least energy solution U n of (1.8) converges ( or along a subsequence when necessary) to a least energy solution U of (1.11) in E and hence the proof of Theorem 1.2 is completed.
Proof of Theorem 1.3. Suppose u n = U n (x, 0) ∈ H s (R N ) is a solution of (1.1), where λ is replaced by λ n . Similarly, such a sequence U n is also bounded in E, we may assume, going if necessary to a subsequence, that U n U in E and U n (·, 0) U (·, 0) in L 2 s (R N ). As in the proofs of Lemma 6.1 and Lemma 5.2, we can prove that U ∈ E 0 and U + n (·, 0) → U + (·, 0) strongly in L 2 s (R N ). By Brezis-Lieb's Lemma, we have Then the solution U n of (1.8) converges ( or along a subsequence when necessary) to a solution U of (1.11) in E. This implies that u n (x) = U n (x, 0) converges strongly along the subsequence in H s (R N ) to a solution of (1.10). The proof of Theorem 1.3 is completed.