MAXIMUM PRINCIPLES FOR A FULLY NONLINEAR NONLOCAL EQUATION ON UNBOUNDED DOMAINS

. In this paper, we study equations involving fully nonlinear nonlocal operators We shall establish a maximum principle for anti-symmetric functions on any half space, and obtain key ingredients for proving the symmetry and monotonicity for positive solutions to the fully nonlinear nonlocal equations. Es- pecially, a Liouville theorem is derived, which will be useful in carrying out the method of moving planes on unbounded domains for a variety of problems with fully nonlinear nonlocal operators.

In order to make sense for the integral, we define L α = u ∈ L 1 loc (R n ) : then it is easy to see that for u ∈ C 1,1 loc ∩ L α , F α (u) is well defined. This kind of operator was introduced by Caffarelli and Silvestre [6].
In the special case when G(·) is an identity map, F α reduces to the fractional Laplacian (−∆) α 2 , 0 < α < 2. When G(t) = |t| p−2 t, α = sp, 0 < s < 1 and 1 < p < ∞, F α becomes fractional p-Laplacian (−∆) s p . The nonlocal nature of these operators make them difficult to study. To circumvent this, Caffarelli and Silvestre [5] introduced the extension method which turns the nonlocal problem involving the fractional Laplacian into a local one in higher dimensions. This method has been applied successfully to study the fractional Laplacian equations and a series of fruitful results have been obtained, we refer to [14] and the references therein. Another way is using the integral equations method, such as the moving planes in integral forms and regularity lifting to investigate equations involving fractional Laplacian by first showing that they are equivalent to the corresponding integral equations (see [11,12,24]). For more articles concerning the method of moving planes for momlocal equations and for integral equations, see [16,21] and references therein.
In 2017, Chen, Li and Li [10] developed a systematic approach to carry out the method of moving planes for problems with fractional Laplace operator. Subsequently, by using this direct method, many authors investigated different equations involving fractional Laplacian, see for instance, Chen and Li [8]; Chen and Wu [13]; Cheng [15] and the references therein.
However, these two effective tools fail to work for the fractional fully nonlinear equations due to the full nonlinearity of the operator, while this kind of operators have been recently used in many applications, including continuum mechanics, population dynamics, and many different non-local diffusion problems, see [1,2] for more applied backgrounds. It is also applied in studying the non-local "Tug-of-War" games [3]. In [9], Chen, Li and Li developed a new method that one can treat with the fully nonlinear nonlocal equations directly. Recently, Wang and Niu [23] studied a fully nonlinear nonlocal system with special nonhomogeneous terms which have u(x) and v(x) simultaneously while u(x) and v(x) have positive coefficients.
Back to equation (1.1), when G(t) = t, α = 2, it is reduced to the well known classical elliptic equation (1. 3) The well known classical Liouville's theorem for harmonic functions states that: If u is bounded from below or from above and ∆u = 0, ∀x ∈ R n , then it must be constant. One of its important applications is in the proof of the Fundamental Theorem of Algebra. It is also a key ingredient in deriving point-wise a priori estimates for solutions to a family of elliptic equations in bounded domains with prescribed boundary values, see [18]. Liouville's theorem can also be used to study geometrical and reaction diffusion problems [20]. and also to derive singularity and decay estimates, see [22].
A similar result has been obtained for s-harmonic functions in [4,7]: Assume that 0 < s < 1. If u is bounded from below or from above and then it must be constant.
In order for the integral convergence, the function u ∈ C 1,1 loc ∩ L 2s with Besides the above mentioned applications, this Liouville theorem has also been used to prove the equivalences between fractional nonlinear equations and the integral equations, thus one can employ integral equations methods, such as method of moving planes in integral forms to study qualitative properties of the solutions for the original fractional equations.
To study fractional harmonic functions, one powerful tool is the Poisson representation [19]: where P r (z, x) is the Poisson kernel: If u is bounded from one side, differentiating under the integral sign and letting r → ∞, we may show that ∂u(x) ∂x i = 0, i = 1, 2, · · · , n, x ∈ R n , and this implies that u is constant. The other effective way in studying fractional harmonic functions in R n is the Fourier transform F by from which one has that F (u)(ξ) = 0 for ξ = 0. So, F (u) consists of a finite combination of the Dirac's delta measure and its derivatives. Hence u is a polynomial. Under further restrictions that u ∈ L 2s and bounded from one side, it must be constant.
As usual, one of the fundamental problems in studying the fractional fully nonlinear equation is the uniqueness of fractional harmonic functions. Since the conventional methods do not work for Eq. (1.1), so far as we know, there has not been any results in this respect.
Motivated by the direct methods introduced in [8, 10,13], in this paper we consider the fully nonlinear nonlocal equation (1.1) and prove a maximum principle for anti-symmetric functions on any half space, which contains key ingredients in proving the symmetry and monotonicity for positive solutions. Especially, we prove that if u is a bounded fractional harmonic function, then it is symmetric about any hyper-plane in R n , and therefore, it must be constant. We summarize it as the following Liouville type theorem:

XIAOMING HE, XIN ZHAO AND WENMING ZOU
n, then one can verify that To study the symmetry of u with respect to a given hyper-plane T , we denote , with x being the reflection of x with respect to plane T . We intend to prove that w(x) ≤ 0 for all x in the half space on one side of the plane. This is actually a maximum principle for anti-asymmetric functions on a half space, in an unbounded region, without assuming that the function vanishes near infinity. Theorem 1.3 (Maximum principle). Let T be any given hyper-plane in R n and Σ be the half space on one side of the plane. Let (1.6) This maximum principle will be powerful in carrying out the method of moving plane on unbounded domains. To illustrate this, we use the well known De Giorgi Conjection as a simple example, which is stated as De Giorgi Conjecture [17]. If u is a solution of equation Then there exists a vector a ∈ R n−1 and a function u 1 : R → R such that Now we consider the fractional fully nonlinear version of (1.7): Then applying Theorem 1.3 we will be able to derive is a solution of the equation (1.8) and verifies that |u(x)| ≤ 1, ∀x ∈ R n , u(x , x n ) → ±1 uniformly in x = (x 1 , · · · , x n−1 ) ∈ R n−1 , as x n → ±∞, (1.
For x ∈ Σ λ , let x λ be the reflection of the point x with respect to plane T λ . Set Then, for sufficiently large λ, (1.9) and (1.10) imply that Now it follows from Theorem 1.3 that Consequently, a standard arguments will lead to (1.11) for all |x n | ≥ M. We remark that inequality (1.12) actually provides a starting point to move the plane T λ in studying the symmetry and monotonicity of solutions for the fully nonlinear fractional equation (1.1). If we we can move the plane all the way down, then we prove that For this purpose, we need to establish a narrow region principle on unbounded domains, without assuming that the function vanishes near infinity, which will be presented in our next paper. Note that precisely in carrying out the method of moving planes on unbounded domains, people usually require that the function u vanishes near infinity, or consider u(x)/g(x) for a proper choice of g(x), so that u(x)/g(x) vanishes near infinity, see for example, [9,23]. Unfortunately, the latter two approaches fail to work on the fully nonlinear equation (1.1). Hence there is a pressing need to develop a method of moving planes for such operators that applies to unbounded domains while only assuming the function be bounded. As illustrated above, our Theorem 1.3 provides a starting point to move the planes in such a situation. The paper is organized as follows. In Section 2 we shall establish the maximum principle and hence prove Theorem 1.3. In Section 3, we will use the maximum principle to derive Theorem 1.1 (Liouville Theorem) and Theorem 1.4.

2.
The maximum principle and its proof. In this section we shall prove the maximum principle for fully nonlinear fractional equations on unbounded domains. We first present a preliminary lemma. Lemma 2.1. Assume that u ∈ C 1,1 loc ∩ L α (R n ) and ψ ∈ C ∞ 0 (R n ), then for all small δ > 0, there holds where C is independent of ε while C δ may dependent on δ.
(2.1) (ii) In the ball B δ (x). In this case, by Taylor expansion, we have The anti-symmetry of ∇v ε (x) · (x − y) for y ∈ B δ (x), and (G 1 ) imply that where C 2 is independent of ε, since for any fixed x, we have Similarly, we have which completes the proof. If the supremum can be attained, say at some point x, then we have To estimate L 1 , we first note that While for the second part in the integral, we have due to the strictly monotonicity of G and the fact that It follows that L 1 > 0.
Then the reflection of the point x about the plane T is We denote by u 0 (x) = u( x), and w(x) = u 0 (x) − u(x). Define the function with a = e such that γ(0) = max R n γ(x) = 1. It is well-known that γ(x) ∈ C ∞ 0 (R n ), and hence F α (γ(x)) ≤ C, ∀ x ∈ R n . Also it is monotone decreasing with respect to |x|.

XIAOMING HE, XIN ZHAO AND WENMING ZOU
and By (G 2 ), we infer that (2.14) Thus, by (2.13), (2.14) we get (2.15) Combining (2.11)-(2.15), we deduce (2. 16) In order to derive a contradiction, we also estimate the upper bound of (2.10). For this aim, using Lemma 2.1, we derive We first choose δ small such that then for such δ, let σ be sufficiently close to 1, hence ε = (1 − σ)A is small such that which contradicts with A > 0. This implies that (1.6) holds true, and completes the proof of the theorem.
3. The proofs of Theorems 1.1 and 1.4. In this section we shall use the maximum principle established in the previous section to prove Theorems 1.1 and 1.4.
Proof of Theorem 1.1. We show that u is symmetric with respect to any hyperplane. To this aim, let x n be any given direction in R n and set be a plane perpendicular to x n -axis. Let be the right region about the plane T λ . For x ∈ Σ λ , let be its reflection about the plane T λ . Denote Under the assumptions of Theorem 1.1, we see that w λ (x) is bounded, and verifies These imply that u(x) is symmetric with respect to plane T λ for any λ ∈ R.
Since the x n -direction can be chosen arbitrarily, (3.1) implies u is radially symmetric about any point, it follows that u(x) ≡ C, which completes the proof of Theorem 1.1.
We next prove Theorem 1.4.
Proof of Theorem 1.4. It is sufficient to prove that w λ (x) = u(x λ ) − u(x) ≤ 0 for sufficiently large λ.

XIAOMING HE, XIN ZHAO AND WENMING ZOU
Similar to the proof in Theorem 1.3, it follows that there exists a point x ∈ B 1 (x 0 ) such that Then we shall be able to estimate Q λ := F α (u λ + εφ λ )(x) − F α (u + εφ)(x) (3.4) at the maximum point x.
Therefore, we derive a contradiction when δ is small and σ is sufficiently close to 1. This completes the proof of Theorem 1.4.