A direct method of moving planes for a fully nonlinear nonlocal system

In this paper we consider the system involving fully nonlinear nonlocal operators: \begin{document}$ \left\{\begin{array}{ll}{\mathcal F}_{α}(u(x)) = C_{n,α} PV ∈t_{\mathbb{R}^n} \frac{F(u(x)-u(y))}{|x-y|^{n+α}} dy=v^p(x)+k_1(x)u^r(x),\\{\mathcal G}_{β}(v(x)) = C_{n,β} PV ∈t_{\mathbb{R}^n} \frac{G(v(x)-v(y))}{|x-y|^{n+β}} dy=u^q(x)+k_2(x)v^s(x),\end{array}\right.$ \end{document} where \begin{document}$0 \begin{document}$p, q, r, s>1, $\end{document} \begin{document}$k_1(x), k_2(x)\geq0.$\end{document} A narrow region principle and a decay at infinity are established for carrying on the method of moving planes. Then we prove the radial symmetry and monotonicity for positive solutions to the nonlinear system in the whole space. Furthermore non-existence of positive solutions to the system on a half space is derived.

1. Introduction. In this paper, we consider the nonlinear system involving fully nonlinear nonlocal operators: F α (u(x)) = v p (x) + k 1 (x)u r (x), G β (v(x)) = u q (x) + k 2 (x)v s (x), with F α (u(x)) = C n,α P V R n F (u(x) − u(y)) |x − y| n+α dy, and G β (u(x)) = C n,β P V R n G(u(x) − u(y)) |x − y| n+β dy, where P V stands for the Cauchy principal value, F and G are at least local Lipschitz continuous, F (0) = 0, G(0) = 0, 0 < α, β < 2. For some constant M > 0, ∀x ∈ R n , we have 0 ≤ k i (x) ≤ M and k i (x) = k i (|x|), where |x| = (x 2 1 + · · · + x 2 n ) 1 2 . If x, y ∈ R n with |x| ≤ |y|, then k i (x) ≥ k i (y), i = 1, 2. The operator F α or G β was introduced by Caffarelli and Silvestre in [6]. Here, we require In the special case that F (·) is an identity map, F α becomes the usual fractional Laplacian (−∆) α 2 . The nonlocal nature of fractional operators brings many new difficulties comparing with the Laplacian. To treat fractional operators, Caffarelli and Silvestre [5] raised the extension method which turns the nonlocal problem involving the fractional Laplacian into a local one in higher dimensions. This method has been applied successfully to deal with equations involving the fractional Laplacian and a series of fruitful results has been obtained (see [1,12], etc.). One can also use the integral equations method, such as the method of moving planes in integral forms (see [3,4,26,19,18]) and regularity lifting to investigate equations involving fractional Laplacian by showing that they are equivalent to corresponding integral equations (see [11,9,10] and the references therein). For more articles concerning the method of moving planes for nonlocal equations and integral equations, see [13,15,16,19,20,21,22,23,24] and the references therein.
For the fully nonlinear nonlocal equations, so far as we know, there is neither any corresponding extension method nor equivalent integral equations that one can work at. A probable reason is that very few results were obtained for the fully nonlinear nonlocal operator. In [8], Chen, Li and Li developed a new method that can probe directly these nonlocal operators. Inspired by the idea, we extend the method in [8] to fully nonlinear nonlocal systems and consider two nonlinear systems involving fully nonlinear nonlocal operators and (1. 2) The narrow region principle and decay at infinity for the systems are established, which play important roles in carrying out the method of moving planes. To state them, for λ ∈ R, denote by the moving plane, by Σ λ = {x ∈ R n |x 1 < λ} the left region of the plane T λ , byΣ λ the reflection of Σ λ about T λ , by the reflection of x about T λ , and denote . For the simplicity of notations, we stand for U λ (x) by U (x) and V λ (x) by V (x) in the sequel. Throughout this and next section, we assume (1.3) Theorem 1.1 (Narrow Region Principle for system).
Let Ω be a bounded narrow region in Σ λ contained in {x|λ − l < x 1 < λ} with small l > 0. Suppose that U (x) ∈ L α ∩ C 1,1 loc (Ω) and V (x) ∈ L β ∩ C 1,1 loc (Ω) are lower semi-continuous onΩ, and satisfy , i, j = 1, 2, are bounded from below in Ω, c 12 (x) and c 21 (x) < 0, then we have for sufficiently small l, if Ω is unbounded, the conclusion (1.5) still holds under the conditions We call (iii) the strong maximum principle later. As we can see from the proof, to conclude (1.7), Ω does not need to be narrow.
Based on Theorems 1.1 and 1.2, we apply the method of moving planes to obtain symmetry and monotonicity of positive solutions to (1.1) in R n , as well as nonexistence of positive solutions to (1.2) on the half space.
(1.12) Then u(x) and v(x) must be radially symmetric and monotone decreasing about the origin.
In Section 2, we prove Theorems 1.1 and 1.2. Section 3 is devoted to the proofs of Theorems 1.3 and 1.4 by using the previous results and the method of moving planes.
2. Proofs of Theorems 1.1 and 1.2. We let and use c and C for general various positive constants that are usually different in different contexts.
Proof of Theorem 1.1. Suppose that (1.5) does not hold, without loss of generality, we assume U (x) < 0 at some point in Ω; then the lower semi-continuity of U (x) on Ω implies that there existsx ∈ Ω such that andx is in the interior of Ω from the condition (1.4). By the expression of F α and (2.2) in [25], we have (2.1) Using it into (2.1), it shows Together with (1.4), we have for l sufficiently small, Similarly to (2.3), we can derive that This contradiction shows that (1.5) must be true.
Hence U (x) must be identically 0 in Σ λ . Since it shows that U (x) ≡ 0, x ∈ R n . Again from the first equation of (1.4), we know that Since we already know that Similarly, one can show that if V (x) attains 0 at one point in Σ λ , then both U (x) and V (x) are identically 0 in R n . This completes the proof.
3. Symmetry of solutions in the whole space R n . Let us prove Theorem 1.3.
Proof of Theorem 1.3. Choose an arbitrary direction as the x 1 -axis. Let Step 1. Start moving the plane T λ from −∞ to the right in the x 1 -direction. We will show that for λ sufficiently negative, To prove (3.1), for the fixed λ and x ∈ Σ λ , by (1.11), u(x) → 0, as |x| → +∞.
Since |x λ | → +∞, as |x| → +∞, it follows Similarly, one can show that for x ∈ Σ λ , Since λ sufficiently negative and the properties of k i (x), it is easy to see by the mean value theorem that

4)
and and v(x); η λ (x) is between u λ (x) and u(x). By Theorem 1.2, it suffices to check the decay rate at the points where V λ (x) and U λ (x) are negative respectively. In fact, since At those points for |x| sufficiently large, the decay assumptions (1.11) and (1.12) instantly yields that Consequently, there exists R 0 > 0, such that ifx andx are negative minima of U λ (x) and V λ (x) in Σ λ respectively, then it holds by Theorem 1.2 that |x| ≤ R 0 or |x| ≤ R 0 . (3.5) Without loss of generality, we may assume For λ sufficiently negative, combining (3.2) with fact that we know that if U λ (x) < 0 in Σ λ , then U λ (x) must have a negative minimum in Σ λ . This contradicts (3.6). Hence we have for λ sufficiently negative, and then we have from (2.10) that This is a contradiction with (3.8) and then V λ (x) cannot attain its negative value in Σ λ . It follows that (3.1) must be true.
Step 2. Keep moving the planes to the right to the limiting position T λ0 as long as (3.1) holds. Let We show that λ 0 = 0, (3.9) and Suppose that λ 0 < 0, we will prove that the plane T λ can be moved to the right a little more and (3.1) is still valid. More rigorously, there exists a small > 0, such that for any λ ∈ (λ 0 , λ 0 + ) we have This is a contradiction with the definition of λ 0 . Hence we must have (3.9). The remaining task is to prove (3.10) by using Theorem 1.1 and Theorem 1.2.
Since U λ0 (x) ≥ 0 and V λ0 (x) ≥ 0 but U λ0 (x) ≡ 0 and V λ0 (x) ≡ 0, from that nonnegative functions U λ0 (x) or V λ0 (x) are positive at some point in Σ λ0 , and the strong maximum principle (iii) in Theorem 1.1, we have Let R 0 be the constant in Theorem 1.2. It follows that for any δ > > 0, By the continuity of U λ (x) and V λ (x) with respect to λ, there exists > 0, such that for all λ ∈ (λ 0 , λ 0 + ), we have On the other hand, it yields from (3.4) that Ifx andx are the negative minima of U λ (x) and V λ (x) in Σ λ respectively, we consider two possibilities.
Case 1. One of the negative minima of U λ (x) and V λ (x) lies in B R0 (0), i.e. in the narrow region Σ λ0+ \Σ λ0−δ , and the other is outside of B R0 (0). Without loss of generality, we may assume the negative minimum of U λ (x) lies in B R0 (0). From We know by (1.9) that c 21 (x)|x| β is small for |x| sufficiently large. Since l = + δ is very narrow and c 12 (x) is bounded from below in Σ λ0+ \Σ λ0−δ , it derives that c 12 (x)l α is small. Consequently, it sees that c 12 (x)l α c 21 (x)|x| β < 1. This is a contradiction with (3.13) and so (3.10) is proved. Case 2. The negative minima of U λ (x) and V λ (x) lie all in B R0 (0), i.e. in the narrow region Σ λ0+ \Σ λ0−δ .
By (2.3), where l = δ + . Together with (1.4), it implies Similarly to (3.14), we derive Noting (3.15), we have for l sufficiently small, This contradiction shows that (3.10) has to be true. Now we have shown that U 0 (x) ≡ 0, V 0 (x) ≡ 0, x ∈ Σ 0 . Since the x 1 direction can be chosen arbitrarily, we actually prove that u(x) and v(x) must be radially symmetric about the origin. Also the monotonicity follows easily from the argument. This completes the proof of Theorem 1.3.

4.
Non-existence of positive solutions on a half space R n + . We investigate the system (1.2).
Proof of Theorem 1.4. Based on (1.13), one can see from the proof of Lemma 2.1 in [25] that In fact, assume u(x) ≡ 0, there exists x 0 such that u(x 0 ) = 0, and i.e. 0 ≤ v p (x) + k 1 (x)u r (x) = F α (u(x)) < 0, which is impossible. Hence if u(x) or v(x) attains 0 somewhere in R n + , then u(x) = v(x) ≡ 0, x ∈ R n + . Now we always assume that u(x) > 0 and v(x) > 0 in R n + . Let us carry on the method of moving planes to the solution u along the x n direction.
Using (2.1) in this proof of Theorem 1.1 to the situation here, we only need to take Σ = Σ λ ∪ R n − , where R n − = {x ∈ R n |x n ≤ 0}. Step 1. It is obvious that, for λ ≤ 0, we have For λ sufficiently small, we have immediately since Σ λ is a narrow region.