American Institute of Mathematical Sciences

In this paper we consider the system involving fully nonlinear nonlocal operators:

$ \left\{\begin{array}{ll}{\mathcal F}_{α}(u(x)) = C_{n,α} PV ∈t_{\mathbb{R}^n} \frac{F(u(x)-u(y))}{|x-y|^{n+α}} dy=v^p(x)+k_1(x)u^r(x),\\{\mathcal G}_{β}(v(x)) = C_{n,β} PV ∈t_{\mathbb{R}^n} \frac{G(v(x)-v(y))}{|x-y|^{n+β}} dy=u^q(x)+k_2(x)v^s(x),\end{array}\right.$

where

$0<α, β<2, $

$p, q, r, s>1, $

$k_1(x), k_2(x)\geq0.$

A narrow region principle and a decay at infinity are established for carrying on the method of moving planes. Then we prove the radial symmetry and monotonicity for positive solutions to the nonlinear system in the whole space. Furthermore non-existence of positive solutions to the system on a half space is derived.