Regularity of extremal solutions of nonlocal elliptic systems

We examine regularity of the extremal solution of nonlinear nonlocal eigenvalue problem

$ \begin{eqnarray*} \left\{ \begin{array}{lcl} \hfill \mathcal L u & = & \lambda F(u,v) \qquad \text{in} \ \ \Omega, \\ \hfill \mathcal L v & = & \gamma G(u,v) \qquad \text{in} \ \ \Omega, \\ \hfill u,v & = &0 \qquad \qquad \text{on} \ \ \mathbb R^{n} \backslash \Omega , \end{array}\right. \end{eqnarray*} $

with an integro-differential operator, including the fractional Laplacian, of the form

$ \begin{equation*} \label{} \mathcal L(u (x)) = \lim\limits_{\epsilon\to 0} \int_{\mathbb R^n\setminus B_\epsilon(x) } [u(x) - u(z)] J(z-x) dz , \end{equation*} $

when $ J $ is a nonnegative measurable even jump kernel. In particular, we consider jump kernels of the form of $ J(y) = \frac{a(y/|y|)}{|y|^{n+2s}} $ where $ s\in (0,1) $ and $ a $ is any nonnegative even measurable function in $ L^1(\mathbb {S}^{n-1}) $ that satisfies ellipticity assumptions. We first establish stability inequalities for minimal solutions of the above system for a general nonlinearity and a general kernel. Then, we prove regularity of the extremal solution in dimensions $ n < 10s $ and $ n<2s+\frac{4s}{p\mp 1}[p+\sqrt{p(p\mp1)}] $ for the Gelfand and Lane-Emden systems when $ p>1 $ (with positive and negative exponents), respectively. When $ s\to 1 $, these dimensions are optimal. However, for the case of $ s\in(0,1) $ getting the optimal dimension remains as an open problem. Moreover, for general nonlinearities, we consider gradient systems and we establish regularity of the extremal solution in dimensions $ n<4s $. As far as we know, this is the first regularity result on the extremal solution of nonlocal system of equations.