A Linear-Quadratic Optimal Control Problem for Mean-Field Stochastic Differential Equations in Infinite Horizon

A linear-quadratic (LQ, for short) optimal control problem is considered for mean-field stochastic differential equations with constant coefficients in an infinite horizon. The stabilizability of the control system is studied followed by the discussion of the well-posedness of the LQ problem. The optimal control can be expressed as a linear state feedback involving the state and its mean, through the solutions of two algebraic Riccati equations. The solvability of such kind of Riccati equations is investigated by means of semi-definite programming method.


Preliminary Results
In this section, we present some preliminary results. First of all, let us consider the following result, whose proof follows a standard argument using contraction mapping theorem, together with Itô's formula.
Now, let us look at the cost functional. We observe that the cost functional J(x; u(·)) defined by (1.2) can also be written as J(x; u(·)) = E In what follows, when the dimension of a matrix, say, Q is clear from the context, we write Q ≥ 0 for Q ∈ S n being positive semi-definite and write Q > 0 for Q ∈ S n being positive definite. We now introduce the following assumption concerning the weighting matrices Q,Q, R,R in the cost functional.
(J) The matrices Q,Q ∈ S n and R,R ∈ S m satisfy the following: Note that in (J), we do not have direct assumption onQ andR, they do not have to be positive (semi-) definite, and actually, they could even be negative definite. Under (J), we see that u(·) ∈ U ad [0, ∞) if and only if for any x ∈ R n , the corresponding state process X(·) ≡ X(· ; x, u(·)) satisfies (2.5) Since Q and/or (Q+Q) might be degenerate, when u(·) ∈ U ad [0, ∞), we might not have X(·) ≡ X(·, ; x, u(·)) ∈ X [0, ∞). The following is a little stronger assumption than (J).

Stability
Now, let us return to state equation (1.1). We know that cost functional J(x; u(·)) is well-defined on R n × U ad [0, ∞), and unlike U[0, ∞), the structure of U ad [0, ∞) seems to be complicated since it involves the state equation and the cost functional. Further, the following example shows that U ad [0, ∞) could even be empty, which leads to that Problem (MF-LQ) is meaningless. Then dX(t) = X(t)dt + xe t + u(t) dW (t), t ≥ 0.
From the above, we see that before investigating Problem (MF-LQ), we should find conditions for the system and the cost functional so that the set U ad [0, ∞) is at least non-empty and hopefully it admits an accessible characterization. To this end, let us first look at the following uncontrolled linear MF-FSDE (which amount to saying that taking u(·) = 0 or letting B =B = D =D = 0): where A,Ā, C,C ∈ R n×n are given matrices. The above uncontrolled system is briefly denoted by [A,Ā, C,C].
Let us make the following remarks.
• When (J) holds but (J) ′ does not hold, the L 2 Q,Q -global integrability of the system does not imply the L 2 -global integrability of the system in general.
• It is not clear if (iii) implies (ii), although these two are equivalent for ODE case.
• The notion that is the most relevant to our Problem (MF-LQ) is the L 2 Q,Q -global integrability. Our next goal is to explore when (ii) implies (i). To this end, we first look the caseĀ =C = 0. In this case, our system becomes system [A, C]: t ≥ 0, For such a system, instead of L 2 Q,Q -global integrability, we may introduce the following.
In the case that Q > 0, the L 2 Q -global integrability is simply called the L 2 -global integrability which is equivalent to X(· ; x) ∈ X [0, ∞) for all x ∈ R n .
We have the following result concerning the L 2 Q -global integrability of [A, C].
Proposition 3.5 Let Q ≥ 0. Then the following are equivalent: The following Lyapunov equation admits a solution P ≥ 0: In the above case, the solution P of the above equation admits the following representation: whereF (·) is the solution to the following: The above result should be standard. However, since the idea contained in the proof will be useful below, for readers's convenience, we present a proof here.
Proof. (i) ⇒ (ii). Suppose system [A, C] is L 2 Q -globally integrable. We want to show that Lyapunov equation (3.4) admits a solution P ≥ 0. To this end, let us consider the following linear ODE: which has a unique solution Θ(·) defined on [0, ∞). For any fixed τ > 0, we definē ThenΘ τ (·) is the solution to the following: For any x ∈ R n , let X(·) ≡ X(· ; x) be the solution of (3.3). Applying Itô's formula to s → Θ τ (s)X(s), X(s) , one has Thus, the solution Θ(·) of (3.6) admits the following representation: From the above, since Q ≥ 0, we see that τ → Θ(τ ) is non-decreasing and by the L 2 Q -global integrability of [A, C], one has the following limit: We claim that such a P ≥ 0 must be a solution to the Lyapunov equation (3.4). In fact, from (3.6), one has Letting t → ∞, we see that (3.4) is satisfied by P .
(ii) ⇒ (i) Suppose there exists a P ≥ 0 satisfying (3.4). Then  This implies (3.4) has a solution P ∈ S n . Then by (3.8), we have (iii) For any Q > 0, the Lyapunov equation (3.4) admits a solution P > 0, and in this case, the representation (3.5) holds for this P ; (iv) System [A, C] is L 2 -asymptotically stable, and for some Q > 0, Lyapunov equation (3.4) admits a solution P ∈ S n .
Proof. The implications (i) ⇒ (ii) ⇒ is clear. The relations (ii) ⇐⇒ (iii) ⇐⇒ (iv) follow from Proposition 3.5. The implication (iii) ⇒ (i) follows from (3.7), together with the positive definiteness of P and Q and Gronwall's inequality. Now, let us return system [A,Ā, C,C]. We have the following result. Theorem 3.7 (i) Suppose system [A,Ā, C,C] is L 2 -asymptotically stable. Then it is necessary that A +Ā is exponentially stable.
Next, if [A, C] is L 2 -globally integrable, then by Proposition 3.6, for Q = I, there exists a P > 0 such that P A + A T P + C T P C + I = 0. This results in Therefore, the system [A,Ā, C,C] is L 2 -exponentially stable. This completes the proof.
Note that the exponential stability of A +Ā together with the L 2 -global integrability of [A, C] or (3.9) are sufficient conditions for the L 2 -exponential stability of system [A,Ā, C,C]. When n = 1, these conditions are also necessary in some sense. To be more precise, let us look at the following one-dimensional system: t ≥ 0, (3.12) We have the following result.
Proof. It suffices to prove the implication (iii)⇒(iv). By (2.3) with P = 1, B =B = D =D = 0, A = a, A =ā, C = c,C =c, we have Thus, Thus, under a +ā < 0, if c +c = 0, then we need Since t 0 e (2ā−c 2 )s ds is increasing, the above must lead to 2a + c 2 < 0. Also, if 2a + c 2 ≥ 0, we must have c +c = 0. This completes the proof. Now, for the L 2 Q,Q -global integrability of system [A,Ā, C,C], we have the following result.
we see that (3.13) follows.
Finally, if (3.13) holds and [A, C] is L 2 Q -globally integrable, then by Proposition 3.5, we can find a P ≥ 0 solving Lyapunov equation (3.4). Let X(·) be the solution of (3.1). Applying Itô's formula to P X(·), X(·) , we get Now, condition (3.14) implies that Consequently, This means that the system is L 2 Q,Q -globally integrable. We point out that condition (3.14) holds if (3.9) is true or Therefore, to have condition (3.14), we do not have to assume (J) ′ .

MF-Stabilizability
We now return to the controlled linear MF-FSDE (1.1) which is denoted by [A,Ā, C,C; B,B, D,D]. With this notation, we see that the uncontrolled MF-FSDE (3.1) is nothing but [A,Ā, C,C; 0, 0, 0, 0]. Note also that in the caseĀ =C = 0 andB =D = 0, the system is a usual controlled linear SDE, which is simply  Q,Q -stabilizable if there exists a pair (K,K) ∈ R n×m × R n×m such that for any x ∈ R n if X K,K (·) is the solution to the following: In this case, the pair (K,K) is called an MF-L 2 Q,Q -stabilizer of the system. In the case that (4.2) is replaced by the following: we simply say that the system is MF-L 2 -stabilizable, and (K,K) is called an MF-L 2 -stabilizer of the system.
is the solution to the following: and In this case, K is called an L 2 Q,Q -stabilizer of the system. In the case thatQ = 0, we simply say that the system is L 2 Q -stabilizable, and K is called an L 2 Q -stabilizer. If (4.4) is replaced by we further simply say that the system is L 2 -stabilizable, and K is called an L 2 -stabilizer of the system.
The importance of the notions defined in the above definition is that if (J) holds and [A, It is seen that when system [A,Ā, C,C; B,B, D,D] is MF-L 2 Q,Q -stabilizable, then the uncontrolled system Moreover, it is clear that the L 2 -stabilizability of system [A, C; B, D] we defined here is the classic stabilizability of the controlled SDE system.
Therefore, the former is a special case of the later. The following example shows that in general, the MF-L 2 -stabilizability does not imply the L 2 -stabilizability.
Then we need and only need a +ā +k ≡ −λ < 0, 2(a + bk) + (c + k) 2 < 0, (4.5) for some k,k ∈ R. The first condition in (4.5) can always be achieved. The second one is equivalent to the following: On the other hand, in order the system to be stabilizable, we need k =k, and for some λ > 0. This is impossible if, say, It is easy to find cases that (4.6)-(4.7) hold. Hence, we see that MF-L 2 -stabilizability does not imply L 2 -stabilizability, in general. Now, we present a result concerning the MF-L 2 Q,Q -stabilizability of system (1.1). (4.8) (ii) Suppose the following holds for someK ∈ R m×n satisfying (4.8): Suppose the following holds for someK ∈ R n×m satisfying (4.8): C +C + (D +D)K = 0. (4.10) Then the controlled MF Proof. Under (4.1), the closed-loop system takes form (4.3). According to Proposition 3.9, we know that if (4.3) is L 2 Q,Q -globally integrable, it is necessary that (4.8) holds, which proves (i). Further, when (4.8) holds, the system (4.3) is L 2 Q,Q -globally integrable if either the system [A + BK, C + DK] is stable and (4.9) holds, which proves (ii), or (4.10) holds with the sameK which proves (iii).
The above leads to the following corollary. (iii) Suppose (4.10) holds for someK ∈ R n×m satisfying (4.11). Then the controlled MF-FSDE system Note that conditions assumed in (ii) of Corollary 4.4 do not involveC andD. However, condition (4.10) involves bothC andD. We point out that (4.10) means that In the case that m < n, the above could be a big restriction on C +C and D +D. Moreover, we have to find the sameK ∈ R m×n such that (4.11) and (4.10) hold at the same time. If we let (D +D) + be the Moore-Penrose pseudo-inverse of D +D ( [7]), then the solution of (4.10) is given bȳ for some K ∈ R m×n . Thus, we need for some K ∈ R m×n , which means the ODE system is stabilizable. Hence, we obtain the following result.
is stabilizable, which is the case, if, in particular, m = n, D +D is invertible, and Condition (4.14) seems that the MF-L 2 -stabilizability of system [A,Ā, C,C; B,B, D,D] could be nothing to do with the stabilizability of the controlled linear SDE system [A, C; B, D]. However, in the case that A =C = 0 andB =D = 0, we have the following controlled linear SDE: Suppose m = n and D −1 exists. Then condition (4.14) becomes In this case, if we take then the closed-loop system becomes then the closed-loop system reads It is not hard to see that the unique solution X(·) of the above is deterministic and given by Therefore the system is also asymptotically stable under feedback control (4.16).

Stochastic LQ Problems
In this section, we study a classic stochastic LQ problem, which will be crucial for Problem (MF-LQ). We consider the following controlled SDE: and cost functional

A Classic Stochastic LQ Problem
We introduce the following assumptions.
(J) * The matrices Q ∈ S n and R ∈ S m satisfy (S) * The system [A, C; B, D] is L 2 Q -stabilizable. Let us pose the following problem.
Problem (LQ). For any x ∈ R n , find a u * (·) ∈ U Q ad [0, ∞) such that We have the following result.
Theorem 5.1 Let (J) * and (S) * hold. Then Problem (LQ) admits a unique optimal control u Q (·) ∈ U Q ad [0, ∞). Moreover, the following ARE admits a solution P ≥ 0: Further, the optimal control u Q (·) is given by with the optimal state process X Q (·) being the solution of closed-loop system: Proof. First of all, it is clear that under (J) * and (S) * , the set U Q ad [0, ∞) is nonempty, and (x, u(·)) → J 0 (x; u(·)) is a quadratic functional, coercive with respect to u(·) ∈ U Q ad [0, ∞). Thus for any x ∈ R n , there exists a unique optimal control u Q (·) ∈ U Q ad [0, ∞), and the value function x → V 0 (x) must be of form (5.2) for some P ≥ 0. We now would like to determine P and the optimal pair (X * (·), u * (·)). To this end, let us introduce where u(·) ∈ U loc [0, ∞) and X(·) = X(· ; x, u(·)). It is standard that under (J) * , there exists a unique u Q T (·) ∈ U[0, T ] such that with P (· ; T ) being the solution to the following differential Riccati equation: Moreover, the optimal control u T (·) can be represented as follows: and X Q T (·) is the solution to the following closed-loop system: Now, it is clear that Therefore, one has 0 ≤ P (0; T ) ≤ P (0;T ), On the other hand, since it is true that Combining the above, we see that This implies that lim ThisP ≥ 0 must be a solution to the algebraic Riccati equation: Further, from (5.6), one has lim T →∞ Note that (suppressing (t; T )) = P A + A T P + C T P C − (P B + C T P D)(R + D T P D) −1 (B T P + D T P C).
6 Optimal MF-LQ Controls Presented via Tackling AREs

Tackling AREs via LMIs
One of the main ideas of this section is to reformulate the AREs as linear matrix inequalities (LMIs, for short). Let us introduce the general notion of LMIs according to [1,27], and develop it to solve our mean-field LQ problem.
Definition 6.1 Let F 0 , F 1 , · · · , F m ∈ S n be given. Inequalities consisting of any combination of the following relations are called LMIs with respect to the variable x = (x 1 , · · · , x m ) T ∈ R m . When the LMI is satisfied by a vector x we say that the LMI is feasible and x is a feasible point.
(ii) There exists a solution to the AREs (6.4).
Moreover, when (i) or (ii) holds, the AREs (6.4) has a maximal solution (P * , Π * ) which is the unique optimal solution to the SDP problem (6.5).
As an immediate consequence of Theorem 6.5, we have the following result for the standard case Q,Q ≥ 0 and R,R > 0. Corollary 6.6 If Q,Q ≥ 0 and R,R > 0, then the AREs (6.4) admits a maximal solution (P * , Π * ) with P * , Π * ≥ 0 which is also the unique solution to the SDP (6.5). In addition, if Q,Q > 0 and R,R > 0, then the maximal solution (P * , Π * ) with P * , Π * > 0 and the feedback control is stabilizing for the system (1.1).

Optimal feedback Control
In this subsection, we show that the value function of Problem MF-LQ can be expressed in terms of the maximal solution to the AREs (6.4). Moreover, if there exists an optimal control of Problem MF-LQ then it is necessarily represented as a feedback via the maximal solution to the AREs.
Theorem 6.7 Assume that Theorem 6.5-(i) holds. Then Problem (MF-LQ) is well-posed and the value function is given by V (x) = x T Π * x, ∀x ∈ R n , where (P * , Π * ) is the maximal solution to the AREs (6.4).
In addition, by Proposition A.10, the feedback control u It is easy to verify that P ǫ , Π ǫ , Γ ǫ andΓ ǫ satisfy the following equations Since lim

27
On the other hand, since P * = lim ǫ→0 P ǫ and Π * = lim ǫ→0 Π ǫ (similar to the proof of Theorem 6.5), we have This completes the proof.
Corollary 6.8 Assume that Theorem 6.5-(i) holds. If there exists an optimal control of Problem (MF-LQ), then it must be unique and represented by the state feedback control where (P * , Π * ) is the maximal solution to the AREs (6.4), and    Proof. Let (X * (·), u * (·)) be an optimal pair of the LQ problem. Then a completion of squares shows As u * (·) is stabilizing, we have

Numerical Examples
In this section, we report our numerical experiments based on the approach developed in the previous sections. Note that the numerical algorithm we have used for checking LMIs or solving SDP [33].
The system dynamics (1.1) in our experiments is specified by the following matrices

Numerical test of MF-L 2 stabilizability
Since we have shown that the controlled MF-FSDE system is MF-L 2 -stabilizable in Proposition A.5 if and only if (A.6) is feasible (with respect to the variables X,X, Y andȲ ), we should check the MF-L 2 stabilizability first by tackling inequalities. After running the calculation of SDP program via Matlab software, the obtained feasible matrices X,X, Y andȲ satisfy Proposition A.5: Its proof can be found in [5].
(v) There are matrices K,K such that for any matrices Y,Ȳ there exist unique solution X,X to the following matrix equations Moreover, if Y,Ȳ > 0 (resp. Y,Ȳ ≥ 0) then X,X > 0 (resp. X,X ≥ 0). Furthermore, in this case the feedback is stabilizing.
(vi) There exist matrices Y,Ȳ and symmetric matrices X,X such that In this case the feedback u(t) = Y X Applying the general result given in the appendix of [19], we have the equivalence between the mean-square stabilizability and each of the assertions (ii)-(v). Furthermore, with Y = KX andȲ =KX the condition Let p * denote the infimum value of the primal SDP (6.2) and d * the supremum value of its dual (6.3). Then we have the following results ( [33,1]). Proposition A.6 p * = d * if either of the following conditions holds: (i) The primal problem (6.2) is strictly feasible, i.e., there exists an x such that F (x) > 0.
If both conditions (i) and (ii) hold, then the optimal sets of both the primal and the dual are nonempty. In this case, the following complementary slackness condition is necessary and sufficient for achieving the optimal values for both problems.
Consider the following SDP problem Proof. The constraints of the general dual problem (6.3) can be formulated equivalently as the constraints of (A.11). To this end, define the dual variable Z ∈ S 4n+2m for (6.3) as By the general duality relation Tr(ZF i ) = c i , i = 1, · · · , m (see (6.3)) it follows that for any (P, Π) ∈ S n ×S n , In particular, since the matrix variables Y 1 , Y 2 , Y 3 , Y 4 , Y 5 and Y 6 do not play any role in the above formulation, they can be dropped. Hence, the condition Z ≥ 0 is equivalent to This completes the proof.
We now show that the MF-L 2 -stability can be regarded as a dual concept of SDP optimality.