LIMIT CYCLE BIFURCATIONS OF A PIECEWISE SMOOTH HAMILTONIAN SYSTEM WITH A GENERALIZED HETEROCLINIC LOOP THROUGH A CUSP

. In this paper we study the limit cycle bifurcation of a piecewise smooth Hamiltonian system. By using the Melnikov function of piecewise smooth near-Hamiltonian systems, we obtain that at most 12 n +7 limit cycles can bifurcate from the period annulus up to the ﬁrst order in ε

1. Introduction and the main results. The perturbed piecewise smooth differential system has been attracted many researchers to study its limit cycles [1,2,8,13,16,19]. The reason is that it can be applied in many natural fields such as in nonlinear oscillations [17], automatic control [18], economics [7], biology [10] and so on. Another interesting and important reason is that this problem can be seen as an extension of the Hilbert's 16th problem to the piecewise differential system. We recall that the Hilbert's 16th problem asks for the maximum number of limit cycles that bifurcate from a perturbation of a period annulus. For the moment this problem remains open, for more details on the 16th Hilbert's problem see for instance [9,12,20] and the references therein.
Consider a piecewise smooth differential system with discontinuity on the y axiṡ Any segment contained in Σ e ∪ Σ sl is called a sliding segment. Any limit cycle Γ of (1.1) such that Γ ∩ (Σ e ∪ Σ sl ) = ∅ is called a non-sliding limit cycle. From [3,5,11], we know that the flow of (1.1), denoted by φ(t, A), can be defined by using the flows φ ± (t, A) of (1.2) and (1.3). For any point A ∈ R 2 + ∪ R 2 − , where R 2 ± = {(x, y)| ± x > 0}, we have For a point A / ∈ R 2 + ∪ R 2 − satisfying f + (A)f − (A) > 0, we define φ(t, A) as follows if f + (t, A) > 0, f − (t, A) > 0; and − is a singular point of the corresponding system (1.2) or (1.3). In this case, A is also called a singular point of (1.1). For a point A / [6] gave the following definition.
Let A 0 = (0, y 0 ) be either a center, a focus or a center-focus of (1.1). We say it is elementary if one of the conditions is satisfied: (iii) A 0 is elementary as a singular point of (1.3), and By [4,5], system (1.1) has four possible types of foci as follows: (i) Points of focus-focus type at A ∈ Σ: both system (1.2) and (1.3) have a critical point at A with complex eigenvalues and their solutions turn around A counterclockwise.
(ii), (iii) Points of focus-parabolic (resp., parabolic-focus) type at A ∈ Σ: the system defined in the right (resp., left) half plane has a critical point of focus type at A while the solutions of the system defined in neighbourhood of the left (resp., right) half plane have a parabolic contact (i.e., a second order contact point) with Σ at A, the solution of which at this contact is locally contained in the right (resp., left) plane.
(iv) Points of parabolic-focus type at A ∈ Σ: the solutions of both system have a parabolic contact at A with Σ in such a way that the flow induced by (1.1) turns around A.
Observe that the parabolic-focus type can be reduced to the focus-parabolic case by applying the change of coordinates (x, y, t) → (−x, −y, t) to (1.1).
In [15], Liu and Han considered the general form of a piecewise near-Hamiltonian system on the plane ẋ = H y + εp(x, y), where (1.5) This system has two subsystems and (1.7) Suppose that (1.4)| ε=0 has a family of periodic orbits around the origin and satisfies the following two assumptions.

Assumption (I).
There exist an interval J = (α, β), and two points A(h) = (0, a(h)) and Assumption (II). The subsystem (1.6)| ε=0 has an orbital arc L + h starting from A(h) and ending at B(h) defined by H + (x, y) = h, x ≥ 0; the subsystem (1.7)| ε=0 has an orbital arc L − h starting from B(h) and ending at Under the Assumptions (I) and (II), (1.4)| ε=0 has a family of non-smooth periodic For definiteness, we assume that the orbits L h for h ∈ J orientate clockwise; see whereh is given in Assumption (I). Then we know (1.10) As in the smooth case, a very important issue associated with (1.4) is to find the number of limit cycles and their distribution. Liang, Han and Romanovski [14] studied the following piecewise near Hamiltonian system which has a generalize homoclinic loop with ε = 0. They studied the Hopf bifurcation, homoclinic bifurcation and Poincaré bifurcation of (1.11). Motivated by [14,15], in this paper, we consider a piecewise smooth near-Hamiltonian system of the form where p ± (x, y) and q ± (x, y) are defined as (1.5). Then we have (1.14) Obviously, the limit cycles of system (1.12)| ε=0 are all non-sliding limit cycles. By Definitions 1.1-1.3, we know that (1.12)| ε=0 has a hyperbolic saddle point (-1,0), a cusp (1,0) and a elementary center (0,0) which is a parabolic-parabolic type generalized singular point. For system (1.12)| ε=0 , there exist a family of periodic orbits as follows  Note that H + y (0, y) ≡ H − y (0, y) for −1 < y < 1. Then by (1.8), we have the first order Melnikov function of system (1.12) satisfying where 0 < h < 1, and Our main results are the following two theorems.
Theorem 1.4. For system (1.12), the first order Melnikov function has the following form ,ḡ i (u) and g i (u) are polynomial of degree i in u, and where 0 < h < 1, µ i (u) is polynomial of degree i in u, and I i0 (h) is defined as (1.17) for i=1,2.
Proof. Applying the Green's formula we obtain Since AB can be represented as Note that p + (0, y) = n j=0 a + 0j y j . It follows that

The definition of I 0 (h) and (2.4) yield
It is easy to get that

JIHUA YANG, ERLI ZHANG AND MEI LIU
If r = 3l, l ∈ N, we see where ν l+k (h) is a polynomial of degree l + k in h. If r = 3l + 1, l ∈ N , then Using the formula It follows that By (2.8), Further, using the following formula , we obtain By (2.11), From (2.9) and (2.12), we have (2.14) Similar to formula (2.8), we have we get Similar to formula (2.11), we have where By (2.17), From (2.16) and (2.18), we obtain , and substituting (2.6), (2.13) and (2.19) into (2.5), we obtain Lemma 2.2. For system (1.12), the expansion of M − ( h 2 ) has the following form where 0 < h < 1, µ i (u) is a polynomial of degree i in u, and J 10 (h) is defined as (1.17).