July  2012, 8(3): 611-621. doi: 10.3934/jimo.2012.8.611

Canonical duality solution for alternating support vector machine

1. 

Department of Computer Science and Technology, School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China

Received  June 2011 Revised  December 2011 Published  June 2012

Support vector machine (SVM) is one of the most popular machine learning methods and is educed from a binary data classification problem. In this paper, the canonical duality theory is used to solve the normal model of SVM. Several examples are illustrated to show that the exact solution can be obtained after the canonical duality problem being solved. Moreover, the support vectors can be located by non-zero elements of the canonical dual solution.
Citation: Yubo Yuan. Canonical duality solution for alternating support vector machine. Journal of Industrial & Management Optimization, 2012, 8 (3) : 611-621. doi: 10.3934/jimo.2012.8.611
References:
[1]

J. O. Berger, "Statistical Decision Theory and Bayesian Analysis,", Second edition, (1985).

[2]

P. J. Bickel and K. A. Doksum, "Mathematical Statistics. Basic Ideas and Selected Topics," Second edition,, Prentice-Hall, (2001).

[3]

C. J. C. Burges, A tutorial on support vector machines for pattern recognition,, Data Mining and Knowledge Discovery, 2 (1998), 121.

[4]

O. Chapelle, V. Vapnik, O. Bousquet and S. Mukherjee, Choosing multiple parameters for support vector machines,, Machine Learning, 46 (2002), 131.

[5]

B. Chen and P. T. Harker, Smooth approximations to nonlinear complementarity problems,, SIAM J. Optimization, 7 (1997), 403.

[6]

C. Chen and O. L. Mangasarian, A class of smoothing functions for nonlinear and mixed complementarity problems,, Computational Optimization and Applications, 5 (1996), 97.

[7]

C. Chen and O. L. Mangasarian, Smoothing methods for convex inequalities and linear complementarity problems,, Math. Programming, 71 (1995), 51. doi: 10.1016/0025-5610(95)00005-4.

[8]

X. Chen, L. Qi and D. Sun, Global and superlinear convergence of the smoothing newton method and its application to general box constrained variational inequalities,, Math. of Computation, 67 (1998), 519.

[9]

X. Chen and Y. Ye, On homotopy-smoothing methods for variational inequalities,, SIAM J. Control and Optimization, 37 (1999), 589. doi: 10.1137/S0363012997315907.

[10]

G. P. Crespi, I. Ginchev and M. Rocca, Two approaches toward constrained vector optimization and identity of the solutions,, Journal of Industrial and Management Optimization, 1 (2005), 549.

[11]

G. W. Flake and L. Steve, Efficient SVM regression training with SMO,, Machine Learning, 46 (2002), 271.

[12]

K. Fukunaga, "Introduction to Statistical Pattern Recognition,", Second edition, (1990).

[13]

G. Fung and O. L. Mangasarian, Finite Newton method for Lagrangian support vector machine classification,, Neurocomputing, 55 (2003), 39.

[14]

D. Y. Gao, Canonical dual transformation method and generalized triality theory in nonsmooth global optimization,, J. Global Optimization, 17 (2000), 127.

[15]

D. Y. Gao, Perfect duality theory and complete set of solutions to a class of global optimization,, Optimization, 52 (2003), 467. doi: 10.1080/02331930310001611501.

[16]

D. Y. Gao, Complete solutions to constrained quadratic optimization problems,, Journal of Global Optimisation, 29 (2004), 377. doi: 10.1023/B:JOGO.0000048034.94449.e3.

[17]

D. Y. Gao, Sufficient conditions and perfect duality in nonconvex minimization with inequality constraints,, Journal of Industrial and Management Optimization, 1 (2005), 53.

[18]

D. Y. Gao, Complete solutions and extremality criteria to polynomial optimization problems,, Journal of Global Optimization, 35 (2006), 131. doi: 10.1007/s10898-005-3068-5.

[19]

L. Gonzalez, C. Angulo, F. Velasco and A. Catala, Dual unification of bi-class support vector machine formulations,, Pattern Recognition, 39 (2006), 1325.

[20]

A. G. Hadigheh and T. Terlaky, Generalized support set invariancy sensitivity analysis in linear optimization,, Journal of Industrial and Management Optimization, 2 (2006), 1.

[21]

Q. He, Z.-Z. Shi, L.-A. Ren and E. S. Lee, A novel classification method based on hyper-surface,, Mathematical and Computer Modelling, 38 (2003), 395. doi: 10.1016/S0895-7177(03)90096-3.

[22]

C. W. Hsu and C. J. Lin, A simple decomposition method for support vector machines,, Machine Learning, 46 (2002), 291.

[23]

T. Joachims, Making large-scale support vector machine learning practical,, in, (1999).

[24]

S. S. Keerthi, K. B. Duan, S. K. Shevade and A. N. Poo, A fast dual algorithm for kernel logistic regression,, Machine Learning, 61 (2005), 151.

[25]

P. Laskov, Feasible direction decomposition algorithms for training support vector machines,, Machine Learning, 46 (2002), 315.

[26]

Y.-J. Lee, W. F. Hsieh and C. M. Huang, $\epsilon$-SSVR: A smooth support vector machine for $\epsilon$-insensitive regression,, IEEE Transaction on Knowledge and Data Engineering, 17 (2005), 678. doi: 10.1109/TKDE.2005.77.

[27]

Y.-J. Lee and O. L. Mangarasian, SSVM: A smooth support vector machine for classification,, Computational Optimization and Applications, 22 (2001), 5.

[28]

O. L. Mangasarian and D. R. Musicant, Successive overrelaxation for support vector machines,, IEEE Transactions on Neural Networks, 10 (1999), 1032. doi: 10.1109/72.788643.

[29]

T. M. Mitchell, "Machine Learning,", McGraw-Hill Companies, (1997).

[30]

T. Mitchell, Statistical Approaches to Learning and Discovery,, The course of Machine Learning at CMU, (2003).

[31]

D. Montgomery, "Design and Analysis of Experiments,", Third edition, (1991).

[32]

D. J. Newman, S. Hettich, C. L. Blake and C. J. Merz, "UCI Repository of Machine Learning Databases,", University of California, (1998).

[33]

P.-F. Pai, System reliability forecasting by support vector machines with genetic algorithms,, Mathematical and Computer Modelling, 43 (2006), 262. doi: 10.1016/j.mcm.2005.02.008.

[34]

N. Panda and E. Y. Chang, KDX: An Indexer for support vector machines,, IEEE Transaction on Knowledge and Data Engineering, 18 (2006), 748. doi: 10.1109/TKDE.2006.101.

[35]

J. Platt, Sequential minimal optimization: A fast algorithm for training support vector machines,, Advances in Kernel Methods-Support Vector Learning [R], (1999), 185.

[36]

K. Schittkowski, Optimal parameter selection in support vector machines,, Journal of Industrial and Management Optimization, 1 (2005), 465.

[37]

B. Schölkoft, "Support Vector Learning,", R. Oldenbourg Verlag, (1997).

[38]

V. Vapnik, "The Nature of Statistical Learning Theory,", Springer-Verlag, (1995).

[39]

V. Vapnik, The support vector method of function estimation NATO ASI Series,, in, (1998).

[40]

V. Vapnik, An overview of statistical learning theory,, in, (1999).

[41]

V. Vapnik, Three remarks on support vector function estimation,, IEEE transactions on Neural Networks, 10 (1999), 988.

[42]

Z. Y. Wu, H. W. J. Lee, F. S. Bai and L. S. Zhang, Quadratic smoothing approximation to $l_1$ exact penalty function in global optimization,, Journal of Industrial and Management Optimization, 1 (2005), 533.

[43]

K. F. C. Yiu, K. L. Mak and K. L. Teo, Airfoil design via optimal control theory,, Journal of Industrial and Management Optimization, 1 (2005), 133.

[44]

Y. Yuan, J. Yan and C. Xu, Polynomial smooth support vector machine (PSSVM),, Chinese Journal Of Computers, 28 (2005), 9.

[45]

Y. Yuan and T. Huang, A polynomial smooth support vector machine for classification,, Lecture Note in Artificial Intelligence, 3584 (2005), 157.

[46]

Y. Yuan and R. Byrd, Non-quasi-Newton updates for unconstrained optimization,, J. Comput. Math., 13 (1995), 95.

[47]

Y. Yuan, A modified BFGS algorithm for unconstrained optimization,, IMA J. Numer. Anal., 11 (1991), 325. doi: 10.1093/imanum/11.3.325.

show all references

References:
[1]

J. O. Berger, "Statistical Decision Theory and Bayesian Analysis,", Second edition, (1985).

[2]

P. J. Bickel and K. A. Doksum, "Mathematical Statistics. Basic Ideas and Selected Topics," Second edition,, Prentice-Hall, (2001).

[3]

C. J. C. Burges, A tutorial on support vector machines for pattern recognition,, Data Mining and Knowledge Discovery, 2 (1998), 121.

[4]

O. Chapelle, V. Vapnik, O. Bousquet and S. Mukherjee, Choosing multiple parameters for support vector machines,, Machine Learning, 46 (2002), 131.

[5]

B. Chen and P. T. Harker, Smooth approximations to nonlinear complementarity problems,, SIAM J. Optimization, 7 (1997), 403.

[6]

C. Chen and O. L. Mangasarian, A class of smoothing functions for nonlinear and mixed complementarity problems,, Computational Optimization and Applications, 5 (1996), 97.

[7]

C. Chen and O. L. Mangasarian, Smoothing methods for convex inequalities and linear complementarity problems,, Math. Programming, 71 (1995), 51. doi: 10.1016/0025-5610(95)00005-4.

[8]

X. Chen, L. Qi and D. Sun, Global and superlinear convergence of the smoothing newton method and its application to general box constrained variational inequalities,, Math. of Computation, 67 (1998), 519.

[9]

X. Chen and Y. Ye, On homotopy-smoothing methods for variational inequalities,, SIAM J. Control and Optimization, 37 (1999), 589. doi: 10.1137/S0363012997315907.

[10]

G. P. Crespi, I. Ginchev and M. Rocca, Two approaches toward constrained vector optimization and identity of the solutions,, Journal of Industrial and Management Optimization, 1 (2005), 549.

[11]

G. W. Flake and L. Steve, Efficient SVM regression training with SMO,, Machine Learning, 46 (2002), 271.

[12]

K. Fukunaga, "Introduction to Statistical Pattern Recognition,", Second edition, (1990).

[13]

G. Fung and O. L. Mangasarian, Finite Newton method for Lagrangian support vector machine classification,, Neurocomputing, 55 (2003), 39.

[14]

D. Y. Gao, Canonical dual transformation method and generalized triality theory in nonsmooth global optimization,, J. Global Optimization, 17 (2000), 127.

[15]

D. Y. Gao, Perfect duality theory and complete set of solutions to a class of global optimization,, Optimization, 52 (2003), 467. doi: 10.1080/02331930310001611501.

[16]

D. Y. Gao, Complete solutions to constrained quadratic optimization problems,, Journal of Global Optimisation, 29 (2004), 377. doi: 10.1023/B:JOGO.0000048034.94449.e3.

[17]

D. Y. Gao, Sufficient conditions and perfect duality in nonconvex minimization with inequality constraints,, Journal of Industrial and Management Optimization, 1 (2005), 53.

[18]

D. Y. Gao, Complete solutions and extremality criteria to polynomial optimization problems,, Journal of Global Optimization, 35 (2006), 131. doi: 10.1007/s10898-005-3068-5.

[19]

L. Gonzalez, C. Angulo, F. Velasco and A. Catala, Dual unification of bi-class support vector machine formulations,, Pattern Recognition, 39 (2006), 1325.

[20]

A. G. Hadigheh and T. Terlaky, Generalized support set invariancy sensitivity analysis in linear optimization,, Journal of Industrial and Management Optimization, 2 (2006), 1.

[21]

Q. He, Z.-Z. Shi, L.-A. Ren and E. S. Lee, A novel classification method based on hyper-surface,, Mathematical and Computer Modelling, 38 (2003), 395. doi: 10.1016/S0895-7177(03)90096-3.

[22]

C. W. Hsu and C. J. Lin, A simple decomposition method for support vector machines,, Machine Learning, 46 (2002), 291.

[23]

T. Joachims, Making large-scale support vector machine learning practical,, in, (1999).

[24]

S. S. Keerthi, K. B. Duan, S. K. Shevade and A. N. Poo, A fast dual algorithm for kernel logistic regression,, Machine Learning, 61 (2005), 151.

[25]

P. Laskov, Feasible direction decomposition algorithms for training support vector machines,, Machine Learning, 46 (2002), 315.

[26]

Y.-J. Lee, W. F. Hsieh and C. M. Huang, $\epsilon$-SSVR: A smooth support vector machine for $\epsilon$-insensitive regression,, IEEE Transaction on Knowledge and Data Engineering, 17 (2005), 678. doi: 10.1109/TKDE.2005.77.

[27]

Y.-J. Lee and O. L. Mangarasian, SSVM: A smooth support vector machine for classification,, Computational Optimization and Applications, 22 (2001), 5.

[28]

O. L. Mangasarian and D. R. Musicant, Successive overrelaxation for support vector machines,, IEEE Transactions on Neural Networks, 10 (1999), 1032. doi: 10.1109/72.788643.

[29]

T. M. Mitchell, "Machine Learning,", McGraw-Hill Companies, (1997).

[30]

T. Mitchell, Statistical Approaches to Learning and Discovery,, The course of Machine Learning at CMU, (2003).

[31]

D. Montgomery, "Design and Analysis of Experiments,", Third edition, (1991).

[32]

D. J. Newman, S. Hettich, C. L. Blake and C. J. Merz, "UCI Repository of Machine Learning Databases,", University of California, (1998).

[33]

P.-F. Pai, System reliability forecasting by support vector machines with genetic algorithms,, Mathematical and Computer Modelling, 43 (2006), 262. doi: 10.1016/j.mcm.2005.02.008.

[34]

N. Panda and E. Y. Chang, KDX: An Indexer for support vector machines,, IEEE Transaction on Knowledge and Data Engineering, 18 (2006), 748. doi: 10.1109/TKDE.2006.101.

[35]

J. Platt, Sequential minimal optimization: A fast algorithm for training support vector machines,, Advances in Kernel Methods-Support Vector Learning [R], (1999), 185.

[36]

K. Schittkowski, Optimal parameter selection in support vector machines,, Journal of Industrial and Management Optimization, 1 (2005), 465.

[37]

B. Schölkoft, "Support Vector Learning,", R. Oldenbourg Verlag, (1997).

[38]

V. Vapnik, "The Nature of Statistical Learning Theory,", Springer-Verlag, (1995).

[39]

V. Vapnik, The support vector method of function estimation NATO ASI Series,, in, (1998).

[40]

V. Vapnik, An overview of statistical learning theory,, in, (1999).

[41]

V. Vapnik, Three remarks on support vector function estimation,, IEEE transactions on Neural Networks, 10 (1999), 988.

[42]

Z. Y. Wu, H. W. J. Lee, F. S. Bai and L. S. Zhang, Quadratic smoothing approximation to $l_1$ exact penalty function in global optimization,, Journal of Industrial and Management Optimization, 1 (2005), 533.

[43]

K. F. C. Yiu, K. L. Mak and K. L. Teo, Airfoil design via optimal control theory,, Journal of Industrial and Management Optimization, 1 (2005), 133.

[44]

Y. Yuan, J. Yan and C. Xu, Polynomial smooth support vector machine (PSSVM),, Chinese Journal Of Computers, 28 (2005), 9.

[45]

Y. Yuan and T. Huang, A polynomial smooth support vector machine for classification,, Lecture Note in Artificial Intelligence, 3584 (2005), 157.

[46]

Y. Yuan and R. Byrd, Non-quasi-Newton updates for unconstrained optimization,, J. Comput. Math., 13 (1995), 95.

[47]

Y. Yuan, A modified BFGS algorithm for unconstrained optimization,, IMA J. Numer. Anal., 11 (1991), 325. doi: 10.1093/imanum/11.3.325.

[1]

Yubo Yuan, Weiguo Fan, Dongmei Pu. Spline function smooth support vector machine for classification. Journal of Industrial & Management Optimization, 2007, 3 (3) : 529-542. doi: 10.3934/jimo.2007.3.529

[2]

Ning Lu, Ying Liu. Application of support vector machine model in wind power prediction based on particle swarm optimization. Discrete & Continuous Dynamical Systems - S, 2015, 8 (6) : 1267-1276. doi: 10.3934/dcdss.2015.8.1267

[3]

Pooja Louhan, S. K. Suneja. On fractional vector optimization over cones with support functions. Journal of Industrial & Management Optimization, 2017, 13 (2) : 549-572. doi: 10.3934/jimo.2016031

[4]

Jian Luo, Shu-Cherng Fang, Yanqin Bai, Zhibin Deng. Fuzzy quadratic surface support vector machine based on fisher discriminant analysis. Journal of Industrial & Management Optimization, 2016, 12 (1) : 357-373. doi: 10.3934/jimo.2016.12.357

[5]

Xin Li, Ziguan Cui, Linhui Sun, Guanming Lu, Debnath Narayan. Research on iterative repair algorithm of Hyperchaotic image based on support vector machine. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1199-1218. doi: 10.3934/dcdss.2019083

[6]

Xinmin Yang. On symmetric and self duality in vector optimization problem. Journal of Industrial & Management Optimization, 2011, 7 (3) : 523-529. doi: 10.3934/jimo.2011.7.523

[7]

Radu Ioan Boţ, Sorin-Mihai Grad. On linear vector optimization duality in infinite-dimensional spaces. Numerical Algebra, Control & Optimization, 2011, 1 (3) : 407-415. doi: 10.3934/naco.2011.1.407

[8]

Fengqiu Liu, Xiaoping Xue. Subgradient-based neural network for nonconvex optimization problems in support vector machines with indefinite kernels. Journal of Industrial & Management Optimization, 2016, 12 (1) : 285-301. doi: 10.3934/jimo.2016.12.285

[9]

K. Schittkowski. Optimal parameter selection in support vector machines. Journal of Industrial & Management Optimization, 2005, 1 (4) : 465-476. doi: 10.3934/jimo.2005.1.465

[10]

Hong-Gunn Chew, Cheng-Chew Lim. On regularisation parameter transformation of support vector machines. Journal of Industrial & Management Optimization, 2009, 5 (2) : 403-415. doi: 10.3934/jimo.2009.5.403

[11]

Qinglei Zhang, Wenying Feng. Detecting coalition attacks in online advertising: A hybrid data mining approach. Big Data & Information Analytics, 2016, 1 (2&3) : 227-245. doi: 10.3934/bdia.2016006

[12]

Zhen Mei. Manifold data mining helps businesses grow more effectively. Big Data & Information Analytics, 2016, 1 (2&3) : 275-276. doi: 10.3934/bdia.2016009

[13]

Sunmoo Yoon, Maria Patrao, Debbie Schauer, Jose Gutierrez. Prediction models for burden of caregivers applying data mining techniques. Big Data & Information Analytics, 2017, 2 (5) : 1-9. doi: 10.3934/bdia.2017014

[14]

Anupama N, Sudarson Jena. A novel approach using incremental under sampling for data stream mining. Big Data & Information Analytics, 2017, 2 (5) : 1-13. doi: 10.3934/bdia.2017017

[15]

Anulekha Dhara, Aparna Mehra. Conjugate duality for generalized convex optimization problems. Journal of Industrial & Management Optimization, 2007, 3 (3) : 415-427. doi: 10.3934/jimo.2007.3.415

[16]

Guolin Yu. Global proper efficiency and vector optimization with cone-arcwise connected set-valued maps. Numerical Algebra, Control & Optimization, 2016, 6 (1) : 35-44. doi: 10.3934/naco.2016.6.35

[17]

Maria Gabriella Mosquera, Vlado Keselj. Identifying electronic gaming machine gambling personae through unsupervised session classification. Big Data & Information Analytics, 2017, 2 (2) : 141-175. doi: 10.3934/bdia.2017015

[18]

Siamak RabieniaHaratbar. Support theorem for the Light-Ray transform of vector fields on Minkowski spaces. Inverse Problems & Imaging, 2018, 12 (2) : 293-314. doi: 10.3934/ipi.2018013

[19]

Keiji Tatsumi, Masashi Akao, Ryo Kawachi, Tetsuzo Tanino. Performance evaluation of multiobjective multiclass support vector machines maximizing geometric margins. Numerical Algebra, Control & Optimization, 2011, 1 (1) : 151-169. doi: 10.3934/naco.2011.1.151

[20]

Feyza Gürbüz, Panos M. Pardalos. A decision making process application for the slurry production in ceramics via fuzzy cluster and data mining. Journal of Industrial & Management Optimization, 2012, 8 (2) : 285-297. doi: 10.3934/jimo.2012.8.285

2018 Impact Factor: 1.025

Metrics

  • PDF downloads (7)
  • HTML views (0)
  • Cited by (5)

Other articles
by authors

[Back to Top]