February  2020, 3(1): 51-64. doi: 10.3934/mfc.2020005

An improved deep convolutional neural network model with kernel loss function in image classification

1. 

Key Laboratory of Education Informatization for Nationalities, Ministry of Education, Yunnan Normal University, Kunming 650500, China

2. 

School of Information Science and Technology, Yunnan Normal University, Kunming 650500, China

* Corresponding author: Tianwei Xu

Received  December 2019 Revised  December 2019 Published  February 2020

Fund Project: This work is supported by National Natural Science Foundation of China (No. 61862068)

To further enhance the performance of the current convolutional neural network, an improved deep convolutional neural network model is shown in this paper. Different from the traditional network structure, in our proposed method the pooling layer is replaced by two continuous convolutional layers with $ 3 \times 3 $ convolution kernel between which a dropout layer is added to reduce overfitting, and cross entropy kernel is used as loss function. Experimental results on Mnist and Cifar-10 data sets for image classification show that, compared to several classical neural networks such as Alexnet, VGGNet and GoogleNet, the improved network achieve better performance in learning efficiency and recognition accuracy at relatively shallow network depths.

Citation: Yuantian Xia, Juxiang Zhou, Tianwei Xu, Wei Gao. An improved deep convolutional neural network model with kernel loss function in image classification. Mathematical Foundations of Computing, 2020, 3 (1) : 51-64. doi: 10.3934/mfc.2020005
References:
[1]

M. Abadi, A. Agarwal and P. Barham, et al., TensorFlow: Large-scale machine learning on heterogeneous distributed systems[J], arXiv preprint, arXiv: 1603.04467, 2016. Google Scholar

[2]

Z. AnS. Li and J. Wang, Generalization of deep neural network for bearing fault diagnosis under different working conditions using multiple kernel method,, Neurocomputing, 352 (2019), 42-53.  doi: 10.1016/j.neucom.2019.04.010.  Google Scholar

[3]

V. BadrinarayananA. Kendall and R. Cipolla, SegNet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans. Pattern Anal. Mach. Intell., 39 (2017), 2481-2495.  doi: 10.1109/TPAMI.2016.2644615.  Google Scholar

[4]

C. K. ChuiS. B. Lin and D. X. Zhou, Deep neural networks for rotation-invariance approximation and learning, Anal. Appl., 17 (2019), 737-772.  doi: 10.1142/S0219530519400074.  Google Scholar

[5]

M. Courbariaux, I. Hubara and D. Soudry, et al., Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1[J], arXiv preprint, arXiv: 1602.02830, 2016. Google Scholar

[6]

K. He, X. Zhang and S. Ren, et al., Delving deep into rectifiers: Surpassing human-level performance on imaget classification, Proceeding of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile, Publisher: IEEE, 2015, 1026–1034. doi: 10.1109/ICCV.2015.123.  Google Scholar

[7]

K. He, X. Zhang, S. Ren, et al., Deep residual learning for image recognition, Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, Publisher: IEEE, 2016,770–778. doi: 10.1109/CVPR.2016.90.  Google Scholar

[8]

M. HeydariE. Shivanian and B. Azarnavid, An iterative multistep kernel based method for nonlinear Volterra integral and integro-differential equations of fractional order, J. Comput. Appl. Math., 361 (2019), 97-112.  doi: 10.1016/j.cam.2019.04.017.  Google Scholar

[9]

G. Huang, Z. Liu and L. V. D. Maaten, et al., Densely connected convolutional networks, IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, Publisher: IEEE, 2017, 2261–2269. doi: 10.1109/CVPR.2017.243.  Google Scholar

[10]

Y. JiangZ. Liang and H. Gao, An improved constraint-based Bayesian network learning method using Gaussian kernel probability density estimator, Expert Syst. Appl., 113 (2018), 544-554.  doi: 10.1016/j.eswa.2018.06.058.  Google Scholar

[11]

A. KrizhevskyI. Sutskever and G. Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM, 60 (2017), 84-90.  doi: 10.1145/3065386.  Google Scholar

[12]

Y. LecunL. Bottou and Y. Bengio, Gradient-based learning applied to document recognition,, Proceedings of the IEEE, 86 (1998), 2278-2324.  doi: 10.1109/5.726791.  Google Scholar

[13]

Y. LecunY. Bengio and G. Hinton, Deep learning, Nature, 521 (2015), 436-444.   Google Scholar

[14]

Y. Lei and D. X. Zhou, Convergence of online mirror descent, Appl. Comput. Harmon. Anal., 48 (2020), 343-373.  doi: 10.1016/j.acha.2018.05.005.  Google Scholar

[15]

S. B. Lin and D. X. Zhou, Optimal learning rates for kernel partial least squares, J. Fourier Anal. Appl., 24 (2018), 908-933.  doi: 10.1007/s00041-017-9544-8.  Google Scholar

[16]

J. Niu, L. X. Sun and M. Q. Xu, et al., A reproducing kernel method for solving heat conduction equations with delay,, Appl. Math. Lett., 100 (2020), 106036, 7 pp. doi: 10.1016/j.aml.2019.106036.  Google Scholar

[17]

Y. QuL. Lin and P. Shen, Joint hierarchical category structure learning and large-scale image classification, IEEE Trans. Image Process., 26 (2017), 4331-4346.  doi: 10.1109/TIP.2016.2615423.  Google Scholar

[18]

A. Radford, L. Metz and S. Chintala, Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[J], arXiv preprint, arXiv: 1511.06434, 2015. Google Scholar

[19]

K. Roushangar and S. Shahnazi, Bed load prediction in gravel-bed rivers using wavelet kernel extreme learning machine and meta-heuristic methods, Int. J. Environ. Sci. Te., 16 (2019), 8197-8208.  doi: 10.1007/s13762-019-02287-6.  Google Scholar

[20]

K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition[J]., arXiv preprint, arXiv: 1409.1556, 2015. Google Scholar

[21]

N. SrivastavaG. Hinton and A. Krizhevsky, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15 (2014), 1929-1958.   Google Scholar

[22]

C. Szegedy, W. Liu and Y. Jia, et al., Going deeper with convolutions, Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, Publisher: IEEE, 2015. doi: 10.1109/CVPR.2015.7298594.  Google Scholar

[23]

D. X. Zhou, Universality of deep convolutional neural networks, Appl. Comput. Harmon. Anal., 48 (2020), 787-794.  doi: 10.1016/j.acha.2019.06.004.  Google Scholar

[24]

D. X. Zhou, Theory of deep convolutional neural networks: Downsampling, Neural Networks, 124 (2020), 319-327.  doi: 10.1016/j.neunet.2020.01.018.  Google Scholar

show all references

References:
[1]

M. Abadi, A. Agarwal and P. Barham, et al., TensorFlow: Large-scale machine learning on heterogeneous distributed systems[J], arXiv preprint, arXiv: 1603.04467, 2016. Google Scholar

[2]

Z. AnS. Li and J. Wang, Generalization of deep neural network for bearing fault diagnosis under different working conditions using multiple kernel method,, Neurocomputing, 352 (2019), 42-53.  doi: 10.1016/j.neucom.2019.04.010.  Google Scholar

[3]

V. BadrinarayananA. Kendall and R. Cipolla, SegNet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans. Pattern Anal. Mach. Intell., 39 (2017), 2481-2495.  doi: 10.1109/TPAMI.2016.2644615.  Google Scholar

[4]

C. K. ChuiS. B. Lin and D. X. Zhou, Deep neural networks for rotation-invariance approximation and learning, Anal. Appl., 17 (2019), 737-772.  doi: 10.1142/S0219530519400074.  Google Scholar

[5]

M. Courbariaux, I. Hubara and D. Soudry, et al., Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1[J], arXiv preprint, arXiv: 1602.02830, 2016. Google Scholar

[6]

K. He, X. Zhang and S. Ren, et al., Delving deep into rectifiers: Surpassing human-level performance on imaget classification, Proceeding of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile, Publisher: IEEE, 2015, 1026–1034. doi: 10.1109/ICCV.2015.123.  Google Scholar

[7]

K. He, X. Zhang, S. Ren, et al., Deep residual learning for image recognition, Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, Publisher: IEEE, 2016,770–778. doi: 10.1109/CVPR.2016.90.  Google Scholar

[8]

M. HeydariE. Shivanian and B. Azarnavid, An iterative multistep kernel based method for nonlinear Volterra integral and integro-differential equations of fractional order, J. Comput. Appl. Math., 361 (2019), 97-112.  doi: 10.1016/j.cam.2019.04.017.  Google Scholar

[9]

G. Huang, Z. Liu and L. V. D. Maaten, et al., Densely connected convolutional networks, IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, Publisher: IEEE, 2017, 2261–2269. doi: 10.1109/CVPR.2017.243.  Google Scholar

[10]

Y. JiangZ. Liang and H. Gao, An improved constraint-based Bayesian network learning method using Gaussian kernel probability density estimator, Expert Syst. Appl., 113 (2018), 544-554.  doi: 10.1016/j.eswa.2018.06.058.  Google Scholar

[11]

A. KrizhevskyI. Sutskever and G. Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM, 60 (2017), 84-90.  doi: 10.1145/3065386.  Google Scholar

[12]

Y. LecunL. Bottou and Y. Bengio, Gradient-based learning applied to document recognition,, Proceedings of the IEEE, 86 (1998), 2278-2324.  doi: 10.1109/5.726791.  Google Scholar

[13]

Y. LecunY. Bengio and G. Hinton, Deep learning, Nature, 521 (2015), 436-444.   Google Scholar

[14]

Y. Lei and D. X. Zhou, Convergence of online mirror descent, Appl. Comput. Harmon. Anal., 48 (2020), 343-373.  doi: 10.1016/j.acha.2018.05.005.  Google Scholar

[15]

S. B. Lin and D. X. Zhou, Optimal learning rates for kernel partial least squares, J. Fourier Anal. Appl., 24 (2018), 908-933.  doi: 10.1007/s00041-017-9544-8.  Google Scholar

[16]

J. Niu, L. X. Sun and M. Q. Xu, et al., A reproducing kernel method for solving heat conduction equations with delay,, Appl. Math. Lett., 100 (2020), 106036, 7 pp. doi: 10.1016/j.aml.2019.106036.  Google Scholar

[17]

Y. QuL. Lin and P. Shen, Joint hierarchical category structure learning and large-scale image classification, IEEE Trans. Image Process., 26 (2017), 4331-4346.  doi: 10.1109/TIP.2016.2615423.  Google Scholar

[18]

A. Radford, L. Metz and S. Chintala, Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[J], arXiv preprint, arXiv: 1511.06434, 2015. Google Scholar

[19]

K. Roushangar and S. Shahnazi, Bed load prediction in gravel-bed rivers using wavelet kernel extreme learning machine and meta-heuristic methods, Int. J. Environ. Sci. Te., 16 (2019), 8197-8208.  doi: 10.1007/s13762-019-02287-6.  Google Scholar

[20]

K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition[J]., arXiv preprint, arXiv: 1409.1556, 2015. Google Scholar

[21]

N. SrivastavaG. Hinton and A. Krizhevsky, Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15 (2014), 1929-1958.   Google Scholar

[22]

C. Szegedy, W. Liu and Y. Jia, et al., Going deeper with convolutions, Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, Publisher: IEEE, 2015. doi: 10.1109/CVPR.2015.7298594.  Google Scholar

[23]

D. X. Zhou, Universality of deep convolutional neural networks, Appl. Comput. Harmon. Anal., 48 (2020), 787-794.  doi: 10.1016/j.acha.2019.06.004.  Google Scholar

[24]

D. X. Zhou, Theory of deep convolutional neural networks: Downsampling, Neural Networks, 124 (2020), 319-327.  doi: 10.1016/j.neunet.2020.01.018.  Google Scholar

Figure 1.  Mini-network replacing the $ 3 \times 3 $ convolutions
Figure 2.  kernel size: $ 3 \times 3 $, stride: 2
Figure 3.  Max pooling operation, kernel size: $ 4\times 4 $, stride: 2
Figure 4.  Dropout workflow
Figure 5.  The improved network structure
Figure 6.  The curve of recognition accuracy of Alexnet network and improved network with the training times on cifar-10
Figure 7.  The curve of recognition accuracy of VGGNet and improved network with the training times on cifar-10
Figure 8.  The curve of recognition accuracy of Google network and improved network with the training times on cifar-10
Figure 9.  The curve of recognition accuracy of Alexnet network and improved network with the training times on Minist
Figure 10.  The curve of recognition accuracy of VGGNet and improved network with the training times on Minist
Figure 11.  The curve of recognition accuracy of GoogleNet network and improved network with the training times on Mnist
Table 1.   
Parameters Value
CPU: Intel core i9-9900k
GPU: NVIDIA GeForce RTX 2080ti
RAM: 16.0 GB
OS: WIN10 64-bit
Develop software: Python3.7 + TensorFlow framework (GPU mode)
Parameters Value
CPU: Intel core i9-9900k
GPU: NVIDIA GeForce RTX 2080ti
RAM: 16.0 GB
OS: WIN10 64-bit
Develop software: Python3.7 + TensorFlow framework (GPU mode)
Table 2.   
Network Cifar Mnist
Alexnet: train acc:0.95, test acc:0.78 train acc:0.98, test acc:0.97
VGGNet: train acc:0.98, test acc:0.83 train acc:0.99, test acc:0.98
Google network: train acc:1.0, test acc:0.90 train acc:1.0, test acc:1.0
Improve network: train acc:1.0, test acc:0.94 train acc:1.0, test acc:1.0
Network Cifar Mnist
Alexnet: train acc:0.95, test acc:0.78 train acc:0.98, test acc:0.97
VGGNet: train acc:0.98, test acc:0.83 train acc:0.99, test acc:0.98
Google network: train acc:1.0, test acc:0.90 train acc:1.0, test acc:1.0
Improve network: train acc:1.0, test acc:0.94 train acc:1.0, test acc:1.0
[1]

Mehdi Bastani, Davod Khojasteh Salkuyeh. On the GSOR iteration method for image restoration. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 27-43. doi: 10.3934/naco.2020013

[2]

Qiang Fu, Yanlong Zhang, Yushu Zhu, Ting Li. Network centralities, demographic disparities, and voluntary participation. Mathematical Foundations of Computing, 2020, 3 (4) : 249-262. doi: 10.3934/mfc.2020011

[3]

Xin Guo, Lexin Li, Qiang Wu. Modeling interactive components by coordinate kernel polynomial models. Mathematical Foundations of Computing, 2020, 3 (4) : 263-277. doi: 10.3934/mfc.2020010

[4]

Bahaaeldin Abdalla, Thabet Abdeljawad. Oscillation criteria for kernel function dependent fractional dynamic equations. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020443

[5]

Shipra Singh, Aviv Gibali, Xiaolong Qin. Cooperation in traffic network problems via evolutionary split variational inequalities. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2020170

[6]

Abdelghafour Atlas, Mostafa Bendahmane, Fahd Karami, Driss Meskine, Omar Oubbih. A nonlinear fractional reaction-diffusion system applied to image denoising and decomposition. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020321

[7]

Yuxia Guo, Shaolong Peng. A direct method of moving planes for fully nonlinear nonlocal operators and applications. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020462

[8]

Leilei Wei, Yinnian He. A fully discrete local discontinuous Galerkin method with the generalized numerical flux to solve the tempered fractional reaction-diffusion equation. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020319

[9]

Zhilei Liang, Jiangyu Shuai. Existence of strong solution for the Cauchy problem of fully compressible Navier-Stokes equations in two dimensions. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020348

[10]

Jia Cai, Guanglong Xu, Zhensheng Hu. Sketch-based image retrieval via CAT loss with elastic net regularization. Mathematical Foundations of Computing, 2020, 3 (4) : 219-227. doi: 10.3934/mfc.2020013

[11]

Mathew Gluck. Classification of solutions to a system of $ n^{\rm th} $ order equations on $ \mathbb R^n $. Communications on Pure & Applied Analysis, 2020, 19 (12) : 5413-5436. doi: 10.3934/cpaa.2020246

[12]

Wenjun Liu, Yukun Xiao, Xiaoqing Yue. Classification of finite irreducible conformal modules over Lie conformal algebra $ \mathcal{W}(a, b, r) $. Electronic Research Archive, , () : -. doi: 10.3934/era.2020123

 Impact Factor: 

Metrics

  • PDF downloads (142)
  • HTML views (328)
  • Cited by (0)

Other articles
by authors

[Back to Top]