Figure 1.
CaffeNet architecture
-
Previous Article
Hybrid binary dragonfly enhanced particle swarm optimization algorithm for solving feature selection problems
- MFC Home
- This Issue
-
Next Article
A survey on security and privacy issues of blockchain technology
May
2018, 1(2): 149-180.
doi: 10.3934/mfc.2018008
How convolutional neural networks see the world --- A survey of convolutional neural network visualization methods
| 1. | George Mason University, 4400 University Dr, Fairfax, VA 22030, USA |
| 2. | Clarkson University, 8 Clarkson Ave, Potsdam, NY 13699, USA |
Nowadays, the Convolutional Neural Networks (CNNs) have achieved impressive performance on many computer vision related tasks, such as object detection, image recognition, image retrieval, etc. These achievements benefit from the CNNs' outstanding capability to learn the input features with deep layers of neuron structures and iterative training process. However, these learned features are hard to identify and interpret from a human vision perspective, causing a lack of understanding of the CNNs' internal working mechanism. To improve the CNN interpretability, the CNN visualization is well utilized as a qualitative analysis method, which translates the internal features into visually perceptible patterns. And many CNN visualization works have been proposed in the literature to interpret the CNN in perspectives of network structure, operation, and semantic concept.
In this paper, we expect to provide a comprehensive survey of several representative CNN visualization methods, including Activation Maximization, Network Inversion, Deconvolutional Neural Networks (DeconvNet), and Network Dissection based visualization. These methods are presented in terms of motivations, algorithms, and experiment results. Based on these visualization methods, we also discuss their practical applications to demonstrate the significance of the CNN interpretability in areas of network design, optimization, security enhancement, etc.
Keywords:
Deep learning,
convolutional neural network,
CNN feature,
CNN visualization,
network interpretability.
Mathematics Subject Classification:
Primary: 58F15, 58F17; Secondary: 53C35.
Citation:
Zhuwei Qin, Fuxun Yu, Chenchen Liu, Xiang Chen. How convolutional neural networks see the world --- A survey of convolutional neural network visualization methods. Mathematical Foundations of Computing,
2018, 1
(2)
: 149-180.
doi: 10.3934/mfc.2018008
![]()
References:
| [1] |
P. Agrawal, R. Girshick and J. Malik, Analyzing the performance of multilayer neural networks for object recognition, in Proceedings of the European Conference on Computer Vision, 2014, 329-344.
doi: 10.1007/978-3-319-10584-0_22. |
| [2] |
M. Arjovsky, S. Chintala and L. Bottou, Wasserstein gan, arXiv preprint, arXiv: 1701.07875. Google Scholar |
| [3] |
D. Bau, B. Zhou, A. Khosla, A. Oliva and A. Torralba, Network dissection: Quantifying interpretability of deep visual representations, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 3319-3327.
doi: 10.1109/CVPR.2017.354. |
| [4] |
D. C. Ciresan, U. Meier, J. Masci, L. Maria Gambardella and J. Schmidhuber, Flexible, High performance convolutional neural networks for image classification, in Proceedings of the International Joint Conference on Artificial Intelligence, vol. 22, 2011, p1237. Google Scholar |
| [5] |
R. Collobert, K. Kavukcuoglu and C. Farabet, Torch7: A matlab-like environment for machine learning, in Workshop on BigLearn, NIPS, 2011. Google Scholar |
| [6] |
G. Csurka, C. Dance, L. Fan, J. Willamowski and C. Bray, Visual categorization with bags of keypoints, in Workshop on statistical learning in computer vision, ECCV, vol. 1, 2004, 1-2. Google Scholar |
| [7] |
N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2005, 886-893.
doi: 10.1109/CVPR.2005.177. |
| [8] |
E. d'Angelo, A. Alahi and P. Vandergheynst, Beyond bits: Reconstructing images from local binary descriptors, in Proceedings of the IEEE Conference on Pattern Recognition, 2012, 935-938. Google Scholar |
| [9] |
E. L. Denton, S. Chintala, R. Fergus et al., Deep generative image models using a Laplacian pyramid of adversarial networks, in Proceedings of the Advances in Neural Information Processing Systems, 2015, 1486-1494. Google Scholar |
| [10] |
A. Dosovitskiy and T. Brox, Generating images with perceptual similarity metrics based on deep networks, in Proceedings of the Advances in Neural Information Processing Systems, 2016, 658-666. Google Scholar |
| [11] |
A. Dosovitskiy and T. Brox, Inverting visual representations with convolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 4829-4837.
doi: 10.1109/CVPR.2016.522. |
| [12] |
A. Dosovitskiy, J. Tobias Springenberg and T. Brox, Learning to generate chairs with convolutional neural networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 1538-1546.
doi: 10.1109/CVPR.2015.7298761. |
| [13] |
J. Duchi, E. Hazan and Y. Singer,
Adaptive subgradient methods for online learning and stochastic optimization, Journal of Machine Learning Research, 12 (2011), 2121-2159.
|
| [14] |
D. Erhan, Y. Bengio, A. Courville and P. Vincent, Visualizing higher-layer features of a deep network, Technical report, University of Montreal, (2009), p3. Google Scholar |
| [15] |
P. F. Felzenszwalb, R. B. Girshick, D. McAllester and D. Ramanan,
Object detection with discriminatively trained part-based models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (2010), 1627-1645.
doi: 10.1109/TPAMI.2009.167. |
| [16] |
R. Fong and A. Vedaldi, Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks, arXiv preprint, arXiv: 1801.03454. Google Scholar |
| [17] |
L. A. Gatys, A. S. Ecker and M. Bethge, A neural algorithm of artistic style, Journal of Vision, 16 (2016), p326, arXiv: 1508.06576.
doi: 10.1167/16.12.326. |
| [18] |
L. A. Gatys, A. S. Ecker and M. Bethge, Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks, arXiv preprint, arXiv: 1505.07376, 12. Google Scholar |
| [19] |
R. B. Girshick, P. F. Felzenszwalb and D. McAllester, Discriminatively trained deformable part models, release 5, http://people.cs.uchicago.edu/~rbg/latent-release5/. Google Scholar |
| [20] |
R. Girshick, J. Donahue, T. Darrell and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, 580-587.
doi: 10.1109/CVPR.2014.81. |
| [21] |
X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2010, 249-256. Google Scholar |
| [22] |
Y. Gong, L. Wang, R. Guo and S. Lazebnik, Multi-scale orderless pooling of deep convolutional activation features, in Proceedings of the European Conference on Computer Vision, 2014, 392-407.
doi: 10.1007/978-3-319-10584-0_26. |
| [23] |
A. Gonzalez-Garcia, D. Modolo and V. Ferrari,
Do semantic parts emerge in convolutional neural networks?, International Journal of Computer Vision, 126 (2018), 476-494.
doi: 10.1007/s11263-017-1048-0. |
| [24] |
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, Generative adversarial nets, in Proceedings of the Advances in Neural Information Processing Systems, 2014, 2672-2680. Google Scholar |
| [25] |
A. Gordo, J. Almazán, J. Revaud and D. Larlus, Deep image retrieval: Learning global representations for image search, in Proceedings of the European Conference on Computer Vision, Springer, 2016, 241-257.
doi: 10.1007/978-3-319-46466-4_15. |
| [26] |
S. Han, H. Mao and W. J. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint, arXiv: 1510.00149. Google Scholar |
| [27] |
K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016, 770-778.
doi: 10.1109/CVPR.2016.90. |
| [28] |
G. E. Hinton, S. Osindero and Y.-W. Teh,
A fast learning algorithm for deep belief nets, Neural Computation, 18 (2006), 1527-1554.
doi: 10.1162/neco.2006.18.7.1527. |
| [29] |
D. H. Hubel and T. N. Wiesel, Receptive fields and functional architecture of monkey striate cortex, The Journal of Physiology, 195 (1968), 215-243, URL http://dx.doi.org/10.1113/jphysiol.1968.sp008455.
doi: 10.1113/jphysiol.1968.sp008455. |
| [30] |
D. H. Hubel and T. N. Wiesel,
Receptive fields of single neurones in the cat's striate cortex, The Journal of Physiology, 148 (1959), 574-591.
doi: 10.1113/jphysiol.1959.sp006308. |
| [31] |
S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the International Conference on Machine Learning, 2015, 448-456. Google Scholar |
| [32] |
S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the International Conference on Machine Learning, 2015, 448-456. Google Scholar |
| [33] |
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama and T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in Proceedings of the International Conference on Multimedia, 2014, 675-678.
doi: 10.1145/2647868.2654889. |
| [34] |
G.-S. Kalanit and M. Rafael, The human visual cortex, Annual Review of Neuroscience, 27 (2004), 649-677. Google Scholar |
| [35] |
K. N. Kay, T. Naselaris, R. J. Prenger and J. L. Gallant,
Identifying natural images from human brain activity, Nature, 452 (2008), p352.
doi: 10.1038/nature06713. |
| [36] |
A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Proceedings of the Advances in Neural Information Processing Systems, 2012, 1097-1150.
doi: 10.1145/3065386. |
| [37] |
N. Kruger, P. Janssen, S. Kalkan, M. Lappe, A. Leonardis, J. Piater, A. J. Rodriguez-Sanchez and L. Wiskott,
Deep hierarchies in the primate visual cortex: What can we learn for computer vision?, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35 (2013), 1847-1871.
doi: 10.1109/TPAMI.2012.272. |
| [38] |
A. Kurakin, I. Goodfellow and S. Bengio, Adversarial examples in the physical world, arXiv preprint, arXiv: 1607.02533. Google Scholar |
| [39] |
Y. LeCun, L. Bottou, Y. Bengio and P. Haffner,
Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86 (1998), 2278-2324.
doi: 10.1109/5.726791. |
| [40] |
Y. LeCun, C. Cortes and C. J. Burges, The mnist database of handwritten digits, 1998. Google Scholar |
| [41] |
C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., Photo-realistic single image super-resolution using a generative adversarial network, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
doi: 10.1109/CVPR.2017.19. |
| [42] |
H. Lee, C. Ekanadham and A. Y. Ng, Sparse deep belief net model for visual area v2, in Proceedings of the Advances in Neural Information Processing Systems, 2008, 873-880. Google Scholar |
| [43] |
H. Lee, R. Grosse, R. Ranganath and A. Y. Ng, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, in Proceedings of the International Conference on Machine Learning, 2009, 609-616.
doi: 10.1145/1553374.1553453. |
| [44] |
T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. Zitnick, Microsoft coco: common objects in context. corr abs/1405. 0312 (2014). Google Scholar |
| [45] |
D. G. Lowe,
Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, 60 (2004), 91-110.
doi: 10.1023/B:VISI.0000029664.99615.94. |
| [46] |
A. Mahendran and A. Vedaldi, Understanding deep image representations by inverting them, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, 5188-5196.
doi: 10.1109/CVPR.2015.7299155. |
| [47] |
A. Mahendran and A. Vedaldi,
Visualizing deep convolutional neural networks using natural pre-images, International Journal of Computer Vision, 120 (2016), 233-255.
doi: 10.1007/s11263-016-0911-8. |
| [48] |
M. Manassi, B. Sayim and M. H. Herzog, When crowding of crowding leads to uncrowding, Journal of Vision, 13 (2013), 10-10. Google Scholar |
| [49] |
A. Mordvintsev, C. Olah and M. Tyka, Inceptionism: Going deeper into neural networks, Google Research Blog. Retrieved June, 20 (2015), 14pp. Google Scholar |
| [50] |
A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox and J. Clune, Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, in Proceedings of the Advances in Neural Information Processing Systems, 2016, 3387-3395. Google Scholar |
| [51] |
A. Nguyen, J. Yosinski and J. Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 427-436.
doi: 10.1109/CVPR.2015.7298640. |
| [52] |
A. Nguyen, J. Yosinski and J. Clune, Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, arXiv preprint, arXiv: 1602.03616. Google Scholar |
| [53] |
S. J. Pan and Q. Yang,
A survey on transfer learning, IEEE Transactions on Knowledge and Data Engineering, 22 (2010), 1345-1359.
doi: 10.1109/TKDE.2009.191. |
| [54] |
M. I. Posner and S. E. Petersen, The attention system of the human brain, Annual Review of Neuroscience, 13 (1990), 25-42. Google Scholar |
| [55] |
C. Poultney, S. Chopra, Y. L. Cun et al., Efficient learning of sparse representations with an energy-based model, in Proceedings of the Advances in Neural Information Processing Systems, 2007, 1137-1144. Google Scholar |
| [56] |
N. Qian,
On the momentum term in gradient descent learning algorithms, Neural Networks, 12 (1999), 145-151.
doi: 10.1016/S0893-6080(98)00116-6. |
| [57] |
R. Q. Quiroga, L. Reddy, G. Kreiman, C. Koch and I. Fried., Invariant visual representation by single neurons in the human brain, Nature, 435 (2005), 1102-1107, URL http://dx.doi.org/10.1038/nature03687.
doi: 10.1038/nature03687. |
| [58] |
S. Ren, K. He, R. Girshick and J. Sun,
Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 1137-1149.
doi: 10.1109/TPAMI.2016.2577031. |
| [59] |
L. I. Rudin, S. Osher and E. Fatemi,
Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268.
doi: 10.1016/0167-2789(92)90242-F. |
| [60] |
H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura and R. M. Summers,
Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning, IEEE Transactions on Medical Imaging, 35 (2016), 1285-1298.
doi: 10.1109/TMI.2016.2528162. |
| [61] |
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam and M. Lanctot,
Mastering the game of go with deep neural networks and tree search, Nature, 529 (2016), 484-489.
doi: 10.1038/nature16961. |
| [62] |
K. Simonyan, A. Vedaldi and A. Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps, arXiv preprint, arXiv: 1312.6034. Google Scholar |
| [63] |
K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint, arXiv: 1409.1556. Google Scholar |
| [64] |
J. Sivic and A. Zisserman, Video google: A text retrieval approach to object matching in videos, in Proceeding of Ninth IEEE International Conference on Computer Vision, 2003, 1470.
doi: 10.1109/ICCV.2003.1238663. |
| [65] |
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov,
Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15 (2014), 1929-1958.
|
| [66] |
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, Going deeper with convolutions, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, 1-9.
doi: 10.1109/CVPR.2015.7298594. |
| [67] |
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow and R. Fergus, Intriguing properties of neural networks, arXiv preprint, arXiv: 1312.6199. Google Scholar |
| [68] |
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P.-A. Manzagol,
Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Journal of Machine Learning Research, 11 (2010), 3371-3408.
|
| [69] |
L. Wang, Y. Zhang and J. Feng, On the euclidean distance of images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27 (2005), 1334-1339. Google Scholar |
| [70] |
D. Wei, B. Zhou, A. Torrabla and W. Freeman, Understanding intra-class knowledge inside cnn, arXiv preprint, arXiv: 1507.02379. Google Scholar |
| [71] |
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs and H. Lipson, Understanding neural networks through deep visualization, arXiv preprint, arXiv: 1506.06579. Google Scholar |
| [72] |
M. D. Zeiler and R. Fergus, Visualizing and understanding convolutional networks, in Proceedings of the European Conference on Computer Vision, 2014, 818-833.
doi: 10.1007/978-3-319-10590-1_53. |
| [73] |
M. D. Zeiler, D. Krishnan, G. W. Taylor and R. Fergus, Deconvolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2010, 2528-2535.
doi: 10.1109/CVPR.2010.5539957. |
| [74] |
M. D. Zeiler, G. W. Taylor and R. Fergus, Adaptive deconvolutional networks for mid and high level feature learning, in Proceedings of the IEEE International Conference on Computer Vision, 2011, 2018-2025.
doi: 10.1109/ICCV.2011.6126474. |
| [75] |
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva and A. Torralba, Object detectors emerge in deep scene CNNs, arXiv preprint, arXiv: 1412.6856. Google Scholar |
show all references
References:
| [1] |
P. Agrawal, R. Girshick and J. Malik, Analyzing the performance of multilayer neural networks for object recognition, in Proceedings of the European Conference on Computer Vision, 2014, 329-344.
doi: 10.1007/978-3-319-10584-0_22. |
| [2] |
M. Arjovsky, S. Chintala and L. Bottou, Wasserstein gan, arXiv preprint, arXiv: 1701.07875. Google Scholar |
| [3] |
D. Bau, B. Zhou, A. Khosla, A. Oliva and A. Torralba, Network dissection: Quantifying interpretability of deep visual representations, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 3319-3327.
doi: 10.1109/CVPR.2017.354. |
| [4] |
D. C. Ciresan, U. Meier, J. Masci, L. Maria Gambardella and J. Schmidhuber, Flexible, High performance convolutional neural networks for image classification, in Proceedings of the International Joint Conference on Artificial Intelligence, vol. 22, 2011, p1237. Google Scholar |
| [5] |
R. Collobert, K. Kavukcuoglu and C. Farabet, Torch7: A matlab-like environment for machine learning, in Workshop on BigLearn, NIPS, 2011. Google Scholar |
| [6] |
G. Csurka, C. Dance, L. Fan, J. Willamowski and C. Bray, Visual categorization with bags of keypoints, in Workshop on statistical learning in computer vision, ECCV, vol. 1, 2004, 1-2. Google Scholar |
| [7] |
N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2005, 886-893.
doi: 10.1109/CVPR.2005.177. |
| [8] |
E. d'Angelo, A. Alahi and P. Vandergheynst, Beyond bits: Reconstructing images from local binary descriptors, in Proceedings of the IEEE Conference on Pattern Recognition, 2012, 935-938. Google Scholar |
| [9] |
E. L. Denton, S. Chintala, R. Fergus et al., Deep generative image models using a Laplacian pyramid of adversarial networks, in Proceedings of the Advances in Neural Information Processing Systems, 2015, 1486-1494. Google Scholar |
| [10] |
A. Dosovitskiy and T. Brox, Generating images with perceptual similarity metrics based on deep networks, in Proceedings of the Advances in Neural Information Processing Systems, 2016, 658-666. Google Scholar |
| [11] |
A. Dosovitskiy and T. Brox, Inverting visual representations with convolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 4829-4837.
doi: 10.1109/CVPR.2016.522. |
| [12] |
A. Dosovitskiy, J. Tobias Springenberg and T. Brox, Learning to generate chairs with convolutional neural networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 1538-1546.
doi: 10.1109/CVPR.2015.7298761. |
| [13] |
J. Duchi, E. Hazan and Y. Singer,
Adaptive subgradient methods for online learning and stochastic optimization, Journal of Machine Learning Research, 12 (2011), 2121-2159.
|
| [14] |
D. Erhan, Y. Bengio, A. Courville and P. Vincent, Visualizing higher-layer features of a deep network, Technical report, University of Montreal, (2009), p3. Google Scholar |
| [15] |
P. F. Felzenszwalb, R. B. Girshick, D. McAllester and D. Ramanan,
Object detection with discriminatively trained part-based models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (2010), 1627-1645.
doi: 10.1109/TPAMI.2009.167. |
| [16] |
R. Fong and A. Vedaldi, Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks, arXiv preprint, arXiv: 1801.03454. Google Scholar |
| [17] |
L. A. Gatys, A. S. Ecker and M. Bethge, A neural algorithm of artistic style, Journal of Vision, 16 (2016), p326, arXiv: 1508.06576.
doi: 10.1167/16.12.326. |
| [18] |
L. A. Gatys, A. S. Ecker and M. Bethge, Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks, arXiv preprint, arXiv: 1505.07376, 12. Google Scholar |
| [19] |
R. B. Girshick, P. F. Felzenszwalb and D. McAllester, Discriminatively trained deformable part models, release 5, http://people.cs.uchicago.edu/~rbg/latent-release5/. Google Scholar |
| [20] |
R. Girshick, J. Donahue, T. Darrell and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, 580-587.
doi: 10.1109/CVPR.2014.81. |
| [21] |
X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2010, 249-256. Google Scholar |
| [22] |
Y. Gong, L. Wang, R. Guo and S. Lazebnik, Multi-scale orderless pooling of deep convolutional activation features, in Proceedings of the European Conference on Computer Vision, 2014, 392-407.
doi: 10.1007/978-3-319-10584-0_26. |
| [23] |
A. Gonzalez-Garcia, D. Modolo and V. Ferrari,
Do semantic parts emerge in convolutional neural networks?, International Journal of Computer Vision, 126 (2018), 476-494.
doi: 10.1007/s11263-017-1048-0. |
| [24] |
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, Generative adversarial nets, in Proceedings of the Advances in Neural Information Processing Systems, 2014, 2672-2680. Google Scholar |
| [25] |
A. Gordo, J. Almazán, J. Revaud and D. Larlus, Deep image retrieval: Learning global representations for image search, in Proceedings of the European Conference on Computer Vision, Springer, 2016, 241-257.
doi: 10.1007/978-3-319-46466-4_15. |
| [26] |
S. Han, H. Mao and W. J. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint, arXiv: 1510.00149. Google Scholar |
| [27] |
K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016, 770-778.
doi: 10.1109/CVPR.2016.90. |
| [28] |
G. E. Hinton, S. Osindero and Y.-W. Teh,
A fast learning algorithm for deep belief nets, Neural Computation, 18 (2006), 1527-1554.
doi: 10.1162/neco.2006.18.7.1527. |
| [29] |
D. H. Hubel and T. N. Wiesel, Receptive fields and functional architecture of monkey striate cortex, The Journal of Physiology, 195 (1968), 215-243, URL http://dx.doi.org/10.1113/jphysiol.1968.sp008455.
doi: 10.1113/jphysiol.1968.sp008455. |
| [30] |
D. H. Hubel and T. N. Wiesel,
Receptive fields of single neurones in the cat's striate cortex, The Journal of Physiology, 148 (1959), 574-591.
doi: 10.1113/jphysiol.1959.sp006308. |
| [31] |
S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the International Conference on Machine Learning, 2015, 448-456. Google Scholar |
| [32] |
S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in Proceedings of the International Conference on Machine Learning, 2015, 448-456. Google Scholar |
| [33] |
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama and T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in Proceedings of the International Conference on Multimedia, 2014, 675-678.
doi: 10.1145/2647868.2654889. |
| [34] |
G.-S. Kalanit and M. Rafael, The human visual cortex, Annual Review of Neuroscience, 27 (2004), 649-677. Google Scholar |
| [35] |
K. N. Kay, T. Naselaris, R. J. Prenger and J. L. Gallant,
Identifying natural images from human brain activity, Nature, 452 (2008), p352.
doi: 10.1038/nature06713. |
| [36] |
A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Proceedings of the Advances in Neural Information Processing Systems, 2012, 1097-1150.
doi: 10.1145/3065386. |
| [37] |
N. Kruger, P. Janssen, S. Kalkan, M. Lappe, A. Leonardis, J. Piater, A. J. Rodriguez-Sanchez and L. Wiskott,
Deep hierarchies in the primate visual cortex: What can we learn for computer vision?, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35 (2013), 1847-1871.
doi: 10.1109/TPAMI.2012.272. |
| [38] |
A. Kurakin, I. Goodfellow and S. Bengio, Adversarial examples in the physical world, arXiv preprint, arXiv: 1607.02533. Google Scholar |
| [39] |
Y. LeCun, L. Bottou, Y. Bengio and P. Haffner,
Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86 (1998), 2278-2324.
doi: 10.1109/5.726791. |
| [40] |
Y. LeCun, C. Cortes and C. J. Burges, The mnist database of handwritten digits, 1998. Google Scholar |
| [41] |
C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang et al., Photo-realistic single image super-resolution using a generative adversarial network, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
doi: 10.1109/CVPR.2017.19. |
| [42] |
H. Lee, C. Ekanadham and A. Y. Ng, Sparse deep belief net model for visual area v2, in Proceedings of the Advances in Neural Information Processing Systems, 2008, 873-880. Google Scholar |
| [43] |
H. Lee, R. Grosse, R. Ranganath and A. Y. Ng, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, in Proceedings of the International Conference on Machine Learning, 2009, 609-616.
doi: 10.1145/1553374.1553453. |
| [44] |
T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár and C. Zitnick, Microsoft coco: common objects in context. corr abs/1405. 0312 (2014). Google Scholar |
| [45] |
D. G. Lowe,
Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, 60 (2004), 91-110.
doi: 10.1023/B:VISI.0000029664.99615.94. |
| [46] |
A. Mahendran and A. Vedaldi, Understanding deep image representations by inverting them, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, 5188-5196.
doi: 10.1109/CVPR.2015.7299155. |
| [47] |
A. Mahendran and A. Vedaldi,
Visualizing deep convolutional neural networks using natural pre-images, International Journal of Computer Vision, 120 (2016), 233-255.
doi: 10.1007/s11263-016-0911-8. |
| [48] |
M. Manassi, B. Sayim and M. H. Herzog, When crowding of crowding leads to uncrowding, Journal of Vision, 13 (2013), 10-10. Google Scholar |
| [49] |
A. Mordvintsev, C. Olah and M. Tyka, Inceptionism: Going deeper into neural networks, Google Research Blog. Retrieved June, 20 (2015), 14pp. Google Scholar |
| [50] |
A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox and J. Clune, Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, in Proceedings of the Advances in Neural Information Processing Systems, 2016, 3387-3395. Google Scholar |
| [51] |
A. Nguyen, J. Yosinski and J. Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 427-436.
doi: 10.1109/CVPR.2015.7298640. |
| [52] |
A. Nguyen, J. Yosinski and J. Clune, Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, arXiv preprint, arXiv: 1602.03616. Google Scholar |
| [53] |
S. J. Pan and Q. Yang,
A survey on transfer learning, IEEE Transactions on Knowledge and Data Engineering, 22 (2010), 1345-1359.
doi: 10.1109/TKDE.2009.191. |
| [54] |
M. I. Posner and S. E. Petersen, The attention system of the human brain, Annual Review of Neuroscience, 13 (1990), 25-42. Google Scholar |
| [55] |
C. Poultney, S. Chopra, Y. L. Cun et al., Efficient learning of sparse representations with an energy-based model, in Proceedings of the Advances in Neural Information Processing Systems, 2007, 1137-1144. Google Scholar |
| [56] |
N. Qian,
On the momentum term in gradient descent learning algorithms, Neural Networks, 12 (1999), 145-151.
doi: 10.1016/S0893-6080(98)00116-6. |
| [57] |
R. Q. Quiroga, L. Reddy, G. Kreiman, C. Koch and I. Fried., Invariant visual representation by single neurons in the human brain, Nature, 435 (2005), 1102-1107, URL http://dx.doi.org/10.1038/nature03687.
doi: 10.1038/nature03687. |
| [58] |
S. Ren, K. He, R. Girshick and J. Sun,
Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 1137-1149.
doi: 10.1109/TPAMI.2016.2577031. |
| [59] |
L. I. Rudin, S. Osher and E. Fatemi,
Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268.
doi: 10.1016/0167-2789(92)90242-F. |
| [60] |
H.-C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura and R. M. Summers,
Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning, IEEE Transactions on Medical Imaging, 35 (2016), 1285-1298.
doi: 10.1109/TMI.2016.2528162. |
| [61] |
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam and M. Lanctot,
Mastering the game of go with deep neural networks and tree search, Nature, 529 (2016), 484-489.
doi: 10.1038/nature16961. |
| [62] |
K. Simonyan, A. Vedaldi and A. Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps, arXiv preprint, arXiv: 1312.6034. Google Scholar |
| [63] |
K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint, arXiv: 1409.1556. Google Scholar |
| [64] |
J. Sivic and A. Zisserman, Video google: A text retrieval approach to object matching in videos, in Proceeding of Ninth IEEE International Conference on Computer Vision, 2003, 1470.
doi: 10.1109/ICCV.2003.1238663. |
| [65] |
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov,
Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15 (2014), 1929-1958.
|
| [66] |
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, Going deeper with convolutions, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, 1-9.
doi: 10.1109/CVPR.2015.7298594. |
| [67] |
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow and R. Fergus, Intriguing properties of neural networks, arXiv preprint, arXiv: 1312.6199. Google Scholar |
| [68] |
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P.-A. Manzagol,
Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Journal of Machine Learning Research, 11 (2010), 3371-3408.
|
| [69] |
L. Wang, Y. Zhang and J. Feng, On the euclidean distance of images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27 (2005), 1334-1339. Google Scholar |
| [70] |
D. Wei, B. Zhou, A. Torrabla and W. Freeman, Understanding intra-class knowledge inside cnn, arXiv preprint, arXiv: 1507.02379. Google Scholar |
| [71] |
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs and H. Lipson, Understanding neural networks through deep visualization, arXiv preprint, arXiv: 1506.06579. Google Scholar |
| [72] |
M. D. Zeiler and R. Fergus, Visualizing and understanding convolutional networks, in Proceedings of the European Conference on Computer Vision, 2014, 818-833.
doi: 10.1007/978-3-319-10590-1_53. |
| [73] |
M. D. Zeiler, D. Krishnan, G. W. Taylor and R. Fergus, Deconvolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2010, 2528-2535.
doi: 10.1109/CVPR.2010.5539957. |
| [74] |
M. D. Zeiler, G. W. Taylor and R. Fergus, Adaptive deconvolutional networks for mid and high level feature learning, in Proceedings of the IEEE International Conference on Computer Vision, 2011, 2018-2025.
doi: 10.1109/ICCV.2011.6126474. |
| [75] |
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva and A. Torralba, Object detectors emerge in deep scene CNNs, arXiv preprint, arXiv: 1412.6856. Google Scholar |
Figure 2.
Convolutional and max-pooling process
Figure 3.
Human vision and CNNs visualization
Figure 4.
First layer of CaffeNet visualized by Activation Maximization
Figure 5.
Hidden layers of CaffeNet visualization by Activation Maximization. Adapted from "Understanding Neural Networks Through Deep Visualization," by J. Yosinski, 2015
Figure 6.
Output layer of CaffeNet visualized by Activation Maximization
Figure 7.
The structure of the Deconvolutional Network
Figure 8.
CaffeNet visualized by DeconvNet
Figure 9.
First and second layer visualization of AlexNet and ZFNet Adapted from "Visualizing and Understanding Convolutional Networks," by M.D. Zeiler, 2014
Figure 10.
Feature evolution during training ZFNet. Adapted from "Visualizing and Understanding Convolutional Networks," by M.D. Zeiler, 2014
Figure 11.
The data flow of the two Network Inversion algorithms
Figure 12.
AlexNet reconstruction by Network Inversion with regularizer and UpconvNet. Adapted from "Inverting Visual Representations with Convolutional Networks," by A. Dosovitskiy, 2016
Figure 13.
AlexNet reconstruction by perturbing the feature maps. Adapted from "Inverting Visual Representations with Convolutional Networks," by A. Dosovitskiy, 2016
Figure 14.
The Broden images that activate certain neurons in AlexNet
Figure 15.
Illustration of network dissection for measuring semantic alignment of neuron in a given CNN. Adapted from "Network Dissection: Quantifying Interpretability of Deep Visual Representations," by D. Bau, 2017
Figure 16.
AlexNet visualization by Network Dissection
Figure 17.
Semantic concept emerging in each layers and under different training conditions
Figure 18.
Network Dissection with single neuron and neuron combinations. Adapted from "Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks," by R. Fong, 2018
Figure 19.
Adversarial noises that manipulate the CNN classification
Figure 20.
Adversarial example visualization
Figure 21.
Style transfer example
Table 1.
Visualization methods
| Method | Interpretation Perspective | Focused Layer | Applied Network | Representative Study |
| Activation Maximization | Individual Neuron with visualized pattern | CLs FLs |
Auto-Encoder, DBN, AlexNet | [26] |
| Deconvolutional Neural Networks | Neuron activation in input image | CLs | AlexNet | [55] |
| Network Inversion | One layer | CLs FLs |
HOG, SIFT, LBD, Bag of words, CaffeNet | [29][64] |
| Network Dissection | Individual Neuron with semantic concept | CLs | AlexNet, VGG, GoogLeNet, ResNet | [32][70] |
| Method | Interpretation Perspective | Focused Layer | Applied Network | Representative Study |
| Activation Maximization | Individual Neuron with visualized pattern | CLs FLs |
Auto-Encoder, DBN, AlexNet | [26] |
| Deconvolutional Neural Networks | Neuron activation in input image | CLs | AlexNet | [55] |
| Network Inversion | One layer | CLs FLs |
HOG, SIFT, LBD, Bag of words, CaffeNet | [29][64] |
| Network Dissection | Individual Neuron with semantic concept | CLs | AlexNet, VGG, GoogLeNet, ResNet | [32][70] |
| [1] |
Editorial Office. Retraction: Honggang Yu, An efficient face recognition algorithm using the improved convolutional neural network. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 901-901. doi: 10.3934/dcdss.2019060 |
| [2] |
Rajendra K C Khatri, Brendan J Caseria, Yifei Lou, Guanghua Xiao, Yan Cao. Automatic extraction of cell nuclei using dilated convolutional network. Inverse Problems & Imaging, 2021, 15 (1) : 27-40. doi: 10.3934/ipi.2020049 |
| [3] |
Hiroaki Uchida, Yuya Oishi, Toshimichi Saito. A simple digital spiking neural network: Synchronization and spike-train approximation. Discrete & Continuous Dynamical Systems - S, 2021, 14 (4) : 1479-1494. doi: 10.3934/dcdss.2020374 |
| [4] |
Ziang Long, Penghang Yin, Jack Xin. Global convergence and geometric characterization of slow to fast weight evolution in neural network training for classifying linearly non-separable data. Inverse Problems & Imaging, 2021, 15 (1) : 41-62. doi: 10.3934/ipi.2020077 |
| [5] |
Lars Grüne. Computing Lyapunov functions using deep neural networks. Journal of Computational Dynamics, 2020 doi: 10.3934/jcd.2021006 |
| [6] |
Qiang Fu, Yanlong Zhang, Yushu Zhu, Ting Li. Network centralities, demographic disparities, and voluntary participation. Mathematical Foundations of Computing, 2020, 3 (4) : 249-262. doi: 10.3934/mfc.2020011 |
| [7] |
Nicholas Geneva, Nicholas Zabaras. Multi-fidelity generative deep learning turbulent flows. Foundations of Data Science, 2020, 2 (4) : 391-428. doi: 10.3934/fods.2020019 |
| [8] |
Shipra Singh, Aviv Gibali, Xiaolong Qin. Cooperation in traffic network problems via evolutionary split variational inequalities. Journal of Industrial & Management Optimization, 2020 doi: 10.3934/jimo.2020170 |
| [9] |
Yicheng Liu, Yipeng Chen, Jun Wu, Xiao Wang. Periodic consensus in network systems with general distributed processing delays. Networks & Heterogeneous Media, 2020 doi: 10.3934/nhm.2021002 |
| [10] |
Jing Zhang, Jianquan Lu, Jinde Cao, Wei Huang, Jianhua Guo, Yun Wei. Traffic congestion pricing via network congestion game approach. Discrete & Continuous Dynamical Systems - S, 2021, 14 (4) : 1553-1567. doi: 10.3934/dcdss.2020378 |
| [11] |
Yu-Jhe Huang, Zhong-Fu Huang, Jonq Juang, Yu-Hao Liang. Flocking of non-identical Cucker-Smale models on general coupling network. Discrete & Continuous Dynamical Systems - B, 2021, 26 (2) : 1111-1127. doi: 10.3934/dcdsb.2020155 |
| [12] |
Gernot Holler, Karl Kunisch. Learning nonlocal regularization operators. Mathematical Control & Related Fields, 2021 doi: 10.3934/mcrf.2021003 |
| [13] |
Mohammed Abdulrazaq Kahya, Suhaib Abduljabbar Altamir, Zakariya Yahya Algamal. Improving whale optimization algorithm for feature selection with a time-varying transfer function. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 87-98. doi: 10.3934/naco.2020017 |
| [14] |
Leslaw Skrzypek, Yuncheng You. Feedback synchronization of FHN cellular neural networks. Discrete & Continuous Dynamical Systems - B, 2020 doi: 10.3934/dcdsb.2021001 |
| [15] |
Hongfei Yang, Xiaofeng Ding, Raymond Chan, Hui Hu, Yaxin Peng, Tieyong Zeng. A new initialization method based on normed statistical spaces in deep networks. Inverse Problems & Imaging, 2021, 15 (1) : 147-158. doi: 10.3934/ipi.2020045 |
| [16] |
Ivanka Stamova, Gani Stamov. On the stability of sets for reaction–diffusion Cohen–Grossberg delayed neural networks. Discrete & Continuous Dynamical Systems - S, 2021, 14 (4) : 1429-1446. doi: 10.3934/dcdss.2020370 |
| [17] |
Kengo Nakai, Yoshitaka Saiki. Machine-learning construction of a model for a macroscopic fluid variable using the delay-coordinate of a scalar observable. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1079-1092. doi: 10.3934/dcdss.2020352 |
| [18] |
Yuan Xu, Xin Jin, Saiwei Wang, Yang Tang. Optimal synchronization control of multiple euler-lagrange systems via event-triggered reinforcement learning. Discrete & Continuous Dynamical Systems - S, 2021, 14 (4) : 1495-1518. doi: 10.3934/dcdss.2020377 |
| [19] |
Pierluigi Colli, Gianni Gilardi, Jürgen Sprekels. Deep quench approximation and optimal control of general Cahn–Hilliard systems with fractional operators and double obstacle potentials. Discrete & Continuous Dynamical Systems - S, 2021, 14 (1) : 243-271. doi: 10.3934/dcdss.2020213 |
| [20] |
Bingyan Liu, Xiongbing Ye, Xianzhou Dong, Lei Ni. Branching improved Deep Q Networks for solving pursuit-evasion strategy solution of spacecraft. Journal of Industrial & Management Optimization, 2020 doi: 10.3934/jimo.2021016 |
Impact Factor:
Tools
Article outline
Figures and Tables
[Back to Top]