# American Institute of Mathematical Sciences

• Previous Article
Global convergence and geometric characterization of slow to fast weight evolution in neural network training for classifying linearly non-separable data
• IPI Home
• This Issue
• Next Article
Reproducible kernel Hilbert space based global and local image segmentation
February  2021, 15(1): 27-40. doi: 10.3934/ipi.2020049

## Automatic extraction of cell nuclei using dilated convolutional network

 1 Department of Mathematical Sciences, The University of Texas at Dallas, Richardson, TX 75080, USA 2 Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas TX 75390, USA

* Corresponding author: Yan Cao

Received  December 2019 Revised  April 2020 Published  August 2020

Fund Project: This work was supported in part by the National Science Foundations Enriched Doctoral Training Program, DMS grant #1514808.

Pathological examination has been done manually by visual inspection of hematoxylin and eosin (H&E)-stained images. However, this process is labor intensive, prone to large variations, and lacking reproducibility in the diagnosis of a tumor. We aim to develop an automatic workflow to extract different cell nuclei found in cancerous tumors portrayed in digital renderings of the H&E-stained images. For a given image, we propose a semantic pixel-wise segmentation technique using dilated convolutions. The architecture of our dilated convolutional network (DCN) is based on SegNet, a deep convolutional encoder-decoder architecture. For the encoder, all the max pooling layers in the SegNet are removed and the convolutional layers are replaced by dilated convolution layers with increased dilation factors to preserve image resolution. For the decoder, all max unpooling layers are removed and the convolutional layers are replaced by dilated convolution layers with decreased dilation factors to remove gridding artifacts. We show that dilated convolutions are superior in extracting information from textured images. We test our DCN network on both synthetic data sets and a public available data set of H&E-stained images and achieve better results than the state of the art.

Citation: Rajendra K C Khatri, Brendan J Caseria, Yifei Lou, Guanghua Xiao, Yan Cao. Automatic extraction of cell nuclei using dilated convolutional network. Inverse Problems & Imaging, 2021, 15 (1) : 27-40. doi: 10.3934/ipi.2020049
##### References:

show all references

##### References:
$3 \times 3$ convolution kernels with different dilation factors 1, 2 and 3 respectively. Red dots indicate nonzero values
A matrix $A$ and its reordering $B$. It also shows a dilated convolution on $A$ and its corresponding convolution on $B$
Source image (left), Target image (middle) and Normalized image by Reinhard method (Right)
Sample training images in the triangle data set with a uniform foreground and a uniform background
Sample training images in the triangle data set with a textured foreground and a textured background
Test images (1st row) with corresponding segmentations using SegNet (2nd row), U-Net3 (3rd row), U-Net4 (4th row) and our dilated convolutional network (5th row)
Test images (1st row) with corresponding segmentations using SegNet (2nd row), U-Net3 (3rd row), U-Net4 (4th row) and our dilated convolutional network (5th row)
Sample normalized image patches and corresponding manual segmentations from the dataset
Test images (1st column) with corresponding segmentations using SegNet (2nd column), U-Net3 (3rd column) and our dilated convolutional network (4th column). Ground truth contours are plotted in red
Comparison of the network architectures of the SegNet and our Dilated Convolutional network, where "Conv" means "Convolutions" and D is the dilation factor. Third column shows how to use matrix splitting and merging procedures to implement dilated convolutions through efficient conventional convolutions
 SegNet Our DCN Our DCN (for efficient training) 128x128x3 Input 128x128x3 Input 128x128x3 Input (or 64x64 Input) (or 64x64 Input) (or 64x64) Input 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU ReLU ReLU ReLU 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Pooling Matrix Splitting 64 3x3 Conv 64 3x3 Conv, D=2 64 3x3 Conv Encoder Normalization & RELU Normalization & RELU Normalization & RELU 64 3x3 Conv 64 3x3 Conv, D=2 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Pooling Matrix Splitting 64 3x3 Conv 64 3x3 Conv, D=4 64 3x3 Conv Batch Normalization Batch Normalization Batch Normalization ReLU ReLU ReLU 64 3x3 Conv 64 3x3 Conv, D=4 64 3x3 Conv Batch Normalization Batch Normalization Batch Normalization ReLU ReLU ReLU Max Pooling Max Unpooling 64 3x3 Conv 64 3x3 Conv, D=4 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Matrix Merging 64 3x3 Conv 64 3x3 Conv, D=2 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Unpooling Matrix Merging 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Decoder Normalization & RELU Normalization & RELU Normalization & RELU 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Unpooling 64 3x3 Conv Normalization & RELU 2 3x3 Conv 2 1x1 Conv 2 1x1 Conv Normalization & RELU Softmax Softmax Softmax Pixel Classification Pixel Classification Pixel Classification
 SegNet Our DCN Our DCN (for efficient training) 128x128x3 Input 128x128x3 Input 128x128x3 Input (or 64x64 Input) (or 64x64 Input) (or 64x64) Input 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU ReLU ReLU ReLU 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Pooling Matrix Splitting 64 3x3 Conv 64 3x3 Conv, D=2 64 3x3 Conv Encoder Normalization & RELU Normalization & RELU Normalization & RELU 64 3x3 Conv 64 3x3 Conv, D=2 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Pooling Matrix Splitting 64 3x3 Conv 64 3x3 Conv, D=4 64 3x3 Conv Batch Normalization Batch Normalization Batch Normalization ReLU ReLU ReLU 64 3x3 Conv 64 3x3 Conv, D=4 64 3x3 Conv Batch Normalization Batch Normalization Batch Normalization ReLU ReLU ReLU Max Pooling Max Unpooling 64 3x3 Conv 64 3x3 Conv, D=4 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Matrix Merging 64 3x3 Conv 64 3x3 Conv, D=2 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Unpooling Matrix Merging 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Decoder Normalization & RELU Normalization & RELU Normalization & RELU 64 3x3 Conv 64 3x3 Conv, D=1 64 3x3 Conv Normalization & RELU Normalization & RELU Normalization & RELU Max Unpooling 64 3x3 Conv Normalization & RELU 2 3x3 Conv 2 1x1 Conv 2 1x1 Conv Normalization & RELU Softmax Softmax Softmax Pixel Classification Pixel Classification Pixel Classification
Quantitative metrics of the segmentation results on triangle data sets. Best values are displayed in bold
 Triangle Global Mean Mean Weighted Mean Dataset Accuracy Accuracy IoU IoU BFScore SegNet Uniform 0.9325 0.9508 0.7829 0.8882 0.4172 U-Net3 Uniform 0.9694 0.9531 0.8784 0.9438 0.6572 U-Net4 Uniform 0.9974 0.9953 0.9884 0.9949 0.9488 Our DCN Uniform 0.9952 0.9941 0.9786 0.9906 0.8946 SegNet Textured 0.8764 0.9280 0.6818 0.8139 0.3605 U-Net3 Textured 0.8119 0.8614 0.5855 0.7359 0.2157 U-Net4 Textured 0.7250 0.8148 0.4945 0.6391 0.2000 Our DCN Textured 0.9658 0.9741 0.8728 0.9386 0.4638
 Triangle Global Mean Mean Weighted Mean Dataset Accuracy Accuracy IoU IoU BFScore SegNet Uniform 0.9325 0.9508 0.7829 0.8882 0.4172 U-Net3 Uniform 0.9694 0.9531 0.8784 0.9438 0.6572 U-Net4 Uniform 0.9974 0.9953 0.9884 0.9949 0.9488 Our DCN Uniform 0.9952 0.9941 0.9786 0.9906 0.8946 SegNet Textured 0.8764 0.9280 0.6818 0.8139 0.3605 U-Net3 Textured 0.8119 0.8614 0.5855 0.7359 0.2157 U-Net4 Textured 0.7250 0.8148 0.4945 0.6391 0.2000 Our DCN Textured 0.9658 0.9741 0.8728 0.9386 0.4638
Quantitative metrics of the segmentation results on the H&E-stained image data set. Best values are displayed in bold
 Image Global Mean Mean Weighted Mean Set Accuracy Accuracy IoU IoU BFScore SegNet Lung 0.8819 0.8975 0.7324 0.8074 0.9204 U-Net3 Lung 0.8917 0.9013 0.7475 0.8190 0.9266 U-Net4 Lung 0.8929 0.8966 0.7484 0.8193 0.9343 Our DCN Lung 0.9045 0.9033 0.7690 0.8355 0.9448 SegNet Breast 0.8691 0.8990 0.7002 0.7917 0.8900 U-Net3 Breast 0.8829 0.9051 0.7183 0.8124 0.8743 U-Net4 Breast 0.8775 0.8977 0.7086 0.8057 0.8700 Our DCN Breast 0.9047 0.9123 0.7538 0.8415 0.9210 SegNet Kidney 0.9122 0.9249 0.7290 0.8634 0.9425 U-Net3 Kidney 0.9133 0.9281 0.7259 0.8639 0.9218 U-Net4 Kidney 0.8993 0.9145 0.7013 0.8462 0.9306 Our DCN Kidney 0.9329 0.9277 0.7725 0.8911 0.9634 SegNet Prostate 0.8956 0.9142 0.7533 0.8271 0.9105 U-Net3 Prostate 0.8949 0.9041 0.7496 0.8255 0.9047 U-Net4 Prostate 0.8961 0.9032 0.7510 0.8271 0.9090 Our DCN Prostate 0.9211 0.9163 0.7962 0.8632 0.9336 SegNet Overall 0.8897 0.9000 0.7383 0.8184 0.9159 U-Net3 Overall 0.8957 0.8976 0.7467 0.8264 0.9069 U-Net4 Overall 0.8914 0.8905 0.7380 0.8201 0.9110 Our DCN Overall 0.9158 0.9039 0.7815 0.8548 0.9407
 Image Global Mean Mean Weighted Mean Set Accuracy Accuracy IoU IoU BFScore SegNet Lung 0.8819 0.8975 0.7324 0.8074 0.9204 U-Net3 Lung 0.8917 0.9013 0.7475 0.8190 0.9266 U-Net4 Lung 0.8929 0.8966 0.7484 0.8193 0.9343 Our DCN Lung 0.9045 0.9033 0.7690 0.8355 0.9448 SegNet Breast 0.8691 0.8990 0.7002 0.7917 0.8900 U-Net3 Breast 0.8829 0.9051 0.7183 0.8124 0.8743 U-Net4 Breast 0.8775 0.8977 0.7086 0.8057 0.8700 Our DCN Breast 0.9047 0.9123 0.7538 0.8415 0.9210 SegNet Kidney 0.9122 0.9249 0.7290 0.8634 0.9425 U-Net3 Kidney 0.9133 0.9281 0.7259 0.8639 0.9218 U-Net4 Kidney 0.8993 0.9145 0.7013 0.8462 0.9306 Our DCN Kidney 0.9329 0.9277 0.7725 0.8911 0.9634 SegNet Prostate 0.8956 0.9142 0.7533 0.8271 0.9105 U-Net3 Prostate 0.8949 0.9041 0.7496 0.8255 0.9047 U-Net4 Prostate 0.8961 0.9032 0.7510 0.8271 0.9090 Our DCN Prostate 0.9211 0.9163 0.7962 0.8632 0.9336 SegNet Overall 0.8897 0.9000 0.7383 0.8184 0.9159 U-Net3 Overall 0.8957 0.8976 0.7467 0.8264 0.9069 U-Net4 Overall 0.8914 0.8905 0.7380 0.8201 0.9110 Our DCN Overall 0.9158 0.9039 0.7815 0.8548 0.9407
Comparison of the training and testing time of SegNet, U-Net3 and our DCN
 SegNet U-Net3 Our DCN Our DCN (efficient) Training Time 15 min 14 sec 18 min 36 sec 26 min 47 sec 19 min 45 sec Testing Time 7.6 sec 37.7 sec 10.1 sec 19.9 sec
 SegNet U-Net3 Our DCN Our DCN (efficient) Training Time 15 min 14 sec 18 min 36 sec 26 min 47 sec 19 min 45 sec Testing Time 7.6 sec 37.7 sec 10.1 sec 19.9 sec
 [1] Wei Wang, Degen Huang, Haitao Yu. Word sense disambiguation based on stretchable matching of the semantic template. Mathematical Foundations of Computing, 2021, 4 (1) : 1-13. doi: 10.3934/mfc.2020022 [2] Lekbir Afraites, Abdelghafour Atlas, Fahd Karami, Driss Meskine. Some class of parabolic systems applied to image processing. Discrete & Continuous Dynamical Systems - B, 2016, 21 (6) : 1671-1687. doi: 10.3934/dcdsb.2016017 [3] Zhihua Zhang, Naoki Saito. PHLST with adaptive tiling and its application to antarctic remote sensing image approximation. Inverse Problems & Imaging, 2014, 8 (1) : 321-337. doi: 10.3934/ipi.2014.8.321 [4] Israa Mohammed Khudher, Yahya Ismail Ibrahim, Suhaib Abduljabbar Altamir. Individual biometrics pattern based artificial image analysis techniques. Numerical Algebra, Control & Optimization, 2021  doi: 10.3934/naco.2020056 [5] Jia Cai, Guanglong Xu, Zhensheng Hu. Sketch-based image retrieval via CAT loss with elastic net regularization. Mathematical Foundations of Computing, 2020, 3 (4) : 219-227. doi: 10.3934/mfc.2020013

2019 Impact Factor: 1.373