American Institute of Mathematical Sciences

Advanced
  • Previous Article
    Eliminating other-race effect for multi-ethnic facial expression recognition
  • MFC Home
  • This Issue
  • Next Article
    Comparisons of different methods for balanced data classification under the discrete non-local total variational framework
February  2019, 2(1): 29-41. doi: 10.3934/mfc.2019003

SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images

Mingyuan Mao 1, , Hewei Zhang 1, , Simeng Li 1, and  Baochang Zhang 1,2,,

1. 

Beihang University, Beijing, China
2. 

Shenzhen Academy of Aerospace Technology, Shenzhen, China

* Corresponding author: bczhang@buaa.edu.cn

Published  March 2019

SLAM (simultaneous localization and mapping) system can be implemented based on monocular, RGB-D and stereo cameras. RTAB-MAP is a SLAM system, which can build dense 3D map. In this paper, we present a novel method named SEMANTIC-RTAB-MAP (SRM) to implement a semantic SLAM system based on RTAB-MAP and deep learning. We use YOLOv2 network to detect target objects in 2D images, and then use depth information for precise localization of the targets and finally add semantic information into 3D point clouds. We apply SRM in different scenes, and the results show its higher running speed and accuracy.

Keywords: Semantic SLAM, YOLO, real-time, depth-image, region growing.
Mathematics Subject Classification: Primary: 58F15, 58F17; Secondary: 53C35.
Citation: Mingyuan Mao, Hewei Zhang, Simeng Li, Baochang Zhang. SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images. Mathematical Foundations of Computing, 2019, 2 (1) : 29-41. doi: 10.3934/mfc.2019003
References:
[1]

R. Q. Charles, H. Su, K. Mo and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017). doi: 10.1109/CVPR.2017.16.  Google Scholar

[2]

R. Girshick and J. Donahue, Trevor Darrell and Jitendra Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, (2013), 580-587. Google Scholar

[3]

K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778. doi: 10.1109/CVPR.2016.90.  Google Scholar

[4]

M. Labbé and F. Michaud, Long-term online multi-session graph-based splam with memory management, Autonomous Robots, 3 (2017), 1-18.   Google Scholar

[5]

M. Labbe and F. Michaud, Online global loop closure detection for large-scale multi-session graph-based SLAM, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2014), 2661-2666. doi: 10.1109/IROS.2014.6942926.  Google Scholar

[6]

M. Labbé and F. Michaud, Appearance-based loop closure detection for online large-scale and long-term operation, IEEE Transactions on Robotics, 29 (2013), 734-745.   Google Scholar

[7]

M. Labbe and F. Michaud, Memory management for real-time appearance-based loop closure detection, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2011), 1271-1276. doi: 10.1109/IROS.2011.6094602.  Google Scholar

[8]

X. Li and R. Belaroussi, Semi-dense 3d semantic mapping from monocular slam, 2016. Google Scholar

[9]

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed and C. Y. Fu, et al, SSD: Single Shot MultiBox Detector. European Conference on Computer Vision, Springer International Publishing, (2016), 21-37. Google Scholar

[10]

J. Mccormac, A. Handa, A. Davison and S. Leutenegger, Semanticfusion: dense 3d semantic mapping with convolutional neural networks, 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017. doi: 10.1109/ICRA.2017.7989538.  Google Scholar

[11]

R. Mur-Artal and J. D. Tardós, Probabilistic semi-dense mapping from highly accurate feature-based monocular SLAM, Robotics: Science and Systems, (2015), 1-9. doi: 10.15607/RSS.2015.XI.041.  Google Scholar

[12]

N. Otsu, A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man, and Cybernetics, 9 (1979), 62-66.  doi: 10.1109/TSMC.1979.4310076.  Google Scholar

[13]

J. Redmon, S. Divvala, R. Girshick and A. Farhadi, You only look once: Unified, real-time object detection, Computer Vision and Pattern Recognition, (2016), 779-788. doi: 10.1109/CVPR.2016.91.  Google Scholar

[14]

J. Redmon and A. Farhadi, YOLO9000: Better, faster, stronger, IEEE Conference on Computer Vision and Pattern Recognition, (2017), 6517-6525. doi: 10.1109/CVPR.2017.690.  Google Scholar

[15]

J. Redmon and A. Farhadi, Yolov3: an incremental improvement, 2018. Google Scholar

[16]

S. Ren, K. He, R. Girshick and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 1137-1149.  doi: 10.1109/TPAMI.2016.2577031.  Google Scholar

[17]

N. Sünderhauf, T. T. Pham, Y. Latif, M. Milford and I. Reid, Meaningful maps with object-oriented semantic mapping., Ieee/rsj International Conference on Intelligent Robots and Systems, IEEE, (2017), 5079-5085. Google Scholar

[18]

T. Whelan, S. Leutenegger, R. S. Moreno, B. Glocker and A. Davison, ElasticFusion: Dense SLAM Without A Pose Graph. Robotics: Science and Systems, 2015. doi: 10.15607/RSS.2015.XI.001.  Google Scholar

show all references

References:
[1]

R. Q. Charles, H. Su, K. Mo and L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017). doi: 10.1109/CVPR.2017.16.  Google Scholar

[2]

R. Girshick and J. Donahue, Trevor Darrell and Jitendra Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, (2013), 580-587. Google Scholar

[3]

K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778. doi: 10.1109/CVPR.2016.90.  Google Scholar

[4]

M. Labbé and F. Michaud, Long-term online multi-session graph-based splam with memory management, Autonomous Robots, 3 (2017), 1-18.   Google Scholar

[5]

M. Labbe and F. Michaud, Online global loop closure detection for large-scale multi-session graph-based SLAM, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2014), 2661-2666. doi: 10.1109/IROS.2014.6942926.  Google Scholar

[6]

M. Labbé and F. Michaud, Appearance-based loop closure detection for online large-scale and long-term operation, IEEE Transactions on Robotics, 29 (2013), 734-745.   Google Scholar

[7]

M. Labbe and F. Michaud, Memory management for real-time appearance-based loop closure detection, IEEE/RSJ International Conference on Intelligent Robots and Systems, (2011), 1271-1276. doi: 10.1109/IROS.2011.6094602.  Google Scholar

[8]

X. Li and R. Belaroussi, Semi-dense 3d semantic mapping from monocular slam, 2016. Google Scholar

[9]

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed and C. Y. Fu, et al, SSD: Single Shot MultiBox Detector. European Conference on Computer Vision, Springer International Publishing, (2016), 21-37. Google Scholar

[10]

J. Mccormac, A. Handa, A. Davison and S. Leutenegger, Semanticfusion: dense 3d semantic mapping with convolutional neural networks, 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017. doi: 10.1109/ICRA.2017.7989538.  Google Scholar

[11]

R. Mur-Artal and J. D. Tardós, Probabilistic semi-dense mapping from highly accurate feature-based monocular SLAM, Robotics: Science and Systems, (2015), 1-9. doi: 10.15607/RSS.2015.XI.041.  Google Scholar

[12]

N. Otsu, A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man, and Cybernetics, 9 (1979), 62-66.  doi: 10.1109/TSMC.1979.4310076.  Google Scholar

[13]

J. Redmon, S. Divvala, R. Girshick and A. Farhadi, You only look once: Unified, real-time object detection, Computer Vision and Pattern Recognition, (2016), 779-788. doi: 10.1109/CVPR.2016.91.  Google Scholar

[14]

J. Redmon and A. Farhadi, YOLO9000: Better, faster, stronger, IEEE Conference on Computer Vision and Pattern Recognition, (2017), 6517-6525. doi: 10.1109/CVPR.2017.690.  Google Scholar

[15]

J. Redmon and A. Farhadi, Yolov3: an incremental improvement, 2018. Google Scholar

[16]

S. Ren, K. He, R. Girshick and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 1137-1149.  doi: 10.1109/TPAMI.2016.2577031.  Google Scholar

[17]

N. Sünderhauf, T. T. Pham, Y. Latif, M. Milford and I. Reid, Meaningful maps with object-oriented semantic mapping., Ieee/rsj International Conference on Intelligent Robots and Systems, IEEE, (2017), 5079-5085. Google Scholar

[18]

T. Whelan, S. Leutenegger, R. S. Moreno, B. Glocker and A. Davison, ElasticFusion: Dense SLAM Without A Pose Graph. Robotics: Science and Systems, 2015. doi: 10.15607/RSS.2015.XI.001.  Google Scholar

Figure 1.  Performance of SRM. (First from Left) Original RGB-D image including a label of handbag. (Second from Left) Result of edge detection on the depth image. (Second from Right) Result of region growing. (First from Right) The corresponding 3D point cloud
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2.  Results by Li et. al. [8]
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 3.  The Structure of Memory Management. [7]
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 4.  YOLOv1 structure. [13]
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 5.  Results by YOLOv2
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 6.  Overview of SRM method
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 7.  Results by YOLOv2
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 8.  The flow chart of our precise localization
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 9.  (Top) Original RGB-D image. (Middle Left) Result of edge detection by Canny operator. (Middle Right) Result of region growing. (Bottom) Semantic RGB-D image
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 10.  The left is the original local point cloud of RTAB-MAP. The right is the corresponding point cloud of SRM. Two refrigerators are recognized by YOLOv2 and painted with green (we assume that green represents refrigerators)
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 11.  (First Column) Original RGB image. (Second Column) Edges of targets extracted in depth image by Canny operator. (Third Column) Corresponding local point clouds. The blue sticks in the point clouds in the first three rows are axes. Because we detected different targets separately, some different targets are painted the same color. When we detect them at the same time, we just need to assign different colors to different classes of objects
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 12.  The left is the original RGB image. The right is the corresponding point cloud. We paint the bottle red, the laptop blue and the handbag green. We don't show Edges of targets extracted in depth image here because SRM process different objects one by one, which means we don't have an image including all of their edges. The handbag is not shown completely in the point cloud because the Kinect2 didn't get the depth data of that area
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 13.  The performance of semi-dense 3D semantic mapping. [8] (Top) Original image. (Bottom) Corresponding point cloud. Red represents buildings. Purple represents cars. Green bounding boxes are added by us. They are not included in the original image
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 14.  The left is the original RGB image. The right is the point cloud
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 15.  The left is the original RGB image. The right is the point cloud
Figure Options

Download full-size image

Download as PowerPoint slide

[1]

José Madrid, João P. G. Ramos. On optimal autocorrelation inequalities on the real line. Communications on Pure & Applied Analysis, , () : -. doi: 10.3934/cpaa.2020271
[2]

Abdelghafour Atlas, Mostafa Bendahmane, Fahd Karami, Driss Meskine, Omar Oubbih. A nonlinear fractional reaction-diffusion system applied to image denoising and decomposition. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020321
[3]

Noah Stevenson, Ian Tice. A truncated real interpolation method and characterizations of screened Sobolev spaces. Communications on Pure & Applied Analysis, 2020, 19 (12) : 5509-5566. doi: 10.3934/cpaa.2020250
[4]

Jia Cai, Guanglong Xu, Zhensheng Hu. Sketch-based image retrieval via CAT loss with elastic net regularization. Mathematical Foundations of Computing, 2020, 3 (4) : 219-227. doi: 10.3934/mfc.2020013
[5]

Lars Grüne, Matthias A. Müller, Christopher M. Kellett, Steven R. Weller. Strict dissipativity for discrete time discounted optimal control problems. Mathematical Control & Related Fields, 2020  doi: 10.3934/mcrf.2020046
[6]

Serena Dipierro, Benedetta Pellacci, Enrico Valdinoci, Gianmaria Verzini. Time-fractional equations with reaction terms: Fundamental solutions and asymptotics. Discrete & Continuous Dynamical Systems - A, 2021, 41 (1) : 257-275. doi: 10.3934/dcds.2020137
[7]

Guido Cavallaro, Roberto Garra, Carlo Marchioro. Long time localization of modified surface quasi-geostrophic equations. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020336
[8]

Cuicui Li, Lin Zhou, Zhidong Teng, Buyu Wen. The threshold dynamics of a discrete-time echinococcosis transmission model. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020339
[9]

Awais Younus, Zoubia Dastgeer, Nudrat Ishaq, Abdul Ghaffar, Kottakkaran Sooppy Nisar, Devendra Kumar. On the observability of conformable linear time-invariant control systems. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020444
[10]

Hoang The Tuan. On the asymptotic behavior of solutions to time-fractional elliptic equations driven by a multiplicative white noise. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020318
[11]

Haixiang Yao, Ping Chen, Miao Zhang, Xun Li. Dynamic discrete-time portfolio selection for defined contribution pension funds with inflation risk. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2020166
[12]

Youshan Tao, Michael Winkler. Critical mass for infinite-time blow-up in a haptotaxis system with nonlinear zero-order interaction. Discrete & Continuous Dynamical Systems - A, 2021, 41 (1) : 439-454. doi: 10.3934/dcds.2020216
[13]

Jianquan Li, Xin Xie, Dian Zhang, Jia Li, Xiaolin Lin. Qualitative analysis of a simple tumor-immune system with time delay of tumor action. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020341
[14]

Reza Lotfi, Zahra Yadegari, Seyed Hossein Hosseini, Amir Hossein Khameneh, Erfan Babaee Tirkolaee, Gerhard-Wilhelm Weber. A robust time-cost-quality-energy-environment trade-off with resource-constrained in project management: A case study for a bridge construction project. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2020158

 Impact Factor: 

Tools

Article outline

Figures and Tables

[Back to Top]

Copyright © 2020 American Institute of Mathematical Sciences