# American Institute of Mathematical Sciences

August  2021, 15(4): 763-785. doi: 10.3934/ipi.2021013

## Synthetic-Aperture Radar image based positioning in GPS-denied environments using Deep Cosine Similarity Neural Networks

 1 Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL 32611, USA 2 Department of Information Systems and Analytics, Miami University, Oxford, OH 45056, USA 3 Air Force Research Laboratory (AFRL/RWWI), Eglin Air Force Base, FL 32542, USA

* Corresponding author: Maciej Rysz

Received  September 2020 Revised  November 2020 Published  January 2021

Fund Project: Distribution A: Approved for public release, distribution is unlimited, 96TW-2020-0170

Navigating unmanned aerial vehicles in precarious environments is of great importance. It is necessary to rely on alternative information processing techniques to attain spatial information that is required for navigation in such settings. This paper introduces a novel deep learning-based approach for navigating that exclusively relies on synthetic aperture radar (SAR) images. The proposed method utilizes deep neural networks (DNNs) for image matching, retrieval, and registration. To this end, we introduce Deep Cosine Similarity Neural Networks (DCSNNs) for mapping SAR images to a global descriptive feature vector. We also introduce a fine-tuning algorithm for DCSNNs, and DCSNNs are used to generate a database of feature vectors for SAR images that span a geographic area of interest, which, in turn, are compared against a feature vector of an inquiry image. Images similar to the inquiry are retrieved from the database by using a scalable distance measure between the feature vector outputs of DCSNN. Methods for reranking the retrieved SAR images that are used to update position coordinates of an inquiry SAR image by estimating from the best retrieved SAR image are also introduced. Numerical experiments comparing with baselines on the Polarimetric SAR (PolSAR) images are presented.

Citation: Seonho Park, Maciej Rysz, Kaitlin L. Fair, Panos M. Pardalos. Synthetic-Aperture Radar image based positioning in GPS-denied environments using Deep Cosine Similarity Neural Networks. Inverse Problems & Imaging, 2021, 15 (4) : 763-785. doi: 10.3934/ipi.2021013
##### References:

show all references

##### References:
Overview of the system for SAR aided navigating by SAR image representation, matching and registration
Example of a patch graph. (a) a SAR map from the UAVSAR dataset (b) patches extracted from the SAR map and its associated graph (c) visualization of the adjacency matrix. White cells show zero edge values whereas black cells show nonzero values
Overview of Product Quantization (PQ) with DCSNN
Comparison between keypoints generated by (a) SAR-SIFT and (b) SIFT
(a) SAR-SIFT based keypoints on two adjacent SAR patches (b) image matching of the keypoints via RANSAC
PolSAR data from UAVSAR dataset for our experiments. (From left) VVVV(R), HVHV(G), HHHH(B) channels, and total RGB image. Best viewed in color. (a) PolSAR map for Hayward Fault Zone in California, US. (b) PolSAR map for Yukon–Kuskokwim Delta in Alaska, US
Precision recall curves on Hayward Fault Zone (Top rows) and Yukon–Kuskokwim Delta (Bottom rows) PolSAR maps. From left column, the feature length is $24$, $48$, $96$, and $120$. AlexNet is used as a backbone
Examples of the retrieved SAR patches before and after reranking processes. First column represents examples of inquiry SAR patches. The first two rows ((a) and (b)) are from Hayward Fault Zone PolSAR and the later two rows ((c) and (d)) are from Yukon-Kuskokwim Delta PolSAR data. Having green box indicates it is correctly retrieved, whereas having red box indicates that it is incorrectly retrieved
Descriptions of the PolSAR map from UAVSAR [1]
 Region (Dataset Name) Usage Acquired Date Pixel Size (Height×Width) #Patches Hayward Fault Zone Training Oct 9, 2018 $23506\times3300$ 6440 Test May 30, 2019 $23476\times3300$ 6412 Yukon–Kuskokwim Delta Training Aug 28, 2018 $19066\times3300$ 5180 Test Sep 17, 2019 $19148\times3300$ 5208
 Region (Dataset Name) Usage Acquired Date Pixel Size (Height×Width) #Patches Hayward Fault Zone Training Oct 9, 2018 $23506\times3300$ 6440 Test May 30, 2019 $23476\times3300$ 6412 Yukon–Kuskokwim Delta Training Aug 28, 2018 $19066\times3300$ 5180 Test Sep 17, 2019 $19148\times3300$ 5208
Mean average precision (mAP) results of DCSNN and binary hashing methods before reranking on Hayward Fault Zone PolSAR map
 Methods Feature length $l$ AlexNet [21] VGG-11 [40] Pretrained (BH) $l=24$ 0.0478 0.0313 $l=48$ 0.0761 0.0522 $l=96$ 0.1547 0.1200 $l=120$ 0.1566 0.1219 Pretrained (PQ+AQD) $l=24$ 0.2332 0.1706 $l=48$ 0.3208 0.2055 $l=96$ 0.3231 0.2676 $l=120$ 0.3856 0.3161 DHN [50] $l=24$ 0.1182 0.1255 $l=48$ 0.1642 0.1693 $l=96$ 0.2274 0.2153 $l=120$ 0.3202 0.2449 DPSH [23] $l=24$ 0.0895 0.1220 $l=48$ 0.2825 0.2632 $l=96$ 0.4545 0.4005 $l=120$ 0.5213 0.4334 DHNN-L2 [24] $l=24$ 0.0451 0.1147 $l=48$ 0.0683 0.1291 $l=96$ 0.2044 0.1329 $l=120$ 0.2190 0.1304 DCSNN (ours) $l=24$ 0.2519 0.4889 $l=48$ 0.6145 0.6301 $l=96$ 0.6481 0.5783 $l=120$ 0.6813 0.5819
 Methods Feature length $l$ AlexNet [21] VGG-11 [40] Pretrained (BH) $l=24$ 0.0478 0.0313 $l=48$ 0.0761 0.0522 $l=96$ 0.1547 0.1200 $l=120$ 0.1566 0.1219 Pretrained (PQ+AQD) $l=24$ 0.2332 0.1706 $l=48$ 0.3208 0.2055 $l=96$ 0.3231 0.2676 $l=120$ 0.3856 0.3161 DHN [50] $l=24$ 0.1182 0.1255 $l=48$ 0.1642 0.1693 $l=96$ 0.2274 0.2153 $l=120$ 0.3202 0.2449 DPSH [23] $l=24$ 0.0895 0.1220 $l=48$ 0.2825 0.2632 $l=96$ 0.4545 0.4005 $l=120$ 0.5213 0.4334 DHNN-L2 [24] $l=24$ 0.0451 0.1147 $l=48$ 0.0683 0.1291 $l=96$ 0.2044 0.1329 $l=120$ 0.2190 0.1304 DCSNN (ours) $l=24$ 0.2519 0.4889 $l=48$ 0.6145 0.6301 $l=96$ 0.6481 0.5783 $l=120$ 0.6813 0.5819
mAP results of the DCSNN and binary hashing methods before reranking on Yukon–Kuskokwim Delta PolSAR map
 Methods Feature length $l$ AlexNet [21] VGG-11 [40] Pretrained (BH) $l=24$ 0.0566 0.0396 $l=48$ 0.1048 0.0550 $l=96$ 0.1927 0.1227 $l=120$ 0.2196 0.1174 Pretrained (PQ+AQD) $l=24$ 0.2684 0.1736 $l=48$ 0.3580 0.2091 $l=96$ 0.3718 0.2579 $l=120$ 0.4184 0.2690 DHN [50] $l=24$ 0.1219 0.1610 $l=48$ 0.2072 0.1910 $l=96$ 0.2715 0.2447 $l=120$ 0.3627 0.2239 DPSH [23] $l=24$ 0.1347 0.1170 $l=48$ 0.2371 0.2516 $l=96$ 0.3649 0.3281 $l=120$ 0.4098 0.3331 DHNN-L2 [24] $l=24$ 0.0621 0.1132 $l=48$ 0.1556 0.1812 $l=96$ 0.2916 0.3166 $l=120$ 0.3227 0.2822 DCSNN (ours) $l=24$ 0.4393 0.4324 $l=48$ 0.5424 0.5196 $l=96$ 0.5734 0.4913 $l=120$ 0.5996 0.4831
 Methods Feature length $l$ AlexNet [21] VGG-11 [40] Pretrained (BH) $l=24$ 0.0566 0.0396 $l=48$ 0.1048 0.0550 $l=96$ 0.1927 0.1227 $l=120$ 0.2196 0.1174 Pretrained (PQ+AQD) $l=24$ 0.2684 0.1736 $l=48$ 0.3580 0.2091 $l=96$ 0.3718 0.2579 $l=120$ 0.4184 0.2690 DHN [50] $l=24$ 0.1219 0.1610 $l=48$ 0.2072 0.1910 $l=96$ 0.2715 0.2447 $l=120$ 0.3627 0.2239 DPSH [23] $l=24$ 0.1347 0.1170 $l=48$ 0.2371 0.2516 $l=96$ 0.3649 0.3281 $l=120$ 0.4098 0.3331 DHNN-L2 [24] $l=24$ 0.0621 0.1132 $l=48$ 0.1556 0.1812 $l=96$ 0.2916 0.3166 $l=120$ 0.3227 0.2822 DCSNN (ours) $l=24$ 0.4393 0.4324 $l=48$ 0.5424 0.5196 $l=96$ 0.5734 0.4913 $l=120$ 0.5996 0.4831
mAP values before and after reranking with SAR-SIFT or SIFT on Hayward Fault Zone PolSAR map
 CNN backbone Feature length $l$ Before reranking After reranking (SAR-SIFT/SIFT) AlexNet [21] $l=24$ 0.2519 0.4074/0.3533 $l=48$ 0.6145 0.7394/0.6850 $l=96$ 0.6481 0.7548/0.6998 $l=120$ 0.6813 0.7760/0.7252 VGG-11 [40] $l=24$ 0.4889 0.6512/0.5813 $l=48$ 0.6301 0.7540/0.6923 $l=96$ 0.5783 0.6799/0.6231 $l=120$ 0.5819 0.6787/0.6216
 CNN backbone Feature length $l$ Before reranking After reranking (SAR-SIFT/SIFT) AlexNet [21] $l=24$ 0.2519 0.4074/0.3533 $l=48$ 0.6145 0.7394/0.6850 $l=96$ 0.6481 0.7548/0.6998 $l=120$ 0.6813 0.7760/0.7252 VGG-11 [40] $l=24$ 0.4889 0.6512/0.5813 $l=48$ 0.6301 0.7540/0.6923 $l=96$ 0.5783 0.6799/0.6231 $l=120$ 0.5819 0.6787/0.6216
mAP values of the DCSNN before and after reranking with SAR-SIFT or SIFT on Yukon–Kuskokwim Delta PolSAR map
 CNN backbone Feature length $l$ Before reranking After reranking (SAR-SIFT/SIFT) AlexNet [21] $l=24$ 0.4393 0.5831/0.5965 $l=48$ 0.5424 0.6591/0.6693 $l=96$ 0.5734 0.6782/0.6888 $l=120$ 0.5996 0.7021/0.7123 VGG-11 [40] $l=24$ 0.4324 0.5909/0.6030 $l=48$ 0.5196 0.6418/0.6521 $l=96$ 0.4913 0.5939/0.6036 $l=120$ 0.4831 0.5840/0.5940
 CNN backbone Feature length $l$ Before reranking After reranking (SAR-SIFT/SIFT) AlexNet [21] $l=24$ 0.4393 0.5831/0.5965 $l=48$ 0.5424 0.6591/0.6693 $l=96$ 0.5734 0.6782/0.6888 $l=120$ 0.5996 0.7021/0.7123 VGG-11 [40] $l=24$ 0.4324 0.5909/0.6030 $l=48$ 0.5196 0.6418/0.6521 $l=96$ 0.4913 0.5939/0.6036 $l=120$ 0.4831 0.5840/0.5940
Positioning accuracy examples
 Inquiry SAR Patch Actual Coordinates [$deg$] Estimated Coordinates [$deg$] Error [$m$] Fig. 8(a) 38.0625, -122.2733 38.0625, -122.2734 5.7288 Fig. 8(b) 37.9836, -122.3599 37.9836, -122.3600 5.7347 Fig. 8(c) 61.0926, -164.1878 61.0926, -164.1879 4.2529 Fig. 8(d) 61.0808, -164.1208 61.0808, -164.1208 5.0970
 Inquiry SAR Patch Actual Coordinates [$deg$] Estimated Coordinates [$deg$] Error [$m$] Fig. 8(a) 38.0625, -122.2733 38.0625, -122.2734 5.7288 Fig. 8(b) 37.9836, -122.3599 37.9836, -122.3600 5.7347 Fig. 8(c) 61.0926, -164.1878 61.0926, -164.1879 4.2529 Fig. 8(d) 61.0808, -164.1208 61.0808, -164.1208 5.0970
Mean and standard deviation of positioning error results
 Data Name Success Cases Ratio [%] Distance Error [$m$] Hayward Fault Zone 98.50 4.9635$\pm$0.1755 Yukon–Kuskokwim Delta 97.70 4.9522$\pm$0.4038
 Data Name Success Cases Ratio [%] Distance Error [$m$] Hayward Fault Zone 98.50 4.9635$\pm$0.1755 Yukon–Kuskokwim Delta 97.70 4.9522$\pm$0.4038
 [1] Jean-Michel Morel, Guoshen Yu. Is SIFT scale invariant?. Inverse Problems & Imaging, 2011, 5 (1) : 115-136. doi: 10.3934/ipi.2011.5.115 [2] Zhihua Zhang, Naoki Saito. PHLST with adaptive tiling and its application to antarctic remote sensing image approximation. Inverse Problems & Imaging, 2014, 8 (1) : 321-337. doi: 10.3934/ipi.2014.8.321 [3] Kaitlyn (Voccola) Muller. SAR correlation imaging and anisotropic scattering. Inverse Problems & Imaging, 2018, 12 (3) : 697-731. doi: 10.3934/ipi.2018030 [4] Dana Paquin, Doron Levy, Eduard Schreibmann, Lei Xing. Multiscale Image Registration. Mathematical Biosciences & Engineering, 2006, 3 (2) : 389-418. doi: 10.3934/mbe.2006.3.389 [5] Zhao Yi, Justin W. L. Wan. An inviscid model for nonrigid image registration. Inverse Problems & Imaging, 2011, 5 (1) : 263-284. doi: 10.3934/ipi.2011.5.263 [6] Zbigniew Gomolka, Boguslaw Twarog, Jacek Bartman. Improvement of image processing by using homogeneous neural networks with fractional derivatives theorem. Conference Publications, 2011, 2011 (Special) : 505-514. doi: 10.3934/proc.2011.2011.505 [7] Yuantian Xia, Juxiang Zhou, Tianwei Xu, Wei Gao. An improved deep convolutional neural network model with kernel loss function in image classification. Mathematical Foundations of Computing, 2020, 3 (1) : 51-64. doi: 10.3934/mfc.2020005 [8] Dana Paquin, Doron Levy, Lei Xing. Hybrid multiscale landmark and deformable image registration. Mathematical Biosciences & Engineering, 2007, 4 (4) : 711-737. doi: 10.3934/mbe.2007.4.711 [9] Christiane Pöschl, Jan Modersitzki, Otmar Scherzer. A variational setting for volume constrained image registration. Inverse Problems & Imaging, 2010, 4 (3) : 505-522. doi: 10.3934/ipi.2010.4.505 [10] Raluca Felea, Romina Gaburro, Allan Greenleaf, Clifford Nolan. Microlocal analysis of Doppler synthetic aperture radar. Inverse Problems & Imaging, 2019, 13 (6) : 1283-1307. doi: 10.3934/ipi.2019056 [11] Kateřina Škardová, Tomáš Oberhuber, Jaroslav Tintěra, Radomír Chabiniok. Signed-distance function based non-rigid registration of image series with varying image intensity. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1145-1160. doi: 10.3934/dcdss.2020386 [12] Marcus Wagner. A direct method for the solution of an optimal control problem arising from image registration. Numerical Algebra, Control & Optimization, 2012, 2 (3) : 487-510. doi: 10.3934/naco.2012.2.487 [13] Angel Angelov, Marcus Wagner. Multimodal image registration by elastic matching of edge sketches via optimal control. Journal of Industrial & Management Optimization, 2014, 10 (2) : 567-590. doi: 10.3934/jimo.2014.10.567 [14] Yangang Chen, Justin W. L. Wan. Numerical method for image registration model based on optimal mass transport. Inverse Problems & Imaging, 2018, 12 (2) : 401-432. doi: 10.3934/ipi.2018018 [15] Jiying Liu, Jubo Zhu, Fengxia Yan, Zenghui Zhang. Compressive sampling and $l_1$ minimization for SAR imaging with low sampling rate. Inverse Problems & Imaging, 2013, 7 (4) : 1295-1305. doi: 10.3934/ipi.2013.7.1295 [16] Mikhail Gilman, Semyon Tsynkov. Statistical characterization of scattering delay in synthetic aperture radar imaging. Inverse Problems & Imaging, 2020, 14 (3) : 511-533. doi: 10.3934/ipi.2020024 [17] Venkateswaran P. Krishnan, Eric Todd Quinto. Microlocal aspects of common offset synthetic aperture radar imaging. Inverse Problems & Imaging, 2011, 5 (3) : 659-674. doi: 10.3934/ipi.2011.5.659 [18] Graciela Canziani, Rosana Ferrati, Claudia Marinelli, Federico Dukatz. Artificial neural networks and remote sensing in the analysis of the highly variable Pampean shallow lakes. Mathematical Biosciences & Engineering, 2008, 5 (4) : 691-711. doi: 10.3934/mbe.2008.5.691 [19] Rongliang Chen, Jizu Huang, Xiao-Chuan Cai. A parallel domain decomposition algorithm for large scale image denoising. Inverse Problems & Imaging, 2019, 13 (6) : 1259-1282. doi: 10.3934/ipi.2019055 [20] Jia Cai, Guanglong Xu, Zhensheng Hu. Sketch-based image retrieval via CAT loss with elastic net regularization. Mathematical Foundations of Computing, 2020, 3 (4) : 219-227. doi: 10.3934/mfc.2020013

2019 Impact Factor: 1.373

## Tools

Article outline

Figures and Tables