# American Institute of Mathematical Sciences

• Previous Article
Variational source conditions and stability estimates for inverse electromagnetic medium scattering problems
• IPI Home
• This Issue
• Next Article
Non-linear Tikhonov regularization in Banach spaces for inverse scattering from anisotropic penetrable media
February  2017, 11(1): 177-202. doi: 10.3934/ipi.2017009

## Foveated compressive imaging for low power vehicle fingerprinting and tracking in aerial imagery

 HRL Laboratories LLC, 3011 Malibu Canyon Road, Malibu, CA 90265-4797, USA

Received  January 2015 Revised  September 2016 Published  January 2017

Fund Project: This work is supported by the Defense Advanced Research Projects Agency and SPAWAR Systems Center Pacific under Contract No. N66001-11-C-4001.

We describe a foveated compressive sensing approach for image analysis applications that utilizes knowledge of the task to be performed to reduce the number of required sensor measurements and sensor size, weight, and power (SWAP) compared to conventional Nyquist sampling and compressive sensing-based approaches. Our Compressive Optical Foveated Architecture (COFA) adapts the dictionary and compressive measurements to structure and sparsity in the signal, task, and scene by reducing measurement and dictionary mutual coherence and increasing sparsity using principles of actionable information and foveated compressive sensing. Actionable information is used to extract task-relevant regions of interest (ROIs) from a low-resolution scene analysis by eliminating the effects of nuisances for occlusion and anomalous motion detection. From the extracted ROIs, preferential measurements are taken using foveation as part of the compressive sensing adaptation process. The task-specific measurement matrix is optimized by using a novel saliency-weighted coherence minimization with respect to the learned signal dictionary. This incorporates the relative usage of the atoms in the dictionary. We utilize a patch-based method to learn the signal priors. A tree-structured dictionary of image patches using K-SVD is learned which can sparsely represent any given image patch with the tree structure. We have implemented COFA in an end-to-end simulation of a vehicle fingerprinting task for aerial surveillance using foveated compressive measurements adapted to hierarchical ROIs consisting of background, roads, and vehicles. Our results show 113× reduction in measurements over conventional sensing and 28× reduction over compressive sensing using random measurements.

Citation: Kang-Yu Ni, Shankar Rao, Yuri Owechko. Foveated compressive imaging for low power vehicle fingerprinting and tracking in aerial imagery. Inverse Problems & Imaging, 2017, 11 (1) : 177-202. doi: 10.3934/ipi.2017009
##### References:

show all references

##### References:
(Top) Flowchart of overall approach for adaptation of foveated measurements and signal representations. (Bottom) Details of online scene-adaptive reconstruction for the vehicle fingerprinting task. Left: Dynamic input scene. Middle: Reconstructed low-resolution background scene that is used to detect ROIs using anomalous motion detection. Right: Reconstructed high resolution ROIs using adapted dictionary overlaid with low-resolution background. This representation reduces the total number of measurements $M = M_{\rm Backg}+ M_{\rm ROI}$ needed for the task.
The Manx Shearwater seabird [19] has multiple hierarchical levels of fovea (Right) for acquisition and tracking.
Our foveated compressive sensing optical architecture generates a composite image frame consisting of low-resolution background contextual information and the high-resolution task-relevant regions of interest (ROIs). A fixed budget of $M$ measurements can be adaptively divided between the background and ROIs, allowing background resolution to be traded for higher resolution ROIs. By adapting both the measurement matrix and the dictionary to the ROIs, the number of measurements needed for a given level of task performance can be greatly reduced
Simulation results comparing conventional imaging, conventional CS imaging, and foveated CS imaging. The conventional CS imaging reconstructs the image from random DCT measurements via $\ell_1$-minimization. Imaging results were all obtained using 3025 measurements of the scene but foveated compressive sensing achieved much higher effective resolution in the region of interest (ROI) than conventional imaging while also reconstructing the context around the ROI
Algorithm 1: Iterative reweighted subspace minimization algorithm that we use to find salient regions in images.
Detection of moving vehicle ROIs in two frames using Actionable Saliency despite camera motions. [10]
Detection of moving vehicle ROIs from images reconstructed from different numbers of compressive sensing measurements.
Top: Example images of cars from OIRDS [18] aerial views used for training our dictionary. Bottom left: Learned tree-structured dictionary with example atoms from each level of the tree. Bottom right: Distribution of coefficients over the training set
Patch-based compressive optical foveated architecture (COFA) optical system
Hierarchical layered regions of interest (ROIs) for the vehicle tracking and fingerprinting task. Layer 1 is the background, Layer 2 is the road, and Layer 3 contains the moving vehicles on the road
Contours of the minimax noise sensitivity $M^*(\delta,\rho)$ in the $(\delta,\rho)$ plane. $\delta=M/N$ is the subsampling rate and $\rho=K/M$ is the sparsity. The dotted black curve graphs the phase boundary $M^*(\delta,\rho{\rm MSE}(\delta))$. Above this curve, $M^*(\delta,\rho)=\infty$. The colored lines represent level sets of $M*(\delta,\rho)$. (From [8])
Reduction in measurements needed over conventional compressive sensing as function of ROI resolution and size for 2 and 3 ROI layers
Reconstruction SNR for CSUAV scenes with 1$\times$, 2$\times$, and 4$\times$ downsampling
Example reconstructed CSUAV frames using the wmc + tree method. Sufficient resolution is maintained with $25\%$ of ($160\times 120$) measurements or $1/64$ of the number of Nyquist samples to detect ROIs corresponding to moving vehicles in the scene
Left, Middle: Reconstruction SNR for vehicles displayed graphically and numerically. Right: Example reconstructed vehicles vs. number of measurements and measurement/dictionary types
Fingerprinting task performance results for reconstructed vehicle windows from CSUAV motion imagery. Baseline performance on original input windows is 76.17$\%$
COFA simulation framework for vehicle fingerprinting. For simplicity, Layer 2 (road ROIs) is not shown
Left: 3-layer ROI hierarchy for COFA pipeline. Right: Multi-resolution composite reconstruction of a CSUAV video frame. Note the variable resolution in the patches corresponding to different ROI types. The Car ROIs have the highest resolution
Reconstruction SNR and noise sensitivity for CSUAV Layer 1 (Background). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in 4$\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $6.25\%$ of measurements of Layer 1
Reconstruction SNR and noise sensitivity for CSUAV Layer 2 (Road). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in $>8\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $6.25\%$ of measurements of Layer 2
Reconstruction SNR and noise sensitivity for CSUAV Layer 3 (Cars). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in $4\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $25\%$ measurements of Layer 3
Vehicle fingerprinting performance and noise sensitivity results for 3-layer pipeline. Left: Correct identification vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Correct identification vs. added measurement noise level ($\%$) with fixed $25\%$ measurements of Layer 3
Tested methods and reconstruction algorithms
 Method Measurement Dictionary rand + flat random Gaussian orthonormal measurements (flat) ksvd dictionary rand + tree random Gaussian orthonormal measurements hierarchical (tree) dictionary mc + flat minimum coherence measurements (flat) ksvd dictionary mc + tree minimum coherence measurements hierarchical (tree) dictionary wmc + tree weighted minimum coherence measurements hierarchical (tree) dictionary
 Method Measurement Dictionary rand + flat random Gaussian orthonormal measurements (flat) ksvd dictionary rand + tree random Gaussian orthonormal measurements hierarchical (tree) dictionary mc + flat minimum coherence measurements (flat) ksvd dictionary mc + tree minimum coherence measurements hierarchical (tree) dictionary wmc + tree weighted minimum coherence measurements hierarchical (tree) dictionary
 [1] Shengxin Zhu, Tongxiang Gu, Xingping Liu. AIMS: Average information matrix splitting. Mathematical Foundations of Computing, 2020, 3 (4) : 301-308. doi: 10.3934/mfc.2020012 [2] Hui Gao, Jian Lv, Xiaoliang Wang, Liping Pang. An alternating linearization bundle method for a class of nonconvex optimization problem with inexact information. Journal of Industrial & Management Optimization, 2021, 17 (2) : 805-825. doi: 10.3934/jimo.2019135 [3] Xuemei Chen, Julia Dobrosotskaya. Inpainting via sparse recovery with directional constraints. Mathematical Foundations of Computing, 2020, 3 (4) : 229-247. doi: 10.3934/mfc.2020025 [4] Jing Qin, Shuang Li, Deanna Needell, Anna Ma, Rachel Grotheer, Chenxi Huang, Natalie Durgin. Stochastic greedy algorithms for multiple measurement vectors. Inverse Problems & Imaging, 2021, 15 (1) : 79-107. doi: 10.3934/ipi.2020066 [5] Sören Bartels, Jakob Keck. Adaptive time stepping in elastoplasticity. Discrete & Continuous Dynamical Systems - S, 2021, 14 (1) : 71-88. doi: 10.3934/dcdss.2020323 [6] Stefan Ruschel, Serhiy Yanchuk. The spectrum of delay differential equations with multiple hierarchical large delays. Discrete & Continuous Dynamical Systems - S, 2021, 14 (1) : 151-175. doi: 10.3934/dcdss.2020321 [7] Wai-Ki Ching, Jia-Wen Gu, Harry Zheng. On correlated defaults and incomplete information. Journal of Industrial & Management Optimization, 2021, 17 (2) : 889-908. doi: 10.3934/jimo.2020003 [8] Nicolas Rougerie. On two properties of the Fisher information. Kinetic & Related Models, 2021, 14 (1) : 77-88. doi: 10.3934/krm.2020049 [9] Nicholas Geneva, Nicholas Zabaras. Multi-fidelity generative deep learning turbulent flows. Foundations of Data Science, 2020, 2 (4) : 391-428. doi: 10.3934/fods.2020019 [10] Peizhao Yu, Guoshan Zhang, Yi Zhang. Decoupling of cubic polynomial matrix systems. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 13-26. doi: 10.3934/naco.2020012 [11] Li-Bin Liu, Ying Liang, Jian Zhang, Xiaobing Bao. A robust adaptive grid method for singularly perturbed Burger-Huxley equations. Electronic Research Archive, 2020, 28 (4) : 1439-1457. doi: 10.3934/era.2020076 [12] Ke Su, Yumeng Lin, Chun Xu. A new adaptive method to nonlinear semi-infinite programming. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2021012 [13] Tengfei Yan, Qunying Liu, Bowen Dou, Qing Li, Bowen Li. An adaptive dynamic programming method for torque ripple minimization of PMSM. Journal of Industrial & Management Optimization, 2021, 17 (2) : 827-839. doi: 10.3934/jimo.2019136 [14] Honglin Yang, Jiawu Peng. Coordinating a supply chain with demand information updating. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2020181 [15] Kengo Nakai, Yoshitaka Saiki. Machine-learning construction of a model for a macroscopic fluid variable using the delay-coordinate of a scalar observable. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1079-1092. doi: 10.3934/dcdss.2020352 [16] Parikshit Upadhyaya, Elias Jarlebring, Emanuel H. Rubensson. A density matrix approach to the convergence of the self-consistent field iteration. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 99-115. doi: 10.3934/naco.2020018 [17] Yueyang Zheng, Jingtao Shi. A stackelberg game of backward stochastic differential equations with partial information. Mathematical Control & Related Fields, 2020  doi: 10.3934/mcrf.2020047 [18] Chuan Ding, Da-Hai Li. Angel capitalists exit decisions under information asymmetry: IPO or acquisitions. Journal of Industrial & Management Optimization, 2021, 17 (1) : 369-392. doi: 10.3934/jimo.2019116 [19] Min Xi, Wenyu Sun, Jun Chen. Survey of derivative-free optimization. Numerical Algebra, Control & Optimization, 2020, 10 (4) : 537-555. doi: 10.3934/naco.2020050 [20] Gang Bao, Mingming Zhang, Bin Hu, Peijun Li. An adaptive finite element DtN method for the three-dimensional acoustic scattering problem. Discrete & Continuous Dynamical Systems - B, 2021, 26 (1) : 61-79. doi: 10.3934/dcdsb.2020351

2019 Impact Factor: 1.373