Foveated compressive imaging for low power vehicle fingerprinting and tracking in aerial imagery

Figure 1.  (Top) Flowchart of overall approach for adaptation of foveated measurements and signal representations. (Bottom) Details of online scene-adaptive reconstruction for the vehicle fingerprinting task. Left: Dynamic input scene. Middle: Reconstructed low-resolution background scene that is used to detect ROIs using anomalous motion detection. Right: Reconstructed high resolution ROIs using adapted dictionary overlaid with low-resolution background. This representation reduces the total number of measurements $M = M_{\rm Backg}+ M_{\rm ROI}$ needed for the task.

Figure 2.  The Manx Shearwater seabird [19] has multiple hierarchical levels of fovea (Right) for acquisition and tracking.

Figure 3.  Our foveated compressive sensing optical architecture generates a composite image frame consisting of low-resolution background contextual information and the high-resolution task-relevant regions of interest (ROIs). A fixed budget of $M$ measurements can be adaptively divided between the background and ROIs, allowing background resolution to be traded for higher resolution ROIs. By adapting both the measurement matrix and the dictionary to the ROIs, the number of measurements needed for a given level of task performance can be greatly reduced

Figure 4.  Simulation results comparing conventional imaging, conventional CS imaging, and foveated CS imaging. The conventional CS imaging reconstructs the image from random DCT measurements via $\ell_1$-minimization. Imaging results were all obtained using 3025 measurements of the scene but foveated compressive sensing achieved much higher effective resolution in the region of interest (ROI) than conventional imaging while also reconstructing the context around the ROI

Figure  .  Algorithm 1: Iterative reweighted subspace minimization algorithm that we use to find salient regions in images.

Figure 5.  Detection of moving vehicle ROIs in two frames using Actionable Saliency despite camera motions. [10]

Figure 6.  Detection of moving vehicle ROIs from images reconstructed from different numbers of compressive sensing measurements.

Figure 7.  Top: Example images of cars from OIRDS [18] aerial views used for training our dictionary. Bottom left: Learned tree-structured dictionary with example atoms from each level of the tree. Bottom right: Distribution of coefficients over the training set

Figure 8.  Patch-based compressive optical foveated architecture (COFA) optical system

Figure 9.  Hierarchical layered regions of interest (ROIs) for the vehicle tracking and fingerprinting task. Layer 1 is the background, Layer 2 is the road, and Layer 3 contains the moving vehicles on the road

Figure 10.  Contours of the minimax noise sensitivity $M^*(\delta,\rho)$ in the $(\delta,\rho)$ plane. $\delta=M/N$ is the subsampling rate and $\rho=K/M$ is the sparsity. The dotted black curve graphs the phase boundary $M^*(\delta,\rho{\rm MSE}(\delta))$. Above this curve, $M^*(\delta,\rho)=\infty$. The colored lines represent level sets of $M*(\delta,\rho)$. (From [8])

Figure 11.  Reduction in measurements needed over conventional compressive sensing as function of ROI resolution and size for 2 and 3 ROI layers

Figure 12.  Reconstruction SNR for CSUAV scenes with 1$\times$, 2$\times$, and 4$\times$ downsampling

Figure 13.  Example reconstructed CSUAV frames using the wmc + tree method. Sufficient resolution is maintained with $25\%$ of ($160\times 120$) measurements or $1/64$ of the number of Nyquist samples to detect ROIs corresponding to moving vehicles in the scene

Figure 14.  Left, Middle: Reconstruction SNR for vehicles displayed graphically and numerically. Right: Example reconstructed vehicles vs. number of measurements and measurement/dictionary types

Figure 15.  Fingerprinting task performance results for reconstructed vehicle windows from CSUAV motion imagery. Baseline performance on original input windows is 76.17$\%$

Figure 16.  COFA simulation framework for vehicle fingerprinting. For simplicity, Layer 2 (road ROIs) is not shown

Figure 17.  Left: 3-layer ROI hierarchy for COFA pipeline. Right: Multi-resolution composite reconstruction of a CSUAV video frame. Note the variable resolution in the patches corresponding to different ROI types. The Car ROIs have the highest resolution

Figure 18.  Reconstruction SNR and noise sensitivity for CSUAV Layer 1 (Background). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in 4$\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $6.25\%$ of measurements of Layer 1

Figure 19.  Reconstruction SNR and noise sensitivity for CSUAV Layer 2 (Road). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in $>8\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $6.25\%$ of measurements of Layer 2

Figure 20.  Reconstruction SNR and noise sensitivity for CSUAV Layer 3 (Cars). The results are averaged over all $16\times 16$ patches in 50 frames of CSUAV-11 video. Non-random measurements and structured dictionary resulted in $4\times$ fewer measurements for the same SNR compared to random measurements. Left: Reconstruction SNR (dB) vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Reconstruction SNR (dB) vs. added measurement noise level ($\%$) with fixed $25\%$ measurements of Layer 3

Figure 21.  Vehicle fingerprinting performance and noise sensitivity results for 3-layer pipeline. Left: Correct identification vs. measurements percentage $(M_{\rm ROI}/N_{\rm ROI})$. Right: Correct identification vs. added measurement noise level ($\%$) with fixed $25\%$ measurements of Layer 3