Let's enhance: A deep learning approach to extreme deblurring of text images

Figure 1.  Examples from the HDC dataset with different blur levels (left and right column). The severity of blurring increases with each level (20 in total, ranging from 0 to 19). The center column shows the corresponding reconstructions with our deep-learning-based pipeline (examples taken from the validation set)

Figure 2.  Schematic depiction of our deblurring pipeline. Top left (green): Data is synthesized for augmenting the available HDC training data (the details of the background removal step described in Section 3.1 are omitted here; see Fig. 6 for a precise depiction of that step). Center (purple): The reconstruction pipeline consists of an inverse lens distortion and a modified U-Net architecture. Bottom: The modified architecture differs from the vanilla U-Net (cf. [41]) by adding an introductory pooling layer and additional down- and up-sampling levels, to increase the overall receptive field

Figure 3.  Visualization of the coordinate deformation caused by radial lens distortion

Figure 4.  Illustration of the HDC experimental setup taken from [46]. Two identical photo cameras target the same e-ink display with the help of a half-transparent beamsplitter mirror

Figure 5.  Examples of HDC training data pairs of sharp and blurred images showing text (top row) and a calibration target (second row). Examples of the test data (third row) and sanity check data (bottom row) were unknown to the participants before the submission deadline

Figure 6.  Illustration of the forward blur model with background removal and addition. The background (bg) $ \mathbf{x}_0 $ and its corresponding blurred version $ \mathbf{y}_0 $ are estimated from the provided calibration target showing a single central point. They are subtracted and added before and after the discrete convolution with the estimated blur kernel, respectively

Figure 7.  Estimated kernels $ \mathbf{k}_B $ for blur levels 4, 9, 14, and 19 with enhanced contrast. The octagonal shape is typical for the polygonal shutter lenses of modern apertures, indicating that these estimated kernels reflect the underlying physical reality

Figure 8.  Examples of sharp and blurred image pairs from the (a) original HDC data, (b) synthetic HDC data, and (c) synthetic sanity check data. All blurred images correspond to the same blur level

Figure 9.  Example reconstruction results (right column) together with the sharp ground-truth images (left column) and blurry input images (middle column) for the blur levels 4, 9, 14, and 19. For the levels 14 and 19, both fonts (upper: Times New Roman, lower: Verdana) are shown. The right column shows the OCR Score for these particular samples, and for the sake of completeness, also the standard evaluation metrics SSIM and PSNR are reported; see Fig. 14, Fig. 19 and Fig. 20 for corresponding average scores

Figure 10.  Example reconstructions of two out-of-distribution images from the HDC sanity check data, shown for blur levels 4, 9, 14, and 19

Figure 11.  Summary of results from the HDC. The figure plots the blur level against the average OCR scores achieved by each participating team on the HDC test set. Our winning submission is highlighted as the bold line. Note that some teams have submitted multiple methods, and we took the most accurate one in each case

Figure 12.  Comparison of a blurry image from the HDC dataset with the results of two simulated forward models: a simple one, using only a convolution with a single spatially invariant blur kernel, and the one from Section 3.1, using a spatially variant blur kernel that accounts for radial lens distortion

Figure 13.  Visualization of the estimated radial lens distortion and the corresponding inverse distortion, shown for the blur levels 4, 9, 14, and 19

Figure 14.  Summary of OCR scores of all ablation studies presented in Section 4.3. Analogously to Fig. 11, the OCR scores are computed as the average over the challenge test set. Each line corresponds to a different modification of our final challenge submission, which is highlighted as the bold line. For analogous plots reporting SSIM and PSNR, see Fig. 19 and Fig. 20 in Appendix A.1

Figure 15.  Comparison of deblurring networks trained only on synthetic data ( = only pre-training), trained only on the original HDC data ( = only fine-tuning), and trained as in Section 3.4 ( = both)

Figure 16.  Comparison of deblurring U-Nets without initial inverse radial distortion (middle column) and with initial inverse radial distortion as described in Section 3.3 (right column); see Fig. 17 for a zoom of the results of level 19

Figure 17.  Zoom of level 19 from Fig. 16, comparing the deblurring results of U-Nets without (left) and with (right) inverse radial distortion as an initial step

Figure 18.  Comparison of deblurring U-Nets of different depths ( = number of down- and up-sampling steps)

Figure 19.  Average SSIM scores for our ablation studies from Section 4.3; cf. Fig. 14

Figure 20.  Average PSNR scores for our ablation studies from Section 4.3; cf. Fig. 14

Figure 21.  Out-of-distribution performance on examples from the test dataset. The images are taken from the blur levels $ i = 4 , 9 , 14 , 17 $, and 19, while the used reconstruction pipelines have been trained on the easier levels $ i-1 $ and $ i-2 $ (right column = pipeline of level $ i $ for reference)

Figure 22.  Out-of-distribution performance on examples from the test dataset. The images are taken from the blur levels $ i = 4 , 9 , 14 $, and 17, while the used reconstruction pipelines have been trained on the harder levels $ i+1 $ and $ i+2 $ (right column = pipeline of level $ i $ for reference). Note that for level $ i = 19 $, there exists no corresponding pipelines, which explains why we have included level 17 instead

Figure 23.  Average OCR Scores on the test data for our final pipeline but applied to adjacent levels