1. Introduction

The angular resolutions of most commercial imaging systems are limited primarily by pixelation effects rather than by diffraction or optical aberrations. Therefore, super-resolution (SR) techniques have begun to receive attention as methods to obtain images with higher spatial resolutions. Existing image SR technology can be mainly classified into multi-image SR (MISR) techniques [1] and single-image SR (SISR) techniques [2].

When the information frequency of the target scene is higher than the sampling frequency of a single detector, the high-frequency information of the scene will degenerate into aliasing information and thus, high-resolution (HR) imaging cannot be achieved. Combined with the MISR imaging algorithm, the aliasing information contained in the multi-frame image data collected by the camera arrays can be reconstructed as high-frequency information of the target scene [3–12]. The maximum likelihood estimation SR method based on L2 regularization (L2TVSR) [13] can quickly restore the high-frequency information of images. The SR method based on bilateral total variation regularization (L1BTVSR) [14] can preserve the edge information of images. The SR method based on the Lucy–Richardson method (LRSR) [8] extends the application of the LR denoising algorithm to the field of image restoration. Single-frame-image SR methods based on deep-learning have a wide range of applications and do not necessarily require complex specially designed optical imaging systems, only needing images acquired by commercial cameras for SR imaging (such as blind SR generative adversarial network (BSRGAN)) [15–20]. However, single-frame SR algorithms require large amounts of prior information: training image datasets are needed to learn the image degradation models. Therefore, due to the limitations of image datasets, tasks that can be handled by SISR methods may be limited to certain application scenarios (such as face SR). To overcome the limitations of the SISR method, researchers use high-resolution reference (HRef) images for analysis to enhance the generalization performance of the model. Therefore, self-supervised dual zoomed SR (selfDZSR) is proposed as a dual-camera fusion super-resolution technique based on self-supervised learning [21]. This technique combines the wide field of view of short-focus cameras and the high-resolution characteristics of long-focus cameras, optimizing the imaging quality of low-resolution (LR) short-focus images by referencing the high-resolution details in long-focus images.

The primary advantage of camera arrays is their ability to overcome the limitations of pixelation effects and increase the angular resolution of images, enabling clearer and more detailed image capture. Inspired by insects, traditional camera arrays have lenses and detectors evenly distributed across the field of view. They achieve super-resolution images by reconstructing non-redundant information from multiple frames of the same object. However, in many applications, high angular resolution in imaging is not the only requirement; a wide field of view is also necessary to provide context and situational awareness [10,22–24].

In comparison to traditional imaging systems, foveal imaging systems offer a broader field of view. They also preserve high resolution through the design of non-uniform light density distribution. This design enriches images with richer contextual information and situational awareness capabilities. It is akin to the human eye, as illustrated in Fig. 1. In 1990, Tistarellia et al. [25] introduced a foveal imaging charge coupled device (CCD) sensor. It was segmented into 30 concentric rings, with pixel sizes increasing from 30 $\mu$m to 412 $\mu$m radially from the center to the periphery in a polar coordinate fashion. Subsequently, in 2000, Guillem et al. [26] achieved foveal imaging with 8,013 pixels on complementary metal oxide semiconductor (CMOS). Compared to CCD, CMOS is more controllable in data storage and easier to interface with a microprocessor, making it more suitable for simulating the acquisition of foveal information. In 2014, Gao, H. et al. [27] employed a non-uniform lens array arranged in a logarithmic polar configuration to capture light on a photosensitive material. This strategy shifts the complexity to the design of the lens array, enhancing its adaptability to various scenarios. Subsequently, in 2016, G. Carles et al. [10] introduced foveal imaging by combining prisms and camera arrays. This achieved high-resolution central imaging and low-redundancy peripheral imaging. They utilized a camera array super-resolution imaging technique to achieve high-resolution imaging at the fovea. Then, in 2017, a 3D printed 2$\times$2 array of microlenses with different focal lengths was positioned directly on the CMOS sensor to achieve foveal imaging [28]. This approach allows rapid design since the four lenses are printed simultaneously without further assembly.

 

Fig. 1. Foveal vision of the human eye. (a) The spatial resolution characteristic of human vision is from the periphery to the center; and the spatial resolution is gradually enhanced. (b) This visual property is due to the higher density of cones in the foveal region of the human eye structure. This feature helps people observe the surrounding environment and reduces the difficulty of obtaining detailed information about the external environment.

Download Full Size | PDF

However, in the current situation, most commercial imaging systems generally adopt uniform imaging across the entire field of view. This results in the need for a more complex lens array design to implement the foveal imaging method. This article, inspired by [10], proposes an easily assembled method. By changing the focal length of the lens array based on the camera array, super-resolution imaging is achieved in the fovea. Additionally, this paper proposes a MISR algorithm based on multi-focal-length (MFL) images. This study successfully implemented the mechanism of foveal imaging and significantly enhanced the angular resolution of foveal imaging by using the MFL image acquisition method, achieving the expected results. This method breaks through the limitations of a single commercial imaging sensor produced by modern manufacturing technology, demonstrating important application significance.

The contributions of this work are as follows.

1. For images captured by cameras with different focal lengths at different spatial scales, this paper proposes an SR method based on the local gradient constraint, which imitates the foveal imaging process. Simultaneously, while acquiring a large-field-of-view image, this method enables high-resolution imaging of the central area.

2. A multi-aperture, multi-focus imaging model is presented. In order to solve the problem of blurring that may occur when using multiple frames of images with different focal lengths for SR reconstruction, we propose a local gradient constraint method to ensure the details and texture of the reconstructed image.

3. Based on the proposed MFL image SR technology, a multi-focal-length SR imaging (MFLSRI) prototype was established and image data captured in indoor and outdoor scenes were used for experimental verification.

The structure of the remainder of this paper is as follows. Section 2 describes the principles of the bio-inspired MFLSRI system. Section 3 introduces the proposed MFLSRI method based on the local gradient constraint. Section 4 presents the experimental results of the proposed method and compares it with other state-of-the-art (SOTA) SR methods. Lastly, section 5 summarizes our work.

2. Principles of the imaging system

2.1 Camera arrays

Most camera arrays are designed as imaging devices that use the same detector and lens to acquire sub-images of the same target scene, which is the case with the existing MISR imaging methods that use camera arrays. In the context of the entire system, the Nyquist frequency $v_{\text {N}}$ must satisfy the condition $v_{\text {A}}/2 > v_{\text {N}} > v_{\text {D}}$. Here, $v_{\text {A}}$ represents the system’s sampling angular frequency, and $v_{\text {D}}$ represents the optical cut-off frequency of the system. The optical cut-off frequency $v_{\text {D}}$ is calculated as $v_{\text {D}} = \frac {F}{1.22 f \lambda _{\text {O}}}$, where $f$ and $F$ denote the focal length and aperture value, respectively. The wavelength of visible light, denoted as $\lambda _{\text {O}}$, is 550 nm in the context of this article. Furthermore, the Nyquist frequency $v_{\text {N}}$ is defined as $v_{\text {N}} = \frac {f}{2p}$, where $p$ represents the pixel size.

Mathematically, the camera arrays must consist of $M\times M\left ( M>int\left ( {{v}_{\text {N}}}/{{v}_{\text {A}}} \right ) \right )$ cameras, where the ${{v}_{\text {D}}}$ of a single lens and the ${{v}_{\text {N}}}$ of a single focal plane array (FPA) are in a ratio with the former being much higher than the latter. From a simple geometric perspective, it can be understood that the pixel count of the reconstructed HR image is $M^{2} \ \times$ that of a single camera. Based on this, the theoretical spatial resolution is expected to be increased by $M \ \times$. The successful implementation of MISR is not solely dependent on high pixel counts. It also necessitates the existence of sub-pixel shifts between array images and the presence of aliasing artifacts within the array images. The SR technique, theoretically, can be used to achieve the spatial resolution of an SR image to the diffraction limit of the lens.

2.2 Proposed imaging system

To facilitate the creation of an easily assembled foveal SR imaging system, this paper explores the concept of implementing variable spatial resolution imaging as opposed to conventional lens stacking. This approach aims to optimize the trade-off between spatial resolution and field of view within the system. According to the characteristics of human eye imaging, the sampling frequency of the target area in the center of the field of view is increased to reduce the number of required lenses. Simultaneously, through the SR reconstruction algorithm, the image resolution of the central area can be close to the diffraction limit of the lens. The prototype of the imaging system is shown in Fig. 2(b). The prototype contains three different types of lenses and nine high-speed imaging FPAs, forming nine cameras in total. The different colors in Fig. 2(b) represent different lenses: green marks four lenses with focal lengths of $f_{16}$= 16 mm (Basler, C125-1620-5M-P, f/2.0); blue marks four lenses with focal lengths of $f_{25}$= 25 mm (Basler, C125-2522-5M-P, f/2.0); and a lens with a focal length of $f_{35}$= 35 mm (Edmund, HR 85-869, f/1.8) is in the center of the camera arrays. All cameras are equipped with the same FPA sensor (Sony IMX287), the pixel size is p = 6.9 μm, and the sensor resolution is $720 \times 540$.

 

Fig. 2. The proposed SR imaging process for the MFLSRI system, the schematic diagram illustrating the configuration of the MFLSRI system, and the curves depicting the imaging angular resolution. (a)Schematic diagram of the super-resolution reconstruction principle of the proposed MFLSRI system. Image pixels captured by the array camera are mapped onto a high-resolution grid of the reference image, and distance-weighted based interpolation is performed on the grid. The pixel values are then aggregated within the grid. (b) MFLSRI system. Among them, the green, blue and yellow lenses are 16-, 25-, and 35-mm-focal-length lenses, respectively. (c) Angular resolution comparison of different types of lenses and the proposed bio-inspired MFLSRI system. Green, blue, and yellow is the sampling frequencies of the 16-, 25-, and 35-mm-focal-length cameras, respectively; red indicates the sampling frequency of the proposed MFLSRI system.

Download Full Size | PDF

The field of view of a single camera in the prototype imaging system and the reconstructed foveal imaging field of view are shown in Fig. 2(a). As a lens with a larger focal length results in a smaller field of view, the closer to the central area, the greater the amount of image overlap, resulting in higher spatial resolution through computational reconstruction. In Fig. 2(a), the peripheral field of view (represented by the green points) is equivalent to the field of view of the image captured by the 16-mm-focal-length lens; the size of the transitional field of view (represented by the blue points) is equal to the size of the field of view of the 25-mm-focal-length lens; and the central field of view (represented by the yellow points) is equal in size to the field of view of the 35-mm-focal-length lens. We established a high-resolution grid based on the coordinate system of the reference image and projected all sample points onto this grid using a registration method. In this paper, the registration method employed is the scale-invariant feature transform (SIFT [29]). In Fig. 2(a), each red coordinate point is derived by applying Euclidean distance interpolation to the adjacent sample points, resulting in the reconstructed high-resolution image.

To deeply explore the performance of this system, a mathematical model was established in this paper. According to the definition of angular resolution, the field of view angle and the sample count $N$ within its range are used in this study to compute the system’s Nyquist angle resolution, given by the formula:

Specifically, ${{\theta }_{\text {h}}}$ and ${{\theta }_{\text {v}}}$ denote the half-angles for the horizontal field of view (HFOV) and the vertical field of view (VFOV). The imaging principle of a single camera is shown in Fig. 3. Considering an imaging system composed of $K$ cameras, each with varying focal lengths, the formula to determine $N$ within its field of view is:

 

Fig. 3. The imaging principle of a single camera. From the image, it can be inferred that when the HFOV, focal length, and pixel size are known, the number of sampling points in the horizontal direction can be determined.

Download Full Size | PDF

For $K=1$, Eq. (3) can determine the Nyquist angle resolution of a camera with focal length $f$. Therefore, in comparison to a camera with focal length $f$, the spatial resolution enhancement factor $m$ for the MFLSRI system is given by:

Equation (3) allows for theoretical computation of the sampling rate for a single aperture and the imaging system in this study, as depicted in Fig. 2(c). Moreover, to achieve MFLSRI with a MFLSRI system, it is necessary to meet the three conditions required for the camera arrays to implement MISR. Firstly, Nyquist angular resolution is greater than the optical cut-off angular resolution of the lens; The second and third conditions require the array images to have sub-pixel displacement and aliasing artifacts. Considering the intrinsic design of the MFLSRI system, the second requirement is inherently met, and the third underscores the justification for utilizing MFLSR.

Hence, in the MFLSRI system, it is imperative that the maximum optical cut-off angular resolution of the lens assembly must be smaller than the minimum Nyquist angular resolution of the camera assembly. As an example, within the lens array, the 16 mm focal length lens possesses the maximum optical cut-off angular resolution, which is calculated as $1/{{v}_{\text {D}}}$=0.084 mrad. Among all the cameras, the one with a 35 mm focal length has the smallest Nyquist angle resolution of 0.40 mrad. Thus, this designed MFLSRI system is capable of achieving SR at the center. Utilizing Eq. (4), the SR factor for relevant regions can be determined. As an illustration, relative to a camera with a 16 mm focal length, in the peripheral zone (solely where the 16 mm lens overlaps), transitional region (where the 16 mm and 25 mm lenses overlap), and the central section (where all lenses overlap), the MFLSRI system has SR multiples of 2, 3.7, and 4.3, respectively. Comparatively, for a camera with a 35 mm focal length, the MFLSRI system displays a SR factor of 2 in the central zone. A more comprehensive calculation outcome is presented in Table 1.

Table 1. Specific imaging parameters of proposed device

View Table | View all tables in this article

3. Method

3.1 Image SR model based on multiple apertures and focal lengths

We assume that there exists a spatially variable-resolution image $\boldsymbol {Y}$. Different shifting, blurring, downsampling, and cropping operations on $\boldsymbol {Y}$ yield nine different low-resolution images ${{\boldsymbol {X}}_{i}}$:

Let $\boldsymbol {H}_i=\boldsymbol {D}_i\boldsymbol {B}_i\boldsymbol {M}_i\boldsymbol {T}_i$, Eq. (5) can be written as

In SR methods, reconstructing SR images is an ill-conditioned problem. When relying exclusively on the data fidelity term, SR methods often fail to achieve high-quality image reconstruction. For these ill-conditioned problems, a set of low-resolution images may reconstruct multiple different SR images. To solve this ill-conditioned problem, an effective prior regular term should be added to constrain the solution of the optimization problem according to the actual situation. Local gradient constraints have certain advantages in multi-focal image SR methods, which is the focus of this paper [30]. In addition, as a certain amount of noise will be introduced in the SR reconstruction process, the method in this paper introduces a TV regular term based on the 1-norm for image denoising. Therefore, the formula of the multi-focal-length SR model proposed in this paper is

The first term is the data fidelity term, the second term is the local gradient constraint term, and the third term is the 1-norm-based TV denoising term. $\boldsymbol {\hat {Y}}$ is the solution that approximates $\boldsymbol {Y}$, $\lambda$ and $\gamma$ are the regularization parameters used to penalize these two terms, and $\nabla$ represents the gradient operator. After $t$ iterations, the SR image result ${{\boldsymbol {\hat {Y}}}^{t}}$ is obtained. It should be noted that $\boldsymbol {\hat {Y}}^0$ is a HR image derived from the sampling point interpolation technique outlined in Section 2.

According to the model of the MFLSRI system proposed in this paper, in practical applications, the acquired images are low-resolution images with different sharpnesses and focal lengths. Analogous to challenges in image fusion, hazy images of low clarity diminish the enhancements introduced by crisper images, mirroring the diffusion effects observed in chemistry, thereby undermining the quality of images reconstructed using super-resolution techniques. Although images with short focal lengths are much blurrier than those with long focal lengths, according to our theoretical calculations in Section 2, all images should provide spatial resolution gains to the reconstructed image. Drawing on panchromatic sharpening and image fusion technology [31–34], it can be considered that there is a linear relationship between the gradients of images of different focal lengths, as shown in Fig. 4. Figure 4(a) shows images taken of the same scene at three focal lengths using an imaging device. Zooming in on the local details of the three images shows that the larger the focal length, the clearer the image and the stronger the edge contrast. Figure 4(b) plots the gradient changes of the three images on line 322. The gradients of the three images with different focal lengths have an obvious linear relationship locally. Therefore, we infer that in the iterative process, there is a local linear relationship between the gradient $\nabla {{\boldsymbol {Y}}^{t}}$ of the $t$-th iteration result and the gradient $\nabla {{\boldsymbol {Y}}^{t+1}}$ of the $t+1$-th iteration result. Simultaneously, as reported in the literature [30,35], the use of gradient constraints in image reconstruction is beneficial for preserving the edge structure of the image. To enhance edge preservation in the SR image, local gradients are employed to impose constraints on the gradient of the SR image. The local gradient constraint term is as follows:

 

Fig. 4. Linear relationship of the image on the gradient between different focal lengths. (a) Local details of the same scene (inside the dotted frame) were captured with lenses of different focal lengths; the focal lengths of the three images are 35 mm, 25 mm, and 16 mm, from left to right. (b) Gradient scanning of three images with different focal lengths at line 322, where red, blue, and green correspond to the gradients of the red, blue, and green horizontal lines in the three images in (a) respectively.

Download Full Size | PDF

$\nabla {{\boldsymbol {\hat {Y}}}^{t-1}}$ represents the gradient map in the SR image during the $t-1$-th iteration. Referring to the maximum value mapping theory in the image fusion method [36], when $t$ = 1, $\nabla {{\boldsymbol {Y}}_{LGC}}=\max \left ( \nabla \left ( \boldsymbol {H}_{i}^{T}{{\boldsymbol {X}}_{i}} \right ) \right )$($i\in 1,2,\ldots,9$). In this paper, $\max \left ( \nabla \left ( \boldsymbol {H}_{i}^{T}{{\boldsymbol {X}}_{i}} \right ) \right )$ refers to projecting each low-resolution image onto a high-resolution grid and selecting the maximum gradient value at each position to generate $\nabla \boldsymbol {Y}_{\text {LGC}}^{1}$. In Eq. (10), $\boldsymbol {A}$ and $\boldsymbol {C}$ are coefficient matrices generating the local linear relationship between $\nabla \boldsymbol {\hat {Y}}$ and $\nabla {{\boldsymbol {Y}}_{LGC}}$. For the $k$-th pixel in the image, the local linear relationship between $\boldsymbol {A}$ and $\boldsymbol {C}$ is

The images may be noisy at the time of acquisition due to hardware, the environment, and other factors. Also, noise is introduced during the iterative process under local gradient constraints. Here, we introduce the L1 total variation (L1TV) term [37] to perform the denoising and smoothing operation on the image; then, the L1TV denoising term is as in Eq. (16):

3.2 Solving the image SR model algorithm

The multi-focal-length image SR model proposed in this paper is obtained by combining Eq. (7), Eq. (9), and Eq. (16) as follows

As Eq. (8) is a typical least absolute shrinkage and selection operator (LASSO) problem [38], we use alternating direction method of multipliers (ADMM) to solve this equation. Introducing the auxiliary variable $\boldsymbol {u}$, we have

Solving for $\boldsymbol {\hat {Y}}$ in Eq. (19) gives

We then use the gradient descent method to solve Eq. (20) obtaining $\boldsymbol {Y}^{t+1}$,

For $\boldsymbol {u}$, we use the shrink method to solve

Solving for $\boldsymbol {z}$ we obtain

Finally, solving for $\alpha$ we obtain

The whole image SR reconstruction algorithm is shown in the Table 2.

Table 2. Framework of the proposed local-gradient-constraint-based SR algorithm

View Table | View all tables in this article

4. Experiment results and discussion

To prove the effectiveness of our proposed system and algorithm, we used the ISO 12233 resolution test chart image captured by the prototype to test its spatial modulation function (modulation transfer function, MTF) value and increase the spatial resolution [39,40]. A higher MTF value indicates greater resolution capability in the optical system or imaging sensor. Field experiments were also conducted to verify the feasibility and effectiveness of the proposed system and the SR method. To verify the performance of the proposed SR imaging method, the SR-RAW dataset [41] was used to conduct simulation experiments, and the peak signal-to-noise ratio (PSNR) and image structural similarity (SSIM) were used to evaluate the simulation results. For the image SR calculation and other image post-processing, the processor model of the computer was an AMD Ryzen 7 3700 and the memory was 32 GB. The SR methods used for comparison were LRSR [8], L1BTVSR [14], L2TVSR [13], BSRGAN [19], and selfDZSR [22]. For the MISR method, the number of convergence iterations was set to 50, and the convergence error was $\varepsilon =\max \left ( mean\left ( {{\boldsymbol {Y}}^{t+1}}-{{\boldsymbol {Y}}^{t}} \right ) \right )=2\times {{10}^{-5}}$. The iterations were stopped when the number of convergence iterations or convergence error was reached, and the SR image was obtained. Importantly, the compared MISR method adopts the same registration approach and imaging model as this study and chooses its parameters based on PSNR iterative tests on the simulation dataset. Through PSNR testing for convergence within iterations, L1BTVSR and L2TVSR were allocated regularization weights of 0.06 and 0.19, respectively. BSRGAN operates as a SISR approach(with the input being the image with the maximum focal length), and was evaluated against SOTA methods in the Real SR domain. In selfDZSR, we used both the maximum and minimum focal length images as reference and target images, respectively, for comparison as a MFLSRI algorithm based on deep-learning methods. Each deep-learning method utilized the testing models furnished by the researchers.

4.1 Experimental results

To evaluate the improvement effect of the proposed SR imaging method on the image spatial resolution, indoor and outdoor experiments were performed using the prototype system. Quantitative and qualitative evaluations were conducted on the SR results. The resolution of the reconstructed SR image was tested using the ISO 12233 resolution test chart. The prototype system was used to capture 287 sets of multi-focal images, each consisting of nine images, for a total of 2583 low-resolution images.

Figure 5 shows the images acquired using the system prototype proposed in this paper and the reconstructed fovea SR image, as well as the local details of each image. Figure 5(b), Fig. 5(c), and Fig. 5(d) represent zoomed-in images of the LR images based on their respective positions in Fig. 5(a). Figure 5(e), Fig. 5(f), and Fig. 5(g) illustrate detailed views of the MFLSRI images at the positions corresponding to the solid boxed areas. The spatial resolution of the MFLSRI image is significantly improved compared with those of the other images. In particular, the spatial resolution of the highest resolution image in the low-resolution image group is approximately 4 line pairs per millimeter (LP/mm) (that is, the spatial resolution of the 35-mm-focal-length image (Fig. 5(d)), whereas the spatial resolution of the SR image can reach 6 LP/mm. That is, the SR effect proposed by the device can reach 1.5 $\times$ the highest spatial resolution in the low-resolution images. The spatial resolution of the 16-mm-focal-length image in the LR image group is approximately 1.5 LP/mm, and the spatial resolution of the SR image is 4 $\times$ that of the 16-mm-focal-length image, which is close to the diffraction limit of the 16-mm-focal-length lens. Due to factors such as registration accuracy, optical distortion, and data redundancy, the actual super-resolution multiple did not reach the target value of 4.3 $\times$ as defined in Section 2. Taking traditional array imaging as an example, at least 16 identical cameras are required to form an imaging system to obtain 16 images of the same target to achieve a spatial resolution of more than 4 $\times$. The device proposed in this paper only needs nine cameras with different focal lengths to achieve this spatial resolution while retaining the field of view of the 16 mm lens. In other words, the advantages of our proposed device are low cost, small size, and fast post-processing.

 

Fig. 5. The central fovea SR image obtained by the imaging device described in this paper was compared with images of other focal lengths in terms of spatial resolution. (a)Depiction of the central fovea SR image. The region bounded by the red dashed box represents the central view. Between this and the green dashed box lies the transitional view, with the area outside being the peripheral view. (b), (c), and (d) individually present detailed views of LR images within the blue, green, and red solid boundaries, corresponding to focal lengths of 16 mm, 25 mm, and 35 mm, respectively. (e), (f), and (g) are SR images mirroring the exact locations depicted in (b), (c), and (d).

Download Full Size | PDF

The results of using the prototype system to capture images of vehicles on the road and perform SR reconstruction on the captured images are shown in Fig. 6. Further analysis of the SR results for car body and tire is provided. Figure 6(b) is the original image captured by the 35-mm-focal-length lens, and Fig. 6(c) is the enlarged image of the bicubic interpolation in Fig. 6(b). In both Fig. 6(d) and Fig. 6(e), tire details are noticeably restored in the SR images derived from LRSR and L1BTVSR methods, yet both images suffer from substantial noise interference. The results from the L2TVSR method exhibit minor noise in the white vehicle body, whereas BSRGAN fails to authentically reproduce the tire details. Even though selfDZSR manages to augment target resolution using a reference image, it doesn’t manage to recuperate the intricate details of the wheel hub. Compared with the results of the other SR methods, the SR image obtained by this method not only restores the detailed information and high-frequency information of the image, but also achieves the best visual effect.

 

Fig. 6. The SR results of images captured by the MFLSRI system. (a)Images acquired during the field tests. (b) Image acquired by the 35-mm-focal-length lens; (c) Bicubic interpolation image of (b); and SR images obtained using (d) LRSR, (e) L1BTVSR, (f) L2TVSR, (g) BSRGAN, (h) selfDZSR, and (i) the proposed method. The magnification is 4$\times$.

Download Full Size | PDF

For the images captured by the prototype system, the foveal SR ground-truth image cannot be obtained directly by any existing imaging systems. Hence, no-reference image quality assessment methods were employed to evaluate the effectiveness of the SR methods. The blind/reference-less image spatial quality evaluator (BRISQUE) [42], natural image quality evaluator (NIQE) [43], SMD2 [44], and contrast enhancement image quality (CEIQ) [45] were used. Table 3 shows the average no-reference image evaluation metrics computed using the SR images corresponding to the 287 sets of actually captured images. In Table 3, the optimal results are marked in bold and sub-optimal results in italics. On the NIQE and SMD2, the method presented in this research secured near-top performances, while it predominantly excelled in the MISR method. Even though BSRGAN and selfDZSR’s data-driven attributes notably enhance the image quality, it doesn’t mean they can authentically reproduce the image’s details, as depicted in Fig. 6.

Table 3. Average no-reference image metrics calculated using the SR results of all captured multi-focal-length image groups

View Table | View all tables in this article

Figure 7 shows the low-resolution and SR images obtained by taking the test chart. Figure 7(a) shows the ground truth (GT) image. Among them, Fig. 7(b) is an image with a focal length of 35 mm in the original image, and Fig. 7(c) is an enlarged cubic interpolation of the image in Fig. 7(b). The spatial resolution of the highest resolution image in the low-resolution image group is approximately 4 LP/mm, and the spatial resolutions of the images corresponding to LRSR, L1BTVSR, L2TVSR, and the proposed method can reach 6 LP/mm. Secondly, the LRSR and L1BTVSR images exhibit a certain amount of noise on the number "6". The BSRGAN and selfDZSR images do not reach the spatial resolution of 6 LP/mm, and the BSRGAN image even has serious artifacts. The L2TVSR image has a good balance between noise and high-frequency details, but the method in this paper uses gradient constraints to make the details more prominent. Therefore, the proposed method achieves a balance between improving resolution and suppressing noise and artifacts, outperforming the other competing methods.

 

Fig. 7. SR results of ISO 12233 resolution test chart images. The magnification is 4$\times$. (a) GT image; (b) Image acquired by 35-mm-focal-length lens; (c) bicubic interpolation image of (b); and SR images obtained using (d) LRSR, (e) L1BTVSR, (f) L2TVSR, (g) BSRGAN, (h) selfDZSR, and (i) the proposed method.

Download Full Size | PDF

To evaluate each SR method quantitatively, the slanted-edge method was used to calculate the MTF of the test chart image to obtain the MTF20 value (the value at which the MTF decays to 20%) of each SR method, as shown in Table 4. Table 4 demonstrates that the image resolution using the SR algorithm proposed in this paper is significantly higher than those of the other multi-frame image SR algorithms but inferior to that of BSRGAN. Concurrently, the MTF of the proposed method is 1.4 $\times$ greater than the LR image. Interestingly, although the spatial resolution of the BSRGAN image is not improved and severe artifacts appear, its resolution is much larger than that of the multi-frame SR image. Considering the principle of the edge method for calculating MTF, this finding shows that the BSRGAN’s processing of edges is extremely good. However, the essence of SR is to improve the spatial resolution of the image to obtain more high-frequency information. Therefore, the processing of BSRGAN images is distorted, which is also one of the characteristics of deep-learning SR methods.

Table 4. Image MTF measure calculated using the SR results showed in Fig. 7

View Table | View all tables in this article

4.2 Simulation

To simulate the characteristics of the multi-focal-length imaging system as much as possible in the simulation experiments, the multi-focal-length images captured for the same scene in the SR-RAW dataset were used as the ground truth images. The multi-focal-length images in the image data set were down-sampled, blurred, and displaced to simulate the imaging model of the multi-focal imaging system and were used to generate low-resolution images captured by the simulated multi-focal-length imaging equipment. The resolution of the ground truth image was 4 $\times$ that of the low-resolution image generated by the simulation. In the above degradation operation, the blur function was a Gaussian function with a mean of 0 and standard deviation of 1. To simulate the sub-pixel displacement between multi-focal-length images collected with different apertures, except for the image with the largest focal length in the image group, the displacement deviations of the other images have absolute values less than 1 at random in the horizontal and vertical directions.

Figure 8 and 9 show the results of the simulation with image groups 105 and 151 in the SR-RAW dataset. These image groups contain one 49-mm-focal-length image, four 35-mm-focal-length images, and four 24-mm-focal-length images. Because the resolution of the ground-truth image is 4 $\times$ the size of the low-resolution image, in the simulation process, the magnification of the SR reconstruction was set to four, so that the sizes of the ground-truth and SR images were consistent, and the reference image evaluation test could be performed. The SR reconstruction results in Fig. 8(l) and Fig. 9(d) show that the LRSR image obviously has more noise, and the edges of the L1BTVSR and L2TVSR images’ are blurred. The SR images obtained by the proposed method outperformed the SR images obtained by the L1BTVSR and L2TVSR methods in terms of object’s edge and detailed texture information. The images obtained by the BSRGAN method optimize the SR results of the object edge details, which is also one of the characteristics of the deep-learning SR methods. However, it is not possible to judge the advantages and disadvantages of a certain SR method only from the high-frequency detail texture information recovered from the SR image. The recovered text in Fig. 8(g) and Fig. 9(g) have more artifacts. While selfDZSR images exhibit minimal noise artifacts, it lags behind other MISR algorithms with respect to detail restoration. Therefore, the method proposed in this paper is superior to other methods in terms of image detail recovery, edge preservation, and denoising.

 

Fig. 8. SR images of image No. 105 in the SR-RAW dataset [41] at a magnification of 4$\times$. (a) Ground truth; (b) 49-mm-focal-length image; (c) bicubic interpolation of (b); and SR images reconstructed using (d) LRSR, (e) L1BTVSR, (f) L2TVSR, (g) BSRGAN, (h) selfDZSR, and (i) the proposed method. Close-ups from (a-i) are represented by (j-r) respectively.

Download Full Size | PDF

 

Fig. 9. SR images of image No. 151 in SR-RAW dataset [41] at a magnification of 4$\times$. (a) Ground truth; (b)49-mm-focal-length image; (c) bicubic interpolation of (b); and SR images reconstructed using (d) LRSR, (e) L1BTVSR, (f) L2TVSR, (g) BSRGAN, (h) selfDZSR, and (i) the proposed method. Close-ups from (a-i) are represented by (j-r) respectively.

Download Full Size | PDF

Tables 5 and 6 show the PSNR and SSIM of the SR processing results of 12 sets of data in the SR-RAW dataset. The optimal values appear in bold and suboptimal values in italics.

Table 5. PSNRs of the SR processing results of 12 sets of data in the SR-RAW dataset [41]

View Table | View all tables in this article

Table 6. SSIMs of the SR processing results of 12 sets of data in the SR-RAW dataset [41]

View Table | View all tables in this article

In most cases, the PSNR and SSIM of the SR results produced by the proposed method are better than those obtained using the other methods, and the average value is the best. The above experimental results demonstrate that the method in this paper can provide better SR reconstruction results, especially in terms of the PSNR and SSIM, compared to the other SR methods.

The running time of each algorithm and the number of iterations to reach convergence are shown in Table 7, and the minimum convergence time is marked in bold. Although the method proposed in this paper is not the fastest approach, it can provide the best PSNR and SSIM, balancing the reconstruction effect and computing power consumption.

Table 7. Running time of the algorithms (where the convergence error is set as shown and the maximum number of iterations is set to 50 and the input is 9 540$\times$720 images)

View Table | View all tables in this article

4.3 Ablation experiment

The algorithm in this study integrates local gradient constraint (LGC) and L1TV denoising terms into the conventional imaging model to enhance the quality of image reconstruction. This section performs ablation experiments to assess the effect of each prior term on image reconstruction, considering three combinations of prior terms:

Nine sets of images are randomly selected from the SR-RAW dataset for the ablation experiments in this study. Each experiment iterates 20 times, with other parameter settings kept consistent with previous experiments. The reference ground truth image is presented in Fig. 10(a). Figure 10(b) depicts a low-resolution image at a 49 mm focal length. Reconstruction results using the aforementioned combinations of prior terms are exhibited in Fig. 10(c) to (f). Sole reliance on data constraint terms for resolving the MISR issue is not practical. The L1TV term in reconstruction results in images with lower noise but potential detail smoothing. Employing local gradient constraint terms markedly improves image details. The constraint emphasizes the comparison between image gradients and expected gradients in the reconstruction process. While enhancing details, clarity, and SR effects, this approach may also lead to increased noise. The study combines local gradient constraints and L1TV priors to strike a balance between gradient enhancement and image smoothing. The SR outcomes, derived from LGC and L1TV priors, bolster image details while suppressing noise. Table 8 illustrates that the proposed approach effectively eliminates noise and enhances image details through the integration of LGC and L1TV terms. Valuable insights for further advancements in imaging methods are provided through these comparative experiments in this section. Moreover, it furnishes references for the selection of imaging strategies in specific applications.

 

Fig. 10. Ablation experiment results of the image NO.148 in the SR-RAW dataset [41].

Download Full Size | PDF

Table 8. Average of PSNR and SSIM for 9 validation image datasets

View Table | View all tables in this article

5. Conclusion

Inspired by the large field of view and the central high-resolution imaging mechanism of the human visual system, a foveal SR imaging optical system was proposed and a prototype imaging system was built for demonstration. The prototype imaging system was verified through simulations, indoor experiments, and field tests. The multi-focal-length images collected by the system were post-processed by the SR algorithm proposed in this paper, and then a large-field-of-view and high-resolution foveal image was reconstructed. For the multi-focal-length images collected by the prototype system, an SR imaging method based on the local gradient constraint was proposed to achieve foveal SR imaging. The proposed system and algorithm were simulated using the SR-RAW public dataset, and the feasibility of the SR algorithm was verified. The simulation results showed that the average PSNR of the results obtained by the proposed method was approximately 0.32 dB higher than the sub-optimal value, confirming the effectiveness of the method. The prototype system was verified experimentally using the ISO 12233 resolution test chart, and field tests were conducted. Using the local-gradient-constraint-based super-resolution algorithm to perform SR reconstruction on the collected images, in the foveal area of the SR result image, the spatial resolution was improved by nearly 4 $\times$ compared to that of the image with the minimum focal length, which was close to the limit of the corresponding lens. The proposed algorithm achieved a spatial resolution of approximately 1.5 $\times$ relative to that of the image at maximum focal length. The field test results confirmed that the proposed bio-inspired imaging system and SR method provided better results than the other SR methods in terms of MTF, spatial resolution, and other metrics, while yielding sub-optimal NIQE values, thus confirming the effectiveness of the proposed method in practical applications. In conclusion, this paper provides an effective solution for multi-aperture, multi-focal-length SR imaging.

For future applications of the proposed method, certain limitations persist, particularly regarding the computational cost required for generating SR images. Enhancing calculation efficiency is paramount. The current approach of using a uniform high-resolution grid to interpolate pixels of array image projections proves to be inefficient, given the lower sampling density in the peripheral field of view compared to the central field of view. Consequently, interpolation methods for the peripheral field of view are inefficient.

To address this challenge, future work will focus on refining the algorithm. Specifically, the aim is to extend the prior knowledge of HR images from the central to the peripheral field of view, thereby reducing computational redundancy. This strategic adjustment is anticipated to enhance SR imaging performance while alleviating the computational burden.