Open Access
Add to BibSonomy
Abstract
Dual-mode visible/near-infrared imaging systems, including a bioinspired six-channel design and more conventional four-channel implementations, have transitioned from a niche in surveillance to general use in machine vision. However, the demosaicing routines that transform the raw images from these sensors into processed images that can be consumed by humans or computers rely on assumptions that may not be appropriate when the two portions of the spectrum contribute different information about a scene. A solution can be found in a family of demosaicing routines that utilize interpolating polynomials and splines of different dimensionalities and orders to process images with minimal assumptions.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
As demand has grown for imaging systems that can capture more than just pleasing photographs, engineers have turned increasingly to dual-mode visible/near-infrared imaging systems that can distinguish between light in the visible spectrum and light in the near-infrared spectrum. Since visible light stimulates a meaningful response in the human eye, but near-infrared light does not, the visible image will be most relevant when a scene is (or can be) illuminated for a human operator and contains information relevant to that operator, permitting the image to capture what the operator would normally see. In contrast, the near-infrared image will be most relevant when a scene is not (or cannot be) properly illuminated for an operator or contains information irrelevant to that operator, permitting the image to capture what the operator would not normally see.
This is especially the case today as visible/near-infrared imaging systems are increasingly finding a spot in the machine vision space. For example, when the user is interested in the physical layout of a scene, a near-infrared projector can provide structured illumination, permitting a near-infrared image to provide a three-dimensional reconstruction over which a visible image can be painted; this may be useful for, e.g., facial biometric authentication [2]. Alternatively, when the user is interested in the biological or chemical makeup of a scene, a near-infrared pigment or dye can serve as a contrast agent, permitting the near-infrared image to provide a compositional analysis under which the visible image can be layered; this may be useful for, e.g., ocular biometric authentication [3]. The patterns induced by the structured illumination or contrast agent may not be meaningful without additional processing, but they will remain invisible to the user and thus won’t interfere with the user’s day-to-day life.
One solution for these applications, inspired by the mantis shrimp visual system and realized by Blair et al. [1], is diagrammed in Fig. 1(A). Incident light first encounters an array of optical filters, arranged in a two-by-one pattern, where each filter transmits either visible light or near-infrared light. Transmitted light then encounters an array of photodiodes, arranged in three layers, where the first photodiodes to be met preferentially absorb shorter wavelengths and the last photodiodes preferentially absorb longer wavelengths. This image sensor can thus be partitioned into two subsets of pixels: One subset of short-pass pixels detects the three primary colors of visible light, while the remaining subset of long-pass pixels detects three more “colors” of near-infrared light (Fig. 1(B)). These bioinspired long-pass/short-pass image sensors can thus detect six channels spanning the visible and near-infrared for every two pixels in the arrays.
Fig. 1. The architecture of the bioinspired imaging system and RGB-IR imaging system. (A) A top-down view (left) and a cross-sectional view (right) of a bioinspired long-pass/short-pass image sensor (or bioinspired image sensor). Three photodiodes are formed per pixel via layering of positively doped (p+) and negatively doped (n+) silicon, and optical filters are bonded in a two-by-one pixel pattern on top. Each photodiode is sensitive to longer wavelengths, shorter wavelengths, or intermediate wavelengths, and each filter transmits long wavelengths or short wavelengths, providing six different channels. (B) The quantum efficiencies for the bioinspired image sensor used in this study. The image sensor is sensitive across three channels in the visible (VIS) spectrum and three channels in the near-infrared (NIR) spectrum. (C) The same views as (A) for a more traditional red/green/blue/near-infrared image sensor (or RGB-IR image sensor). One photodiode is formed per pixel via layering of p+ and n+ silicon, and optical filters are deposited in a two-by-two-pixel pattern on top. Each photodiode is sensitive across a broad band, and each filter transmits a narrow band, providing four different channels. (D) The same plot as (B) for the RGB-IR image sensor used in the study. The image sensor is sensitive across three channels in the VIS spectrum and one channel in the NIR spectrum. PD, photodiode; SP, short-pass; LP, long-pass. (Note: The quantum efficiencies for both the bioinspired image sensor and the RGB-IR image sensor were adapted from the bioinspired image sensor in [1] to help control variables across the two sensors.)
Another solution explored by the research community [4–14] is diagrammed in Fig. 1(C). As with the bioinspired sensor, incident light first encounters an array of optical filters, but it may pass through one of four filters, not two, that transmit either blue light, green light, red light, or near-infrared light. Likewise, transmitted light then encounters an array of photodiodes, but it may interact with a single layer of photodiodes, not three, that absorbs light at both short wavelengths and long wavelengths. The image sensor can thus be logically partitioned into four subsets of pixels: The subsets of blue pixels, green pixels, and red pixels detect the three primary colors of visible light, while the remaining subset of near-infrared pixels detect an additional “color” of near-infrared light (Fig. 1(D)). These more traditional red/green/blue/near-infrared image sensors, also known as RGB-IR or RGB-NIR image sensors, can thus detect four channels spanning the visible and near-infrared for every four pixels in the arrays.
While the bioinspired image sensor captures six channels and the RGB-IR image sensor captures four channels, it should be noted that neither image sensor captures every channel at every pixel. Instead, the bioinspired image sensor captures each of its channels across one-half of its pixels, and the RGB-IR image sensor does the same across one-quarter of its pixels. To ensure that each channel can be queried at each pixel, a demosaicing routine is needed that encodes a presumed relationship between the pixels where a channel is observed and the pixels where the channel is not observed. Only then can the demosaicing routine take the actual observations that were made at the former pixels and yield the anticipated observations that would be made at the latter pixels.
While there has been no work to date on demosaicing routines for the bioinspired image sensor, there have been several works describing demosaicing routines for the RGB-IR image sensor [7,15–17]. However, the demosaicing routines that have been proposed make assumptions on the correlation between the visible channels and the near-infrared channels in an attempt to exploit the relationship between the channels that were observed at a pixel and the channels that were not observed at that pixel. In commoditized RGB-IR image sensors, which utilize pigmented filters with a poor stopband attenuation, this assumption may be appropriate since the signal for a channel will include a meaningful contribution from both in-band light and out-of-band light, but in high-performance RGB-IR image sensors, which utilize interference filters with a considerable stopband attenuation, it may not be appropriate since the signal for a channel will include a meaningful contribution from in-band light alone (Supplement 1). The situation is further complicated in tasks with a security component, like facial and ocular biometric authentication, where the correlations enforced by a demosaicing routine may cause information to leak between channels—producing an unknown bias in the statistical features extracted from an image that may affect the error rate of fault-sensitive tasks lying beyond the image processing pipeline while also causing cryptographic secrets hidden in one channel to display in other channels. Although there are as of yet no known exploits that take advantage of this vulnerability, there is forensics research that indicates that the source of an image can be determined from the interchannel correlations enforced by a demosaicing routine [18].
It is therefore relevant to consider how demosaicing routines can be developed alongside image sensors, not just to produce the perceptually agreeable images provided by conventional techniques, but also to capture the physically accurate image needed in demanding applications. This paper thus establishes a set of demosaicing routines that are compatible with the bioinspired image sensor and the RGB-IR image sensor and are free from any assumptions on interchannel correlations (Section 2). It also describes how these demosaicing routines and image sensors can be evaluated on natural images, proposing a set of metrics that reflect the actual and perceived drop in image quality during image processing and capture (Section 3). An analysis of a single-band visible image and a set of dual-band visible/near-infrared images is then provided as a proof-of-concept for these techniques for both the bioinspired image sensor and the RGB-IR image sensor and their associated demosaicing routines (Sections 4 and 5).
2. Implementation of demosaicing routines
The objective of a demosaicing routine is to restore the unmosaiced image that would have been registered if every channel could have been observed at every pixel. It thus maps a mosaiced image where each channel is observed across a limited subset of pixels to a demosaiced image where each channel is estimated across the full set of pixels (Fig. 2). In general, this problem is ill-posed since an infinite number of images can be constructed where a small number of pixels take known intensities; however, it can be made well-posed if a singular image can be made acceptable via constraints on the unknown intensities. Since a dual-band imaging system cannot necessarily expect a correlation between the visible light and the near-infrared light, these constraints should not be made on the correlations between pixels in one channel and pixels in another; however, they can be made on the correlations between pixels within a single channel.
Fig. 2. The effect of mosaicing and demosaicing on the bioinspired imaging system and RGB-IR imaging system. The dual-mode visible/near-infrared imaging systems that are of interest collect three visible (VIS) channels and up to three near-infrared (NIR) channels. Ideally, the imaging systems would collect unmosaiced images where each channel is observed at all of the pixels, but realistically, the imaging systems will collect mosaiced images where each channel is observed at some fraction of the pixels. As a result, a demosaicing routine must produce a demosaiced image where every channel is either observed or estimated at all of the pixels. The demosaicing routine can be evaluated by computing the error between the final demosaiced image and the initial unmosaiced image. (A) Due to the three layers of photodiodes and two-by-one filter pattern, the bioinspired image sensor can collect all six channels, with one-half of the pixels observing each channel. The demosaicing routine can recover the unknown pixels by making assumptions from the dense known pixels. (B) Due to the single layer of photodiodes and two-by-two filter pattern, the RGB-IR image sensor can collect only four channels, with one-quarter of the pixels observing each channel. The demosaicing routine cannot recover the missing two channels but can recover the unknown pixels by making assumptions from the sparse known pixels.
From this perspective, each channel can be taken as a function of two variables, the row number and the column number, that should be estimated at a subset of unknown pixels given the observations at a subset of known pixels. Therefore, each channel can be populated by interpolating the values at the unknown pixels from the values at the known pixels. Since interpolating polynomials can approximate a wide range of functions, they can be adopted, in four variations, to impute the values of the unknown pixels from the values of the known pixels. These variations include one-dimensional polynomials, one-dimensional splines, two-dimensional polynomials, and two-dimensional splines—all of which can be linear or cubic in the row number and column number (Supplement 1).
Since each technique reconstructs a function using polynomials, piecewise or otherwise, they have similar (but not identical) representative power; however, the four techniques will not return identical functions since they implement different objectives. For example, the polynomial-based methods fit functions to portions of the rows, columns, or images using only the values observed within those regions, while the spline-based methods fit functions to the entirety of the rows, columns, or images utilizing both the values observed and derivatives assumed across those regions. As a result, the polynomial-based methods, with their small neighborhoods and unconstrained derivatives, may be more responsive to local fluctuations in intensity, while the spline-based methods, with their large neighborhoods and smoothed derivatives, may be more responsive to global trends in intensity. It is worth noting, though, that a technique that appears more flexible is not necessarily a technique that is more effective. The one-dimensional methods fit functions along individual rows and columns and may miss features that span multiple rows and columns, but the two-dimensional methods may identify features along the diagonals that do not exist in the original image. Likewise, the polynomials and splines that are linear in degree fit functions that vary either upwards or downwards at a constant rate and may miss features that jump upwards and downwards, but the polynomials and splines that are higher in order may identify oscillations that do not exist.
3. Evaluation of demosaicing routines
If the objective of a demosaicing routine is to estimate an unmosaiced image, where the intensity is observed at every pixel, with a demosaiced image, where the intensity is observed at some pixels and predicted at others, then the performance of the demosaicing routine should be judged by the error between the unmosaiced image and the demosaiced image (Fig. 2). But while the demosaicing routine uses the knowledge of the intensities at one subset of known pixels to guess the intensities at another subset of unknown pixels, looking at each channel one-by-one, it cannot be evaluated by looking at either subset of pixels or any one of the channels in isolation.
This is partially because the impact of the demosaicing routine will depend on the application of the imaging system. For example, a human reviewing an image and a computer analyzing that same image will be sensitive to different types of error expressed over different regions of the image but will also fail for different thresholds and distributions of that error within the image. But this is also because the impact of the demosaicing routine will depend on multiple components of the imaging system. As one example, the performance of the demosaicing routine will indirectly be affected by the design of the image sensor. If the sensor can be redesigned so that it produces more known pixels and fewer unknown pixels, then the demosaicing routine can incorporate the greater amount of knowledge from the known pixels as it makes a smaller number of guesses at the unknown ones. As another example, the performance of the demosaicing routine will be directly affected by the implementation of the demosaicing routine. If the routine can be reimplemented so that it imposes constraints between the known pixels and unknown pixels that better reflect the scenes-of-interest, then the demosaicing routine can more effectively utilize the same amount of knowledge as it makes the same number guesses.
To permit a holistic analysis of a demosaicing routine that encompasses the applications of the imaging systems and the components of it, unmosaiced images and demosaiced images can be compared across four measures of image quality:
- 1.& 2. The sum, over all pixels, of the absolute errors between the unmosaiced image and demosaiced image, and the mean, over unknown pixels, of the absolute errors between the unmosaiced image and demosaiced image. Such statistics based on the absolute error are preferred to statistics based on the squared error because the sum and mean over the absolute errors will follow the disparity between the unmosaiced image and the demosaiced image in a proportionate manner—ensuring that simple indicators of improvement, like the percentage change, have a straightforward interpretation.
- 3. The 99th percentile, over unknown pixels, of the structural dissimilarity index between the unmosaiced image’s channels and the demosaiced image’s channels. The structural dissimilarity index is defined as one minus the structural similarity index where the structural similarity index can be computed with the “ssim” implementation in [19]; the percentile can be computed on each channel by itself or multiple channels pooled together.
- 4. The 99th percentile, over unknown pixels, of the color difference between the unmosaiced image’s channels and the demosaiced image’s channels. The color difference can be computed in the CIELAB color space via the CIEDE2000 definition of ΔE implemented in [20] after an appropriate transform from the image sensor’s RGB color space (where defined).
The absolute error and related statistics describe the actual difference between the unmosaiced image and the demosaiced image, while the color difference, structural dissimilarity, and related statistics describe the perceived difference between the unmosaiced image and the demosaiced image. Ideally, the actual difference and the perceived difference would track each other as the image sensor or demosaicing routine is modified, but the relationship between the two may be broken by the intricacies of the human visual system; as a result, it is important to identify cases where the actual difference is low, permitting consumption of images by computational methods, and cases where the perceived difference is low, permitting consumption of images by human observers.
Beyond this distinction in the interpretation of the statistics, it is worthwhile to note the other differences between these metrics. First, the sum of absolute errors and the mean absolute error both describe the actual error that can be attributed to the demosaicing routine; however, the sum, evaluated over all pixels, weights an indirect improvement from the image sensor more than a direct improvement to the demosaicing routine since it accounts for both the zero error from the image sensor’s observations of the known pixels and the non-zero error from the demosaicing routine’s predictions of the unknown pixel, while the mean, evaluated over unknown pixels, does the opposite since it accounts for error from the unknown pixels but not the known pixels. Second, the color difference and the structural similarity both describe the perceived error that can be attributed to the demosaicing routine; however, the color difference indicates the perceived change in the spectrum at each pixel, and the structural similarity indicates the perceived change in the texture around each pixel. The 99th percentile acts as an upper bound on these metrics, ensuring that rare but important image patches are not discounted during the evaluation.
4. Results for a single-mode visible image
To explore the performance of the image sensors and demosaicing routines on a scene with a wide variety of visible colors and textures, an unmosaiced image of an architectural structure was generated before mosaiced images for each image sensor and demosaiced images for each demosaicing routine were derived; the unmosaiced image and the demosaiced images were then compared.
4.1 Data collection
A commercial camera (DP1x, Sigma) including a stacked photodiode image sensor with no filter array was setup. A scene illuminated by the sun was then imaged by the camera (Fig. 3(A)). Since the commercial camera, like most color cameras, was equipped with an infrared cut-off filter the three-channel image that it provided could be treated like the three-channel image collected from a stacked photodiode image sensor covered entirely with short-pass filters (Fig. 3(A)). This process yielded a single 2640-by-1760-pixel still image of the Electrical and Computer Engineering Building at the University of Illinois at Urbana-Champaign which is shown in Fig. 3(B).
Fig. 3. The experimental setup and unmosaiced image for the single-mode dataset along with some demosaiced patches for various demosaicing routines. (A) A natural scene, with sunlight illumination, was imaged with a stacked photodiode image sensor equipped with no filter array. A near-infrared cut-off filter passed visible (VIS) light and rejected near-infrared (NIR) light, permitting collection of an unmosaiced visible image without interference from near-infrared light. (B) One unmosaiced image was included in the dataset, showing the front of the Electrical and Computer Engineering Building on the campus of the University of Illinois at Urbana-Champaign. (C) Two patches were examined in detail: One from the façade of the building and another from a tree in front of the building. The patches are presented for both imaging systems and all demosaicing routines to illustrate the difference between these systems and routines.
By properly sampling this unmosaiced visible image, we could emulate the short-pass mosaiced image that the bioinspired camera would see, with three short-pass observations covering the same half of the array, and the blue-green-red mosaiced image that the RGB-IR camera would see, with blue observations, green observations, and red observations covering different quarters of the array. By then interpolating these mosaiced visible images, we could compute the demosaiced images from the bioinspired camera and the RGB-IR camera—permitting comparisons between the unmosaiced image and the demosaiced images and evaluation of the imaging systems and their demosaicing routines.
4.2 Quantitative data analysis
Tables 1 and 2 present the sum of absolute errors, mean absolute errors, structural dissimilarities, and color differences computed for the visible channels across image sensors and demosaicing routines. Table S1 (Supplement 1) presents the sum of squared errors and the mean squared errors, computed similarly to their absolute counterparts, to facilitate comparisons with existing works that utilize the squared error instead of the absolute error.
Table 1. Sum of absolute errors (SAE) for image from single-mode dataset.a
Table 2. Mean absolute error (MAE), 99th percentile structural dissimilarity index (DSSIM), and 99th percentile color difference (ΔE) for image from single-mode dataset.a
The largest change in image quality was observed when migrating between image sensors, with the bioinspired sensor outperforming the RGB-IR sensor in every metric. The sum of absolute errors, averaged across channels, dropped by 47%–58%, depending on the demosaicing routine, while the mean absolute error dropped by 21%–37%. Furthermore, the structural dissimilarity, evaluated across the visible channels, dropped by 59%–66%, depending again on the demosaicing routine, while the color difference dropped by 41%–46%. The smallest decrease and largest decrease for the measures of actual error were associated with one-dimensional bilinear interpolation and two-dimensional bicubic polynomial interpolation, respectively, while the same quantities for the measures of perceived error were associated with two-dimensional bicubic spline interpolation and two-dimensional bicubic polynomial interpolation.
Recall that the sum of absolute errors accounts for both the known pixels with zero error and the unknown pixels with non-zero error, while the mean absolute error accounts only for the unknown pixels with non-zero error. Migrating from the RGB-IR sensor to the bioinspired sensor increases the number of known pixels by a factor of two—from one known pixel per channel to two known pixels per channel in each two-by-two-pixel block. The improvement in the sum of absolute errors thus suggests that this change to the image sensor induces a direct change to the image quality, by increasing the available number of known pixels, while the improvement in the mean absolute error indicates that this change to the image sensor induces another indirect change to the image quality, by increasing the available information on the unknown pixels.
Table 3. 99th percentile structural dissimilarity index (DSSIM) and 99th percentile color difference (ΔE) for two patches taken from single-mode dataset.a
A smaller change in image quality was observed when migrating between demosaicing routines, with the image sensor determining the superior demosaicing routine. For the bioinspired sensor, the mean absolute error, averaged across channels, dropped by 12% from its maximum for one-dimensional bicubic spline interpolation to its minimum for two-dimensional bicubic polynomial interpolation. Meanwhile, the structural dissimilarity and color difference, evaluated across the visible channels, dropped by a maximum of 14% and 12%, respectively, with both metrics again minimized using two-dimensional bicubic polynomial interpolation. For the RGB-IR sensor, the mean absolute error dropped by 10% from its maximum for two-dimensional bicubic polynomial interpolation to its minimum for one-dimensional bilinear interpolation. Likewise, the structural dissimilarity and color difference dropped by a maximum of 6% and 4%, respectively, with both metrics again minimized using one-dimensional bilinear interpolation.
Based on these observations, it is possible to recommend a combination of image sensor and demosaicing routine that will optimize image quality for natural scenes. If a bioinspired sensor can be used, the highest performance across both sensors is achieved by pairing a bioinspired sensor with a two-dimensional bicubic polynomial demosaicing routine. If an RGB-IR sensor must be used, though, the highest performance for that sensor is achieved by pairing an RGB-IR sensor with a one-dimensional bilinear demosaicing routine. Comparing the choices for a bioinspired sensor and an RGB-IR sensor in this way reveals a notable difference between the sensors. Using the bioinspired sensor in its optimum configuration requires the demosaicing routine with the most representative power, indicating that the bioinspired sensor and its dense spatial samples permit a more complex function to be fitted to the scene. In contrast, using the RGB-IR sensor in its optimum configuration requires the demosaicing routine with the least representative power, indicating that the RGB-IR sensor and its sparse spatial samples only permit a simple function to be fitted to the scene.
4.3 Qualitative data analysis
Figure 3(C) shows two patches of the image that were analyzed as a part of this dataset in both their original unmosaiced forms and their final demosaiced forms as the image sensor and demosaicing routine were varied. Table 3 then presents, for each patch, the structural dissimilarities and color differences computed under these same variations.
For a given demosaicing routine, it can be seen that there is a dramatic difference in the demosaiced images produced from the bioinspired sensor and the RGB-IR sensor. Looking first at the patch taken from the building, which includes high frequency features confined to a single direction, the demosaiced images from the bioinspired sensor maintain the diagonal structure and red color pallet that is present in the original images, while the demosaiced images from the RGB-IR sensor introduce a horizontal structure and a yellow-purple color pallet that is not present in the original images.
Looking next at the patch taken from the tree, which includes high frequency features oriented in all directions, the demosaiced images from the bioinspired sensor looks sharp, with the highlights off the leaves and the lowlights between the branches maintaining their shape and position and the color pallet naturally confined to shades of green; meanwhile, the demosaiced images from the RGB-IR sensor looks dull, with the highlights and lowlights blurring together and the color pallet unnaturally expanding to include red, orange, yellow, cyan, blue, and violet.
For a given image sensor, though, it can also be seen that there is a much subtler difference between the different demosaicing routines. For example, the demosaiced images from the bioinspired sensor showing the patch taken from the building illustrate an alternating pattern of bright pixels and dark pixels along edges known as a zipper artifact. This occurs when interpolation is applied in a direction that is not completely perpendicular to an edge but is instead partially parallel to that edge (for example, when interpolation is applied horizontally or vertically while the edge is diagonal). The images from the one-dimensional bilinear demosaicing routines include a strong zipper artifact in the vicinity of the bright colored slats, while the images from the one-dimensional bicubic demosaicing routines include an additional zipper artifact in the voids between these slats. The images from the two-dimensional bicubic demosaicing routines are not wholly free of zipper artifacts, but they do produce a cleaner-looking result by controlling the strength and extent of the artifact. When operating along one dimension, the demosaicing routine must interpolate along the rows or the columns even if an edge lies along the diagonal, with a bilinear routine keeping the resulting artifact to a smaller neighborhood and the bicubic routine permitting it to spread to a larger neighborhood. When operating along two dimensions, though, the demosaicing routine can effectively interpolate an edge along the rows, the columns, or any line in between so long as the edge is adequately represented in the available pixels.
Alternatively, the demosaiced images from the RGB-IR sensor showing the patch taken from the tree illustrate unnatural colors between the highlights and the lowlights known as false color artifacts. This occurs when interpolation accumulates mismatches between the color channels (for example, when interpolation is applied to color channels sampled at different pixels). Nearly every demosaicing routine produces continuous regions of false colors that blur together, but the two-dimensional bicubic polynomial demosaicing routine produces individual pixels of false colors that stand apart from each other. The migration to a two-dimensional routine from a one-dimensional routine adds degrees of freedom to the interpolating function, while the application of a polynomial routine instead of a spline routine emphasizes local features while fitting the interpolating function. This combination of factors may cause a large deviation in the demosaicing routine’s output due to a small deviation in the demosaicing routine’s input (i.e., statistical overfitting) which can produce a mismatch between adjacent pixels even if the colors should be very similar.
5. Results for dual-mode visible/near-infrared images
To extend the evaluation of the image sensors and demosaicing routines already presented to a series of scenes including both visible content and near-infrared content, unmosaiced images, mosaiced images, and demosaiced images of multiple architectural structures were constructed and analyzed.
5.1 Data collection
A custom camera [1] including a stacked photodiode image sensor with no filter array was provisioned with a zoom lens (18-200mm F3.5-6.3 DC OS HSM, Sigma) set to an 18-mm focal length and an f/2.8 aperture. Scenes illuminated by the sun were then imaged by the camera. Although the custom camera did not include an infrared cut-off filter by design, it could be equipped with either a 700-nm short-pass filter (FESH0700, Thorlabs) or a 700-nm long-pass filter (FELH0700, Thorlabs) to capture the three-channel images that would be collected from a stacked photodiode image sensor covered entirely with short-pass filters or long-pass filters. As such, a six-channel unmosaiced image could be constructed by stacking the three-channel visible image captured with a short-pass filter and the three-channel near-infrared image captured with a long-pass filter. This process yielded ten videos, with a 1344-by-895-pixel resolution and a 10-frame length, which were cropped to an 1149-by-765-pixel resolution (to remove any filter-induced vignetting at peripheral pixels) and truncated to a single frame in length (to suppress any motion across the frames). The resultant still images, which underwent color correction at all pixels and color artifact suppression at saturated pixels, showcased various buildings and landmarks on the campus of the University of Illinois at Urbana-Champaign (Fig. 4).
Fig. 4. The unmosaiced images for the dual-mode dataset. Natural scenes, with sunlight illumination, were imaged with a stacked photodiode image sensor equipped with no filter array. The temporary installation of a 700-nm short-pass filter in front of the image sensor stopped most near-infrared light and permitted collection of unmosaiced visible images, while the similar installation of a 700-nm long-pass filter in front of the image sensor stopped most visible light and permitted collection of unmosaiced near-infrared images. Ten unmosaiced images were included in the dataset, showing several buildings and landmarks on the campus of the University of Illinois at Urbana-Champaign in the visible spectrum (A) and the near-infrared spectrum (B). The visible images and near-infrared images are both displayed by taking the values of the three visible channels and three near-infrared channels as the amounts of the three primary display colors.
By properly sampling this unmosaiced visible image, we could emulate both the short-pass mosaiced image and the long-pass mosaiced image that the bioinspired camera would see as well as the blue-green-red-near-infrared mosaiced image than the RGB-IR camera would see. Interpolation of the mosaiced image, computation of the demosaiced image, comparison against the unmosaiced image, and evaluation of the imaging systems and demosaicing routines then proceeded as previously described.
5.2 Data analysis
Tables 4 and 5 present the sum of absolute errors, mean absolute errors, structural dissimilarities, and color differences computed for the visible and near-infrared channels across image sensors and demosaicing routines. Table S2 (Supplement 1) presents the sum of squared errors and the mean squared errors, computed similarly to their absolute counterparts, to facilitate comparisons with existing works that utilize the squared error instead of the absolute error.
Table 4. Sum of absolute errors (SAE) for images from dual-mode dataset.a
Table 5. Mean absolute error (MAE), 99th percentile structural dissimilarity index (DSSIM), and 99th percentile color difference (ΔE) for images from dual-mode dataset.a
Although it had already been observed with the single-mode image, the image quality observed from the bioinspired sensor’s visible channels still surpassed that observed from the RGB-IR sensor’s visible channels. When averaged across these channels, the sum of absolute errors and the mean absolute error dropped by 44%–48% and 15%–23% depending on the demosaicing routine, and when evaluated across the same channels, the structural dissimilarity and the color difference fell by 48%–55% and 4%–13%. As could only be observed with the dual-mode images, though, the image quality observed from the bioinspired sensor’s near-infrared channels also exceeded that from the RGB-IR sensor’s near-infrared channels. When evaluated at the shared near-infrared channel, the sum of absolute errors and mean absolute error dropped by 46%–55% and 20%–33% depending once more on the demosaicing routine, and when evaluated over the available near-infrared channels, the structural dissimilarity fell by 63%–70%. This improvement in the three visible channels and one near-infrared channel common to both sensors was accompanied with strong performance from the two near-infrared channels available only from the bioinspired sensor and not from the RGB-IR sensor. Comparing the structural dissimilarities and color differences observed here in this analysis of the dual-mode images with those quantities observed earlier during the qualitative analysis of the single-mode image (Table 3), it can be appreciated that the textures in and relationships amongst channels should be similar, if not slightly better, for the dual-mode images’ near-infrared channels than the single-mode image’s visible channels (Fig. 3(C)).
Consistent with the single-mode image, performance for the dual-mode images was largely optimized with a two-dimensional bicubic polynomial demosaicing routine for the bioinspired sensor, with performance on the visible structural dissimilarity and visible color difference optimized instead with a one-dimensional bilinear routine. Likewise, performance was chiefly optimized with a one-dimensional bilinear demosaicing routine for the RGB-IR sensor, with performance on the near-infrared mean absolute error alone optimized with a one-dimensional bicubic polynomial routine.
6. Conclusion
There has been growing interest in dual-mode imaging systems sensitive to both visible light and near-infrared light as consumers increasingly expect their devices to see the world as a human would while providing insights beyond the human ken. This perceived opportunity, which spans markets, has motivated the development of a six-channel bioinspired imaging system as well as more conventional four-channel RGB-IR imaging systems which have been used in biomedicine, agriculture, consumer devices, and industrial processes. Yet there has been comparatively little interest in the demosaicing routines needed to transform the raw partial-resolution images captured by these imaging systems into the processed full-resolution images comprehensible to humans and computers. The work that has been done in demosaicing routines for RGB-IR imaging systems usually assumes a correlational structure between the visible channels and the near-infrared channels—an assumption that may not be safe for all channels from all imaging systems, especially when those channels encode different information.
Polynomials and splines, which have long been used in the demosaicing routines for color imaging systems, can also be used in the demosaicing routines for dual-mode visible/near-infrared imaging systems while also ensuring that each channel is treated independently and identically. The choice of interpolating function determines the scope of the demosaicing routine, offering an optimization for small, disjointed patches or large, contiguous ones, while the dimensionality and order determine the flexibility and resiliency, permitting a tradeoff between accuracy on spatially varying signals and sensitivity to spatial noise. Of course, it is not always obvious whether a demosaicing routine should operate on one-dimension or two-dimensions or evaluate to first-order or third-order, so it is necessary to assess the demosaicing routines on a set of representative scenes. Such a comparison between the unmosaiced images projected onto the imaging system and the demosaiced image read out of it should encompass measures of the actual error relevant to a quantitative analysis by a computer as well as measures of perceived error relevant to a qualitative observation by a person.
Towards this end, we implemented demosaicing routines for both a bioinspired image sensor and an RGB-IR image sensor and evaluated those demosaicing routines on one visible image and ten visible/near-infrared images. The switch between image sensors yielded the largest reduction in error between unmosaiced images and demosaiced images, with the migration from an RGB-IR image sensor to a bioinspired image sensor pulling down the mean absolute error by 21%–37%, the structural dissimilarity by 59%–66%, and the color difference by 41%–46%. Nevertheless, the switch between demosaicing routines still yielded a substantial reduction in this error, with the respective transitions for the RGB-IR sensor and the bioinspired sensor reducing the mean absolute error by 10% and 12%, the structural dissimilarity by 6% and 12%, and the color difference by 4% and 14%. While the overall image quality from the RGB-IR image sensor thus paled in comparison to that from the bioinspired sensor, the performance from the former sensor was still optimized by using a one-dimensional bilinear demosaicing routine, while performance from the latter sensor was instead optimized by using a two-dimensional bicubic polynomial demosaicing routine. This result isn’t necessarily intuitive since one might expect that the less complicated RGB-IR image sensor might benefit from a more powerful demosaicing routine and that the more complicated bioinspired image sensor might benefit from a less powerful one; however, it points to a need for imaging engineers to consider every aspect of their imaging systems if performance is to be maximized.
As additional research is therefore directed at demosaicing routines for dual-band imaging systems, one limitation of this study that should be addressed regards the design of the RGB-IR imaging system. The uniform RGB-IR imaging system that was considered here utilizes an array of optical filters that packs each two-by-two-pixel block with four different optical filters, thus sampling each channel with equal frequency and regularity. But other RGB-IR imaging systems proposed in the literature [17,21–23] have looked to favor the visible channels or the near-infrared channels by filling larger blocks with those same optical filters, thus sampling each channel with a different frequency or regularity. Selecting the optimal system for a given application requires prior knowledge of the contents of the channels and the correlations between them. By focusing on the uniform RGB-IR imaging system, this study was committed to an implicit assumption that each channel was equally critical and complex, in addition to the explicit assumption that the channels were uncorrelated. But by expanding to the alternative RGB-IR imaging systems, a future study could inform applications where the cost of an error, and the risk of that error, may not be the same from channel to channel. A discussion of multiple imaging systems utilizing different filter arrays would also invite an exploration into the similarities and difference between these imaging systems in the frequency-domain—a technique which has proved effective in comparisons between other sensors [24].
Another limitation that should be addressed regards the design of the demosaicing routines. In single-mode imaging applications, it has long been understood that there is a high degree of correlation between the chromaticity coordinates of natural images [25], so it only makes sense that a demosaicing routine can successfully exploit that correlation when applied to a natural image [26,27]. But in dual-mode imaging applications, where either the visible channels or the near-infrared channels may not necessarily follow the statistics of natural images because of structure that has been specifically engineered, it is important that such correlations be exploited only where they can safely be assumed. The demosaicing routines explored in this work operate as if every channel is statistically independent when, in reality, those channels from similar bands are likely to be related—like the color channels in color demosaicing routines. The demosaicing routines should thus be extended so that they can exploit the correlations that exist, not amongst all channels, but within defined subsets of channels since these correlations contribute prior information that should improve performance across image sensors.
For uniform RGB-IR image sensors, alternative RGB-IR image sensors, and bioinspired image sensors alike, the application of some correlational structure may permit the unknown channels at each pixel to be informed by the known channels at that pixel. As such, a demosaicing routine may recover more structure from an image since the effective spatial resolution of a set of channels with an established relationship amongst each other will be greater than the actual spatial resolution of a single channel taken by itself. For the bioinspired image sensor, though, this may also permit the three unknown channels at each pixel to inform each other. Since the sensor provides images where all three visible channels are available at one subset of pixels and all three near-infrared channels are available at another subset of pixels, the joint distribution for the visible channels or the near-infrared channels can be empirically estimated by simply indexing into one subset of pixels or the other. This makes the extraction of the correlation between the channels at the known pixels and the enforcement of that correlation between the channels at the unknown pixels especially elegant since it can be done on a per-image basis.
Funding
Office of Naval Research (N00014-19-1-2400, N00014-21-1-2177); National Science Foundation (2030421); Air Force Office of Scientific Research (FA9550-18-1-0278); Congressionally Directed Medical Research Programs (W81XWH-19-1-0299).
Acknowledgments
The authors would like to thank Zhongmin Zhu and Zuodong Liang, of the University of Illinois at Urbana-Champaign, for their help with the development of the bioinspired imaging system. The authors would also like to thank Zhongmin Zhu for his help with the collection of the dual-band visible/near-infrared images.
Disclosures
V.G. is listed as an inventor on Patent US20200120293A1, “Multispectral imaging sensors and systems,” which encompasses an imaging system described in this work. The authors are not aware of any other competing interests.
Data availability
All results associated with this study are present in the paper. The code used in the analyses is available in Code 1 (Ref. [28]), while the data consumed in the analyses is available in Dataset 1 (Ref. [29]).
Supplemental document
See Supplement 1 for supporting content.
References
1. S. Blair, M. Garcia, T. Davis, Z. Zhu, Z. Liang, C. Konopka, K. Kauffman, R. Colanceski, I. Ferati, B. Kondov, S. Stojanoski, M. B. Todorovska, N. T. Dimitrovska, N. Jakupi, D. Miladinova, G. Petrusevska, G. Kondov, W. L. Dobrucki, S. Nie, and V. Gruev, “Hexachromatic bioinspired camera for image-guided cancer surgery,” Sci. Transl. Med. 13(592), eaaw7067 (2021). [CrossRef]
2. J. M. D. Martino, F. Suzacq, M. Delbracio, Q. Qiu, and G. Sapiro, “Differential 3D Facial Recognition: Adding 3D to Your State-of-the-Art 2D Method,” IEEE Trans. Pattern Anal. Mach. Intell. 42(7), 1582–1593 (2020). [CrossRef]
3. S. G. Crihalmeanu and A. A. Ross, “Multispectral Ocular Biometrics,” in Face Recognition Across the Imaging Spectrum, T. Bourlai, ed. (Springer International Publishing, 2016), pp. 355–380.
4. K. Kidono and Y. Ninomiya, “Visibility Estimation under Night-time Conditions using a Multiband Camera,” in 2007 IEEE Intelligent Vehicles Symposium, (IEEE, 2007), 1013–1018.
5. S. Koyama, Y. Inaba, M. Kasano, and T. Murata, “A Day and Night Vision MOS Imager With Robust Photonic-Crystal-Based RGB-and-IR,” IEEE Trans. Electron Devices 55(3), 754–759 (2008). [CrossRef]
6. Z. Chen, X. Wang, and R. Liang, “RGB-NIR multispectral camera,” Opt. Express 22(5), 4985–4994 (2014). [CrossRef]
7. M. Martinello, A. Wajs, S. Quan, H. Lee, C. Lim, T. Woo, W. Lee, S.-S. Kim, and D. Lee, “Dual Aperture Photography: Image and Depth from a Mobile Camera,” in 2015 IEEE International Conference on Computational Photography (ICCP), (IEEE, 2015), 1–10.
8. N. Cui, T. York, R. Marinov, S. Mondal, S. Gao, J. Margenthaler, S. Achilefu, and V. Gruev, “A 110 × 64 150 mW 28 frames/s integrated visible/near-infrared CMOS image sensor with dual exposure times for image guided surgery,” in 2016 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2016), 101–104.
9. M. Garcia, M. Zayed, K.-M. Park, and V. Gruev, “A 1600 by 1200, 300 mW, 40 fps multi-spectral imager for near-infrared fluorescence image-guided surgery,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), 1–4.
10. B. Geelen, N. Spooren, K. Tack, A. Lambrechts, and M. Jayapala, “System-level analysis and design for RGB-NIR CMOS camera,” in Photonic Instrumentation Engineering IV, (SPIE, San Francisco, CA, 2017).
11. S. Blair, M. Garcia, N. Cui, and V. Gruev, “A 120 dB, Asynchronous, Time-Domain, Multispectral Imager for Near-Infrared Fluorescence Image-Guided Surgery,” in 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS), (IEEE, 2018), 1–4.
12. M. Garcia, C. Edmiston, T. York, R. Marinov, S. Mondal, N. Zhu, G. P. Sudlow, W. J. Akers, J. Margenthaler, S. Achilefu, R. Liang, M. A. Zayed, M. Y. Pepino, and V. Gruev, “Bio-inspired imager improves sensitivity in near-infrared fluorescence image-guided surgery,” Optica 5(4), 413–422 (2018). [CrossRef]
13. S. Blair, A. Deliwala, E. Chen, S. Subashbabu, A. Li, M. George, M. Garcia, N. Cui, Z. Zhu, S. Andonovski, B. Kondov, S. Stojanoski, M. B. Todorovska, G. Petrusevska, G. Kondov, and V. Gruev, “A 3.47 e− Read Noise, 81 dB Dynamic Range Backside-Illuminated Multispectral Imager for Near-Infrared Fluorescence Image-Guided Surgery,” in 2020 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2020), 1–5.
14. S. Blair, N. Cui, M. Garcia, and V. Gruev, “A 120 dB Dynamic Range Logarithmic Multispectral Imager for Near-Infrared Fluorescence Image-Guided Surgery,” in 2020 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2020), 1–5.
15. H. Tang, X. Zhang, S. Zhuo, F. Chen, K. N. Kutulakos, and L. Shen, “High Resolution Photography with an RGB-Infrared Camera,” in 2015 IEEE International Conference on Computational Photography (ICCP), (IEEE, 2015), 1–10.
16. I. Shopovska, L. Jovanov, and W. Philips, “RGB-NIR Demosaicing Using Deep Residual U-Net,” in 2018 26th Telecommunications Forum (TELFOR), (IEEE, 2018), 1–4.
17. Y. Monno, H. Teranaka, K. Yoshizaki, M. Tanaka, and M. Okutomi, “Single-Sensor RGB-NIR Imaging: High-Quality System Design and Prototype Implementation,” IEEE Sens. J. 19(2), 497–507 (2019). [CrossRef]
18. J. S. Ho, O. C. Au, J. Zhou, and Y. Guo, “Inter-channel demosaicking traces for digital image forensics,” in 2010 IEEE International Conference on Multimedia and Expo, (2010), 1475–1480.
19. S. van der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, T. Yu, and t. s.-i. contributors, “scikit-image: image processing in Python,” PeerJ 2, e453 (2014). [CrossRef]
20. T. Mansencal, M. Mauderer, M. Parsons, N. Shaw, K. Wheatley, S. Cooper, J. D. Vandenberg, L. Canavan, K. Crowson, O. Lev, K. Leinweber, S. Sharma, T. J. Sobotka, D. Moritz, M. Pppp, C. Rane, P. Eswaramoorthy, J. Mertic, B. Pearlstine, M. Leonhardt, O. Niemitalo, M. Szymanski, M. Schambach, S. Huang, M. Wei, N. Joywardhan, O. Wagih, P. Redman, J. Goldstone, and S. Hill, “Colour 0.3.16,” (Zenodo, 2020), retrieved https://doi.org/10.5281/zenodo.3757045.
21. T. Nedelcu, S. Thavalengal, and P. Corcoran, “Hybrid visible + near infrared color filter array for handheld consumer imaging applications,” in 2016 IEEE International Conference on Consumer Electronics (ICCE), (IEEE, 2016), 551–552.
22. N. Spooren, B. Geelen, K. Tack, A. Lambrechts, M. Jayapala, R. Ginat, Y. David, E. Levi, and Y. Grauer, “RGB-NIR active gated imaging,” in Electro-Optical and Infrared Systems: Technology and Applications XIII, (SPIE, 2016),
23. O. Skorka, S. Micinski, A. Perkins, B. Gravelle, X. Li, and R. Ispasoiu, “1.4 µm Pixel, 8 MP, > 5 µm epi, RGB-IR Image Sensor,” in International Image Sensor Workshop, (International Image Sensor Society, 2021).
24. E. Dubois, “Color Filter Array Sampling of Color Images: Frequency-Domain Analysis and Associated Demosaicking Algorithms,” in Single-Sensor Imaging: Methods and Applications for Digital Cameras, 1 ed., R. Lukac, ed. (CRC Press, Boca Raton, FL, 2009), pp. 183–212.
25. C. D. Hendley and S. Hecht, “The Colors of Natural Objects and Terrains, and Their Relation to Visual Color Deficiency,” J. Opt. Soc. Am. 39(10), 870–873 (1949). [CrossRef]
26. B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005). [CrossRef]
27. L. Xin, G. Bahadir, and Z. Lei, “Image demosaicing: a systematic survey,” in Visual Communications and Image Processing 2008, (SPIE, 2008).
28. S. Blair and V. Gruev, Code and Data for “Interpolant-based demosaicing routines for dual-mode visible/near-infrared imaging systems,” Zenodo, 2022https://doi.org/10.5281/zenodo.6564775.
29. S. Blair and V. Gruev, “Code and Data for “Interpolant-based demosaicing routines for dual-mode visible/near-infrared imaging systems”, Dryand, 2022https://doi.org/10.5061/dryad.fj6q573xj.
