Polarization imaging has recently gained interest since the emergence of snapshot devices such as polarization filter array (PFA) [1] cameras and color PFA (CPFA) cameras [2]. The latter is a combination of two filter arrays, a PFA and a color filter array (CFA), one on top of the other. A spatial modulation on the focal-plane array permits to sample the intensities ${I_{\theta ,c}}$ of the light field through several polarizing directions $\theta$ and spectral bandpass filters $\textit{c}$. The most common CPFA is a 12-channel sensor, which combines four angles of analysis equally distributed between ${0^ \circ}$ and ${180^ \circ}$ ($\theta = {0^ \circ}{,45^ \circ}{,90^ \circ}{,135^ \circ}$), and three color filters ($c = \textit{r},\textit{g},b$) arranged in a quad-Bayer [3] spatial configuration (see Fig. 1). The SONY IMX250 MYR [2] is one of these sensors and is commercially available. A linear polarization imager with four measurements is overdetermined, as only three different polarization states are needed to estimate the first three elements of the Stokes vector [4]. In this Letter, a high dynamic range (HDR)-like scheme is applied to combine the pre-processed data from multiple redundant Stokes element estimations. It leads to a dynamic range improvement in the Stokes images from a single color-polarization filter image.

HDR of irradiances and polarization signatures can occur in some scenes, e.g., a scene containing highly specular surfaces [5], diffuse reflection at low zenith angles, and shadow areas [6]. On one hand, low irradiance with a high degree of polarization [7] induces weak signal and thus a noisy image, inducing the degradation of the polarization information. On the other hand, high irradiance makes the camera response approach the saturation level, where the sensor operates in its nonlinear area. These particular scenarios can be present simultaneously in one snapshot CPFA image, which can be classified as follows:

These four scenarios are illustrated in Fig. 2, where the corresponding theoretical camera responses are shown for each polarizing angle of analysis. The dots are the four discrete intensity measurements ${I_\theta}$ that the polarimeter would have made under the specific cases. Plain lines are the simulated signals from the Malus law. According to it, intensity transmitted by a polarizer varies sinusoidally in the presence of a polarized incident beam. Even by employing a good auto-exposure algorithm, a polarization state image can exhibit pixels close to the saturation and noise levels. Combining such a noisy frame will result in noise in Stokes elements. If some outputs are still usable (not completely saturated or below the noise level), then it is still possible to estimate the Stokes elements. One may take care of minimizing the influence of noisy values in the calculation of these elements to prevent it from being further amplified, since the degree of linear polarization (DOLP) and the angle of linear polarization (AOLP) are computed from the Stokes elements by nonlinear processing, and thus are very sensitive to noise [4].

 

Fig. 1. CPFA sensor case. Each $2 \times 2$ super-pixel is covered by four linear polarization filters and one spectral filter.

Download Full Size | PDF

 

Fig. 2. Simulated theoretical polarization camera responses, with a varying input signal according to the Malus law.

Download Full Size | PDF

HDR imaging (HDRI) [8,9] is a kind of “hardware augmentation,” where multiple exposure images are fused through a computational imaging algorithm. Inputs are registered images of the same scene (i.e., the same irradiance levels) taken under different exposure times. A weighted average is applied during image fusion, taking into account the contribution of each image in the calculation. It is largely assumed that a pixel value can be judged to be within a “trustable” or a “dubious” zone of the digital value ranges of the camera. Several weighting functions have been investigated, taking into account either dubious pixels near the extremes, noise model of the camera, digital output variance, etc. [10]. Noise can likely be encountered in the dubious zone where undesired degradation in images may occur during capture, transmission, and/or processing, i.e., photon shot, dark current, or readout noise. I believe that the weighted average in HDRI is applicable on a single CPFA image where multiple redundant information can be reconstructed. As three polarization states are sufficient to compute the linear Stokes vector, one can use multiple ways of Stokes element calculations, and combine them. I present the proposed pipeline hereafter.

After a snapshot CPFA image acquisition, some pre-processing steps are performed on raw data such as linearization, polarimetric calibration [11], and spatial interpolation [12]. Here I assume that the input data ${I_{\theta ,c}}$ have been previously calibrated [11] and spatially interpolated. Only a sub-sampling is employed in this Letter to not introduce additional spatial noise. According to the above paragraphs, the noise considerations made are: (a) dubious pixels near the extremes should be excluded as much as possible from the Stokes calculation [13], and (b) the image noise structure is signal dependent: signal-to-noise ratio (SNR) increases with signal intensity [9].

These considerations are taken into account by means of a weighting function $\omega ({I_{\theta ,c}})$, applied to the 12 camera channels individually. The confidence given to the observed data is materialized by its individual weight. A well-known weighting function from HDRI [14,15] is used. The function is a combination of a broad hat function that eliminates dubious pixel values near the extreme levels, with the inverse of the camera response function (${\rm CRF}_{\theta ,c}^{- 1}$) and its derivative:

The weighting function (for the channel $\theta {= 0^ \circ}$, $c = r$) used for the IMX250 MYR sensor is shown in Fig. 3. The CRF was estimated using the Mitsunaga and Nayar method [9]. Note that only one CRF and one weighting function among the 12 channels are shown for convenience, since the different curves are very similar for this sensor. Figure 4 is a visualization of the well-exposedness where all channels are represented in false color. The scene is a plastic roll tape with a uniform and flat black background ($HpLi$ zone), and a directional illuminant coming from the right, which generates highlights. It can be noted that the polarization and color channels have very different results of well-exposedness, in particular in the light and dark areas.

 

Fig. 3. Weighting function $\omega ({I_\theta})$ used in this Letter.

Download Full Size | PDF

 

Fig. 4. (a)–(d) Sub-sampled polarization RGB images (without color correction) captured by an IMX250 MYR sensor. (e)–(p) Applying the weighting function over the 12 channels.

Download Full Size | PDF

After applying Eq. (1), I then estimate the Stokes elements ${S_0}$, ${S_1}$, and ${S_2}$. The key is to exploit the redundancy of intensity information that is inherent to PFA sensors, to take into account the potential non-equal contributions of noise during Stokes reconstruction. This redundancy has already been exploited in the literature to remove dead pixels [16], or for spatial resolution enhancement [17]. From the electromagnetic basis theory, ${S_0}$, ${S_1}$, and ${S_2}$ can be recovered from all of these equations:

${\hat S_1}$ and ${\hat S_2}$ are computed with Eq. (5), using a weighted average similar to that of the intensity. A normalization with respect to ${S_0}$ is applied to keep each term proportional to radiometric energy [19]:

Excess noise or nonlinearity in the data can still cause a large variation in the different estimates in Eq. (2). A reasonable assumption made in the literature is that polarization changes slightly with wavelength in the visible spectrum. Thus, ${S_1}$ and ${S_2}$ should change slightly over the three spectral bands. This assumption is experimentally highlighted in Refs. [20,21], where the differences in degrees of polarization among all spectral channels do not exceed 10%. To verify it on a large dataset, I did a statistical analysis on an existing dataset of noise-corrected RGB-polarization images from Ref. [22]. By computing the DOLP differences by pixel and over all of the observations, it appears that $\approx 99\%$ of the total amount of pixels exhibit less than 20% of DOLP differences (see histogram in Fig. S1 in Supplement 1). Therefore, to further reduce the effect of noise in the case of a high variability of Stokes elements, I use a weighted average over the color channels. A threshold $T$ is used, which permits to finally estimate ${S_1}$ and ${S_2}$ as this:

Visual results of applying the proposed method versus the standard one are shown in Fig. 5. The total intensity image ${S_0}$, along with ${S_0}$, ${S_1}$, ${S_2}$, AOLP, and DOLP for channel $c = r$ are visualized. The standard Stokes elements are computed from Eq. (2), where $M = N = 1$, which is the most employed way to compute Stokes components from four measurements. Figure 5(g) demonstrates the effect of the proposed methodology over a HpHi zone, where highlights are present. The dynamic range in highlight zones have been visually enhanced, especially on the white right side of the tape roll in ${S_0}$ and ${S_1}$. In DOLP images, the standard version shows more variance in the dark side of the tape roll and its shadow (LpLi). The same remarks can be made for the AOLP results. For the bottom right side of the tape roll (LpHi), the zone exhibits more smoothness in the AOLP image. A complete visualization for all spectral bands is available in Figs. S3, S4, and S5 in Supplement 1. The noise reduction is even better distinguished for the blue band whose pixel values are close to the noise level of the sensor.

 

Fig. 5. Top row: standard (std) Stokes computations. Bottom row: proposed Stokes computation [Eq. (7) with $T = 20\%$].

Download Full Size | PDF

A quantitative evaluation is done using reference HDR data. HDR data are calculated from five exposure times: $\Delta t = 10$, 20, 40, 80, and 160 ms. The middle exposure $\Delta t = 40\,\,\rm ms$ is used as a test image. Details about the HDR generation is given in Supplement 1. Then, the reference HDR Stokes images is computed from HDR data, using Eq. (2) with $M = N = 1$. It is important to note that the Stokes HDR images are generated from the same equations as for the standard one (i.e., without any averaging). peak signal to noise ratio (PSNR) results are shown in Table 1. We can see that the proposed method enhances significantly the Stokes components ${S_1}$ and ${S_2}$, especially where low irradiance is present (i.e., for $c = b$). The method does not enhance the ${S_0}$ component for all the spectral bands.

Table 1. PSNR Results for Both Computation Methods^a

View Table

To conclude, future works on this HDR scheme applied to a single PFA image can be envisaged through an extended quantitative evaluation, along with studying the effect of changing $T$. Thus, a validation over a large amount of CPFA HDR data could benefit further improvement of the methodology. Moreover, to highlight the potential effects of the proposed weighted average, investigations can be done on computer vision applications that involve ground-truth data, such as for shape from polarization, illumination direction estimation, or diffuse/specular separation.