1. Introduction

1.1 Background

There are three fundamental properties of light, and they are intensity, wavelength and polarization. The first two properties of light are encoded by our visual system as brightness and color, and the current imaging technology aims to replicate these two properties in high resolution with low spatial and temporal noise sensors. However, the third property of light, i.e. polarization, has been largely ignored by the imaging technology in part by the fact that the human eye is polarization insensitive.

Due to the recent advancements in nanotechnology and nanofabrication, polarization imaging sensors for the visible spectrum known as the division-of-focal-plane (DoFP) polarimeters have emerged on the scene [1–5]. These sensors monolithically integrate and combine pixelated polarization filters with an array of imaging elements. The pixelated polarization filters are composed of metallic nanowires, which are typically oriented at 0°, 45°, 90° and 135°. An example of this type of polarization imaging sensor is presented in Fig. 1 . One of the main advantages of division-of-focal-plane sensors is the capability of capturing polarization information at every frame. The polarization information captured by this class of sensors can be used to extract various parameters from an imaged scene, such as microscopy for tumor margin detection [6], 3-D shape reconstruction from a single image [7], underwater imaging [8], material classification [9,10] and cancer diagnosis [11,12].

 

Fig. 1 Block diagram of division-of-focal-plane polarization imaging sensor. The array of charge coupled device (CCD) imaging elements is covered with four pixelated linear polarization filters oriented at 0°, 45°, 90° and 135°

Download Full Size | PDF

There are two main disadvantages of the DoFP polarization sensors and they are: loss of spatial resolution and inaccuracy of the captured polarization information. Due to the sampling of the imaged scene with four spatially distributed pixelated polarization filters, the instantaneous field of view for the neighboring pixels in a 2-by-2 super-pixel configuration can be different from each other. Hence, the computed polarization information, such as the first three Stokes parameters, angle and degree of linear polarization, will contain an error from the true polarization signature of a target. Furthermore, the four types of polarization pixels which are distributed throughout the imaging array sub-sample the imaged environment by a factor of four and lead to loss of spatial resolution [13]. These two shortcomings need to be addressed in order to take full advantage of the real-time imaging capabilities of division-of-focal-plane polarization sensors.

Similar problems were encountered in color imaging sensors, when the Bayer pattern for pixelated color filters was introduced in the 1970s [14]. In order to recover the loss of spatial resolution in color sensors and improve the accuracy of the captured color information, various image interpolation algorithms have been developed in the last 30 years. Recent color interpolation methods have focused on developing computationally efficient algorithms for embedded hardware with high reconstruction accuracy [15, 16]. Although non-linear based interpolation methods have been widely used in the color domain, these algorithms do not directly translate to an implementation in the polarization domain due to the inherent difference of these two imaging modalities. For example, edge detection in the color domain is performed on the green channel in an image due to the high spatial resolution of this channel. Polarization domain does not favor any particular polarization orientation and therefore requires development of novel gradient-based interpolation methods tailored for this modality.

Interpolation algorithms for DoFP polarization sensors are still at the infancy stage, and to date, only few interpolation algorithms have been implemented and evaluated, such as bilinear [13], bicubic, bicubic spline interpolation [17, 18] and correlation-based interpolation [19]. These interpolation methods for division-of-focal-plane imaging sensors assume that information across the imaging array varies relatively smoothly and that there are no discontinuities in the low resolution image. This assumption fails in typically imaged scenarios, where multiple objects are imaged against a background. The discontinuity in either polarization or intensity domain, which is generated at the boundary between an object and the background, will generate false and often strong polarization signatures at the boundary. In other words, bilinear and bicubic interpolation methods are essentially low pass filters and smooth out the intensity information recorded by the four feature angle images. Due to the smoothing of the intensity images, artifacts at the edges of objects in the angle and degree of polarization image will be generated. These artifacts are undesirable for most imaging scenarios and should be suppressed through proper image processing techniques.

In this paper, we introduce a new gradient-based interpolation method with adaptive threshold selection for DoFP polarization sensors. Preliminary results of the proposed interpolation method are presented in [20]. In this paper, we have expanded the previously published results and have included additional theoretical analysis and measurements for the interpolation method. Specifically, we have included detailed theoretical overview of the proposed algorithm, performance evaluation of the proposed method as a function of scene brightness and included interpolation results implemented on a division-of-focal-plane sensor operating in the visible spectrum. The performance of the proposed interpolation algorithm is compared against the performance of the previously published interpolation methods, such as bilinear, bicubic spline interpolation and bicubic convolution [13, 17, 18]. Correlation-based interpolation method for polarization images has been proposed by Xu et al. [19]. This interpolation method has been evaluated on synthetic images and can achieve higher accuracy compared to bilinear and bicubic interpolation methods in the reconstructed high resolution images [19]. Unfortunately this interpolation method only applies to specific cases of interpolation on synthetic images and cannot be implemented on real-life images as presented in [19]. Hence, the performance of this interpolation algorithm is not compared against our proposed gradient-based interpolation method.

Based on the evaluation results, the proposed algorithm outperforms previously published interpolation algorithms for DoFP sensors in terms of highest accuracy of the recovered high resolution images. The effects of the scene dynamic range on the accuracy of the interpolated polarization information are also explored in this paper. The motivation for this evaluation comes from the fact that many polarization imaging applications are performed under poor light conditions such as haze imaging [21] and twilight imaging [22]. The dynamic range of a scene is relatively poor in these examples and the accuracy of the interpolation algorithms is of a particular interest.

1.2 Linear polarization computation

The DoFP imaging sensors capture both intensity and polarization information of an imaged scene at every frame. In terms of polarization properties of an imaged scene, typically two polarization sub-properties are of interest and they are the angle of polarization (AoP) and the degree of linear polarization (DoLP). The intensity, DoLP and AoP are computed via Eq. (1) through (3) respectively:

where I(x^o) is the intensity of the light wave filtered with a linear polarization filter in the x degree orientation. Following Eqs. (1) through (3), an imaging sensor capable of recovering polarization information from a scene has to sample the imaged environment with four linear polarization filters offset by 45°. In the case of DoFP polarization sensors, the linear polarization filters are pixelated and embedded at the focal plane of the imaging sensor (see the block diagram in Fig. 1). The fourth Stokes parameter is not captured with the sensor described in Fig. 1 due to the lack of quarter-wave retarder embedded at the pixel level in the imaging array. These filters can be incorporated in future versions of DoFP polarization imaging sensors by adding pixelated plasmonic quarter-wave retarders in combination with linear polarization filters [23–25].

The structure of this paper is as follows: in Section II the bicubic convolution and the gradient-based interpolation are presented; in Section III the experimental setup is introduced. In Section IV both visual and root mean square error comparisons are provided on a set of polarization images and for several different interpolation methods. In this section, adaptive threshold selection and the scene’s dynamic range influence on the proposed interpolation method are also discussed. Interpolation results obtained from a DoFP imager are presented at the end of this section. Concluding remarks are provided at the end of the paper.

2. Gradient-based interpolation

In this section, the bicubic convolution interpolation method is first briefly revisited. An approximation of the bicubic interpolation method is presented followed by an overview of the proposed gradient-based interpolation method. The proposed gradient-based interpolation method employs both the approximated bicubic interpolation method and bilinear interpolation in order to recover spatial resolution with high accuracy.

2.1 Bicubic convolution interpolation method

The commonly-used bicubic interpolation method attempts to fit a surface between four corner points of the low resolution image using a third-order polynomial function. In order to compute the intensity value of the target pixel via the bicubic interpolation method, the spatial derivatives in the horizontal, vertical and diagonal directions at the four corner points are initially computed. A total of twelve spatial derivatives are computed and combined with intensity values from the four corner points in order to compute the interpolated pixel value. Hence, the bicubic interpolation algorithm is very computational intensive algorithm and difficult to be realized in real-time on high resolution images.

The bicubic convolution method is an approximation of the bicubic interpolation method and is a more computationally-efficient algorithm than the bicubic interpolation at the cost of lower reconstruction accuracy [26]. The bicubic convolution process is first computed in the x-direction followed by convolution in the y-direction. The raw image, as recorded by the DoFP sensor, is composed of four sub-images sampled with four linear polarization filters respectively. Each one of the four sub-images has a quarter of the total resolution of the DoFP imaging array. For example, if the DoFP sensor is composed of 1000 by 1000 pixels, the sub-image corresponding to the pixels covered with 0° linear polarization filters contains 500 by 500 pixels. The sub-images corresponding to the pixels covered with 45°, 90° and 135° also contain 500 by 500 pixels respectively. These four sub-images are referred as low resolution images and the aim of the interpolation algorithms is to recover the full 1000 by 1000 pixels resolution for all four sub-images with the highest accuracy.

The interpolated pixel I(x,y) located between the low-resolution pixels f(x,y) and f(x + 1,y) is computed via the convolution process described by Eq. (4).

where the bicubic convolution kernel, W(s), can be described via Eq. (5):

In Eqs. (4) and (5), N is the number of pixels in the x-direction; M is the number of pixels in the y-direction; s is the relative location of a pixel in the convolution kernel. Expanding Eq. (4) into a matrix form, the bicubic convolution interpolation is computed via Eq. (6):

Special boundary conditions are applied to the first and last pixel of each row in the image. The conditions are described by Eq. (7).

Once the image is convolved in the x-direction, the same convolution method is repeated in the y-direction of the image in order to complete the interpolation process.

2.2 Gradient-based interpolation method

In this paper, we propose a gradient-based interpolation method in order to improve the quality of polarization information captured by a DoFP polarization sensor. The fundamental idea behind the gradient-based algorithm is the following: if an edge is encountered in the scene, then interpolation is performed along the edge and not across the edge. In the proposed implementation, bicubic convolution interpolation is performed along an edge in the image. Bilinear interpolation method is implemented in the smooth areas of the image, i.e. in the areas where no edges have been detected. For low spatial frequency features, both bilinear and bicubic interpolation results have similar error reconstruction and modulation transfer function (MTF) response [18]. Bicubic interpolation outperforms bilinear interpolation in terms of RMSE and MTF for medium and high spatial frequency features. Due to the lower computation complexity of executing bilinear interpolation over bicubic convolution and the similar accuracy in the reconstructed images between both methods, bilinear interpolation is employed on the smooth features of the image.

In order to implement this algorithm, the boundary discontinuities (or edges) in an image are initially computed. An edge in an image is defined as a place where the spatial gradient exceeds a given threshold, and typically identifies an area in the image where one object ends and another one starts. For example, in color images, a high gradient exists between two objects with different colors or intensities. In polarization images, a high gradient exists between objects with different intensities, angles of polarization or degrees of linear polarization. These gradients imply that edges also exist in the 4 composite images, i.e. in the 0°, 45°, 90° and 135° images.

In the gradient-based interpolation method, the bicubic convolution and the bilinear interpolation methods are expanded to incorporate gradient selectivity features. To this extent, the gradients across the vertical, horizontal and two diagonal directions in a 7-by-7 low-resolution pixel neighborhood on all the four images (0°, 45°, 90° and 135°) are computed using Eq. (8).

In the above equations, D_0° is the gradient in the horizontal direction; D_90° is the gradient in the vertical direction and so on. Figure 2 illustrates how the gradient in the 135° diagonal and horizontal direction are computed. The black pixel in Fig. 2 denotes the target pixel of the interpolation computation.

 

Fig. 2 (a) 135° direction gradient example. (b) 0° direction gradient example. Note: the blue pixels are pixels of known value, the black one is the target pixel, the gray ones are the pixels with same type as the target pixel, and the white ones are pixels of unknown value.

Download Full Size | PDF

An optimal gradient detection filter should satisfy the following three goals: an accurate detection selectivity of edges, an accurate localization of edges and minimum response to false edges [27]. Following the above mention criteria, a small convolution kernel (i.e. 2 by 2 pixels window) will be very sensitive to image noise and will result in many false-negative edges. A large convolution kernel will smooth out the image, which will suppress many edges that should be detected. We have selected an intermediate size for the convolution kernel (i.e. 7 by 7 pixels window), which provides a balance between suppressing many false edges due to noise as well as keep the image fidelity in order to detect the desirable edges of objects.

Depending on the relative positions between known and unknown pixels, the interpolation is implemented first on the pixels with known value in the diagonal directions, followed by interpolation on the pixels with known values in its horizontal or vertical direction. As shown in the following steps:

1) If the ratio of the gradient in the 45° direction over the gradient in the 135° direction exceed a threshold, i.e. D_45°/ D_135°>T_threshold which indicates an edge along the 135° diagonal direction, the bicubic convolution method is applied to the target pixel along the 135° diagonal direction. Similarly, if the ratio of the gradient in the 135° direction over the gradient in the 45° direction exceed a threshold, i.e. D_135° / D_45° > T_threshold, the bicubic convolution method is applied to the target pixel along the 45° diagonal direction. In all other cases, the intensity value of the target pixel is the average of the intensity values of its four adjacent pixels in the diagonal directions.

2) If the ratio of the gradient in the 0 degree direction over the gradient in the 90 degree direction exceed a threshold, i.e. D_0°/D_90°>T_threshold which indicates an edge along the vertical direction, the bicubic convolution is applied to the target pixel along the vertical direction. If the ratio of the gradient in the 90 degree direction over the gradient in the 0 degree direction exceed a threshold, i.e. D_90°/D_0°>T_threshold, bicubic convolution is applied to the target pixel along the horizontal direction. In all other cases, the intensity value of the target pixel is the average of its four adjacent pixels in horizontal and vertical directions.

The T_threshold value is decided using histogram distribution of the derivative ratios in diagonal direction (D_45°/ D_135°) across the imaging scenes for all four angles (0°, 45°, 90° and 135°). From our experimental results, we have determined that setting the T_threshold value to be between 55% and 65% of the gradient’s ratio cumulated distribution function (CDF), yields to best reconstruction results as explained in the results section (see Sec. 4.3).

The computational complexity of an interpolation method is an important consideration for real-time implementation. The proposed gradient-based interpolation is computational more intensive compared to bilinear and bicubic convolution method. For example, the edge convolution filter requires 2 multiplications, 34 additions and one thresholding operation per pixel. Bicubic convolution method requires 24 multiplications, 23 additions and bilinear interpolation requires 4 multiplications and 3 additions. The overall computational complexity of the gradient-based method will depend on the spatial features of the imaged scene, while bilinear and bicubic interpolation will have a deterministic number of computational steps.

3. Experimental setup

In order to evaluate the accuracy of different interpolation methods, the true high-resolution polarization image must be known a priori. Division-of-focal-plane (DoFP) polarization imaging sensors can only generate low-resolution images. Thus, the performance of the interpolation algorithms cannot be compared against the true polarization signature since it is not known a priori. In order to circumvent this problem, a set of images are acquired by using a gray-scale CCD imaging sensor (Kodak KAI-4022, 40dB SNR, 60dB Dynamic Range) together with a linear-polarized filter rotation stage (Thorlabs NR360s, 5 arcmin accuracy) placed in front of the sensor. The linear-polarized filter is rotated at 0°, 45°, 90° and 135° orientation thus obtaining four co-registered high-resolution images. These four images are regarded as the “true” high-resolution images since they sample the optical field at every pixel with four different linear polarization filters offset by 45 degrees.

Next, the four “true” high-resolution images are decimated by following the sampling pattern of the DoFP polarization imaging sensor shown in Fig. 1. Hence, four low resolution images at 0°, 45°, 90° and 135° feature angles are obtained and essentially simulate an image that would be acquired from a DoFP sensor. The four low resolution images are interpolated via different methods and high resolution images are computed. The final high resolution interpolated images are compared against the true high-resolution images that were originally recorded. This experimental set-up aims to provide fair comparison of the reconstruction error between bilinear, bicubic and gradient-based interpolation methods.

The four high resolution images are recorded with a gray-scale camera with a rotating linear polarization filter and might contain alignment errors in the optical set-up. The optical misalignment in the image acquisition set-up will generate errors in the acquired polarization data. These errors are not critical for our set of experiments since the interpolation algorithms are implemented on the decimated images computed from the original high resolution images. The interpolated images are compared against the original high resolution images in order to estimate the accuracy of the method. Hence, any optical misalignment errors would equally affect the original high resolution images as well as the interpolated images.

In Figs. 3(a) -3(c), the intensity, degree of linear polarization (DoLP) and angle of polarization (AoP) results of a still toy muse are presented (Integration time: 100 msec). This figure presents the “true” high-resolution images and will be used to compare the accuracy of the interpolated images. The polarization sub-properties, DoLP and AoP, are mapped to grayscale and false color scales respectively (Figs. 3(b), 3(c)). A second example of a toy soldier scene is presented in Figs. 3(d)-3(f), which is used to verify the validation of the adaptive threshold selection (Sec. 4.3).

 

Fig. 3 The true high-resolution image (a) muse-Intensity, (b) muse-DoLP, (c) muse-AoP, (a) soldier-Intensity, (b) soldier -DoLP, (c) soldier –AoP.

Download Full Size | PDF

4. Experimental results

In this section, visual comparison and root mean square error (RMSE) analysis are applied to the first test image in order to evaluate the performance of the different interpolation methods. Furthermore, the adaptive threshold selection of the proposed gradient-based interpolation method and scene’s dynamic range impacts on the interpolated results are presented in this section. Finally, the interpolated image results from an existing DoFP polarization imager are shown for further comparison.

4.1 Image visual comparison

A small region in toy muse images shown in Figs. 3(a)-3(c), depicted by a white square, is used for visual evaluation of the different interpolation algorithms. The region in the white box is expanded and presented in Fig. 4 .

 

Fig. 4 Comparison of different interpolation methods on the intensity, DoLP and AoP. (a) True polarization, (b) bilinear interpolation, (c) bicubic spline interpolation, (d) bicubic convolution and (e) gradient-based interpolation.

Download Full Size | PDF

The set of images in Fig. 4 are organized as follows: the first column of images presents the intensity images; the second column of images presents the degree of linear polarization images and the third column presents the angle of linear polarization images. The first row of images (Fig. 4(a)) presents the true high-resolution polarization images and are used to visually compare the reconstruction accuracy of the different interpolation methods presented in the following four rows.

The images in Fig. 4(b) are obtained via bilinear interpolation and show strong pixilation effects in the intensity, DoLP and AoP images. In all three images, the details of the muse's horn are pixelated; the edges (marked as A, B and C) and the background are accentuated and show strong “toothed artifacts” due to the large error introduced by the bilinear interpolation.

The images presented in Figs. 4(c) and 4(d) are obtained via the bicubic spline and bicubic convolution interpolation methods respectively. The two bicubic-based interpolation methods show similar results. In these sets of images, the details of the muse's horn in both DoLP and AoP are recovered with higher accuracy than with bilinear interpolation. Pixilation effects are visibly reduced using bicubic interpolation when compared to bilinear technique. However, the DoLP and AoP images present visible “ripple effect”, especially in the edge area (marked as A, B and C) between the muse's horn and the background.

The images presented in Fig. 4(e) are the results of gradient-based interpolation (T_threshold: 60% CDF). This set of images is well-recovered and closely resembles the true polarization images. There are no large pixilation artifacts observed in any of the three polarization images. Hence, the gradient-based interpolation method visually recovers a high-resolution image with the highest accuracy.

4.2 RMSE comparison

In order to provide a numeric comparison of the accuracy of the different interpolation methods, the root mean square error (RMSE) evaluating method is applied and is shown in Eq. (9):

where O_c(i, j) is the true value of the target pixel, i.e. the image presented in Fig. 4(a), I_c(i, j) is the interpolated intensity value of the target pixel, M and N represent the number of rows and columns in the image array respectively.

For the toy muse test images, the RMSE results for the different interpolation methods are shown in Table 1 . The minimum RMSE for I(0°), I(45°), I(90°), I(135°), intensity, DoLP and AoP images is obtained via the gradient-based interpolation method. The bilinear interpolation method introduces the largest error for all the comparisons, while the bicubic spline and bicubic convolution interpolation methods showed similar error performance with the latter one being computationally efficient.

Table 1. The RMSE performance comparison for toy muse

View Table

4.3 Adaptive threshold selection

Accurate threshold selection for discerning edges in an image is a key part for most edge-detection methods. In order to determine an edge in an image, most algorithms use an experimentally determined edge-threshold and assign it as a constant for a set of images. Although this method is computationally efficient, it does not yield to a good edge detection results as edge-thresholds tend to vary between images.

In the proposed gradient-based interpolation method, the threshold is determined using the cumulated distribution function (CDF) of the ratio between two orthogonal spatial derivatives, i.e. D_0°/D_90° or D_45°/D_135°., for the intensity image of each angle separately, and one single threshold value is assigned to each intensity image by assuming the spatial derivatives in different directions have the same histogram distribution. The edge-thresholds are determined from the CDF by selecting different percentile of the function. For example, selecting the threshold value to be 100% of the CDF effectively suppresses edge detection and yields the same RMSE as a bilinear method. If the threshold value is set to 30% or lower of the CDF, many false positive edges can be selected which will lead to RMSE similar to the bicubic convolution. If the edge-threshold is set to 0% of the CDF, all areas in the image will be defined as edges and bicubic convolution interpolation will be applied across the entire image.

In order to determine a well-suited threshold value, the RMSE of the DoLP and AoP image are computed for different threshold values. Since both show the similar trend, only AoP results are shown in Fig. 5 to keep the briefness.

 

Fig. 5 Different CDF threshold selection (a) the normalized AoP RMSE of the toy muse, (b) the normalized AoP RMSE of the toy soldier .

Download Full Size | PDF

The normalized AoP RMSE (the ratio of the RMSE to the maximum value) of the muse image is evaluated for five different integration times of the imaging sensor in order to emulate different dynamic ranges in a scene and is presented in Fig. 5(a). For example, low integration times allow for few photons to be registered by the array of photodiodes and the dynamic range of a scene is very low. Long integration time allow for higher number of photons to be collected by the array of photodiodes and hence have a higher dynamic range of the scene (see Fig. 7). The normalized RMSE of the AoP image for the toy soldier image as a function of the edge-threshold is shown in Fig. 5(b). The RMSE error for both set of images follow similar trend with respect to the edge-threshold value.

For all five different integration times, the minimum RMSE is obtained when the threshold value is set between 55% and 65% of the CDF. Hence, the threshold for computing edges in a scene is set to 60% of the CDF. This rule of thumb is employed on several different sample images that we have collected and consistent results for minimum RMSE are obtained for each image. Therefore, setting the edge-detection threshold between 55% and 65% of the CDF is a good guideline for any image.

4.4 Dynamic range impact on interpolation

The DoFP polarization sensors are used to image scenes with various dynamic ranges, ranging from very dim scenes to very bright ones. Hence, the accuracy of the interpolated polarization information as a function of a scene’s dynamic range needs to be closely evaluated. In this section, a toy muse scene, which is presented in Figs. 3(a)-3(c), is imaged under different integration times. The integration time of the imaging sensor is swept between 0.5 msec and 120 msec. Note, similar results are obtained when the aperture of the sensor is modulated or the illumination of the scene is varied between experiments. For briefness of the paper, we only present the data collected under different integration periods.

The normalized RMSE (the ratio of the RMSE to the maximum value) results for the intensity, DoLP and AoP as a function of integration time are shown in Figs. 6(a) -6(c) respectively. Figures 6(d)-6(f) present the corresponding normalized standard deviation (STD) of the RMSE for the intensity, DoLP and AoP respectively. The RMSE and STD figures are computed across all pixels in the imaging array. The STD plot follows similar trend as the RMSE plots presented in Figs. 6(a)-6(c). Furthermore, the gradient-based interpolation method yields the smallest variations in the error measurements when compared to bilinear and bicubic interpolation across different integration times.

 

Fig. 6 The normalized RMSE results (a) Intensity, (b) DoLP, and (c)AoP; The normalized STD of RMSE (d) Intensity, (e) DoLP, (f) AoP, for different interpolation methods by scanning integration time.

Download Full Size | PDF

Sample images of the intensity, DoLP and AoP as a function of different integration times are presented in Fig. 7 . The images are collected with a normal camera with linear dynamic range of 60dB, i.e. high dynamic range camera is not used for collecting data. The images presented in Fig. 7(a) are collected with integration time of 0.5 msec. In this set of images, the intensity image does not provide visible contrast to the object against the background and the dynamic range of the scene, defined as the ratio of the difference between the maximum intensity and the minimum intensity pixel within the scene to the maximum allowed intensity of the pixel, is 0.5%. The dynamic range of the scene is increased to 3% in the second row of images and steadily increases in the rest of the images. The dynamic range of the scene for the intensity image in the last row is 70%. Based on the results shown in Fig. 7, the DoLP and AoP always provide good contrast and details of the target in a wide intensity range (any integration time larger than 4msec), while having inaccurate information for small dynamic range of the scene.

 

Fig. 7 Image results from the sensor under different integration time. (a) 0.5 msec, (b) 4 msec, (c) 12 msec, (d) 20 msec, (e) 40 msec and (f) 100 msec.

Download Full Size | PDF

Figure 6(a) shows the error in the intensity image for different interpolation methods. The RMSE linearly increases with the scene brightness, i.e. integration time of the imaging sensor. As the integration time increases, the intensity values in an image also increase. The interpolation methods attempt to compute a missing pixel value and the error will be larger for brighter scenes. For dim scenes, the intensity values are low and hence the interpolation errors will be naturally lower. For bright scenes, the intensity values are high and the interpolation errors will be accordingly higher than the errors from dim scenes. Another important observation can be made is that the gradient-based interpolation method has the lowest error compared to the other two interpolation methods.

Figures 6(b) and 6(c) present the normalized RMSE for the degree of linear polarization and angle of polarization images as a function of the integration time respectively. The error in the angle and degree of linear polarization is constant for majority of the different integration times. This can be explained by observing Eqs. (2) and (3). The angle and degree of linear polarization equations do not depend on the intensity measurements of the target. Hence, the errors introduced during the interpolation step in the image processing algorithm are going to be constant for different integration times, i.e. for different dynamic ranges of the scene. For low light conditions, the reconstruction errors increase due to the fact that the signal to noise ratio (SNR) of the imaging sensor is low. Hence, the interpolated value will have lower SNR or higher temporal variations. These temporal variations will lead to higher inaccuracies in the reconstructed values of the raw pixel values. From Fig. 6, it can be observed that the proposed gradient-based interpolation method outperforms bilinear and bicubic interpolation method under both short and long integration times of the imaging sensor.

Figures 6(d) through 6(f) present the corresponding normalized standard deviation (STD) of the RMSE for the intensity, DoLP and AoP respectively. The RMSE and STD figures are computed across all pixels in the imaging array. The STD plot follows similar trend as the RMSE plots presented in Figs. 6(a)-6(c). Furthermore, the gradient based-interpolation method yields the smallest variations in the error measurements when compared to bilinear and bicubic interpolation across different integration times.

4.5 Interpolation results of DoFP polarization imager

The three interpolation methods (bilinear, bicubic-spline and gradient-based) are applied to the image output of a high-resolution DoFP polarization imaging sensor [1], so that their reconstruction performance can be evaluated. The division of focal plane sensor is inserted in an underwater casing and real-life images of a lobster in the natural habitat have been recorded. The lobster images are presented in Fig. 8 . Since the camera is submerged underwater, there is not specular reflection from the surface of water in this imaging set-up.

 

Fig. 8 Interpolated results of DoFP imager on the intensity, DoLP and AoP. (a) bilinear interpolation, (b) bicubic spline interpolation and (c) gradient-based interpolation.

Download Full Size | PDF

Figure 8 shows the interpolated results of a lobster body recorded underwater. Figure 8(a) shows the bilinear interpolation results. In these set of images, strong horizontal and vertical artifacts can be observed on the lobster’s leg and tail part in both DoLP and AoP images. Figure 8(b) shows the bicubic interpolation results and many of the artifacts observed in the bilinear interpolation have been suppressed. Figure 8(c) shows the gradient-based interpolation results. In this set of images, many of the edge artifacts that caused strong polarization signatures in the bilinear and bicubic interpolation are suppressed with the gradient-based interpolation method.

Examining the above results for each interpolation method, the proposed gradient-based interpolation method presents the best visual results, and eliminates most of the noise and artifacts, which could be misinterpreted as “real” polarization information. It has been reported in the literature that lobsters, do not have polarization information between the shell ridges [28]. Furthermore, polarization information has not been observed between the shell ridges when the lobster has been imaged with division of time imaging sensors [29]. The divisions of time polarimeters produce accurate polarization information at the edges of stationary objects. The polarization information at the edges of the shell ridges, when recorded with division of focal plane polarimeters, is incorrectly computed via the bilinear and bicubic interpolation method and is suppressed via our gradient-based method.

4. Conclusion

In this paper, we have presented a new gradient-based interpolation method for the division-of-focal-plane (DoFP) polarization imaging sensor. The performance of the proposed gradient-based interpolation method is compared against the performance of several other interpolation methods in terms of visual testing and RMSE comparison. According to the results, the adaptive gradient-based interpolation method outperforms bilinear and bicubic interpolation methods using both visual quantitative evaluation as well as qualitative evaluation using an RMSE method.

The improvements in the reconstruction accuracy using the proposed gradient-based interpolation method are achieved by applying bicubic convolution interpolation to the edge area and bilinear interpolation the non-edge areas. The original discontinuity within the edge area can be well reconstructed, while the smoothness within the non-edge area can also be maintained. The gradient-based interpolation algorithm outperforms bilinear and bicubic interpolation algorithms under various dynamic range of a scene ranging from dim to bright conditions, and it could also bring a large improvement to the output quality of the real DoFP polarization imager.