Flexible focus function consisting of convex function and image enhancement filter

Kai Wang; Yuntao Qian; Minchao Ye; Zhijian Luo

doi:10.1364/OE.22.018668

1. Introduction

A focus function is always modeled as a function that has a maximum value at the best-focused image and decreases as the defocus increases [1]. Efforts are under way to develop focus functions relevant to widely used applications, such as automated microscopy, holography, deblurring, image refocusing, iris recognition, and depth of focus estimation [2]. In the past 40 years, more than 30 focus functions have been proposed. Some of them have the same principles or similar patterns, and some have the same formulae but different parameters. Thus, researchers have presented several categories of focus functions.

A large category of focus functions is derivative-based functions, which assume that well-focused images have more high-frequency content than defocused images [3,4], and apply image enhancement filters to extract the high-frequency content. (In this paper, an image enhancement filter is a convolution mask that uses smoothing or sharpening of the image contrast to make an image more useful for analysis.) Brenner et al. constructed a focus function by summing the squares of the second-order differences of the image [5]. Groen et al. presented three categories of focus functions and applied methods to sum the normalized variation of the image [6]. Krotov convolved the image by a Sobel filter and then summed the squares of the gradients of the image [7]. Subbarao et al. convolved the image by a Laplace filter and then summed the squares of the gradients of the image [8]. Nayar et al. convolved the image by a Laplacian filter and then used the l₁-norm of the gradient to construct the focus function [9]. Santos et al. presented several focus functions by summing the difference of the image higher than a threshold, summing the squares of the differences of the image higher than a threshold, and summing the squares of the image values higher than a threshold [10]. Daugman proposed a sharpening filter and summed the total high-frequency power of the 2D Fourier spectrum of the image [11]. Kang et al. proposed a sharpening filter smaller than Daugman’s and argued that the filter contains more high-frequency power [12]. Langehanenberg et al. used the determination of the logarithmically weighted cumulated Fourier spectra to determine the image sharpness of the reconstructed amplitude distributions [13].

A small category of focus functions is mid-frequency-based functions. Lee et al. detected objects distributed in the image using a mid-frequency discrete cosine transform (DCT) focus measure and then selected the target object through fuzzy reasoning [14]. Feng et al. presented an image clarity evaluation function based on the center blocking DCT method [15]. Wang et al. used combination filters to strengthen the image medium-frequency information in order to construct focus functions [16]. Another small category of focus functions is those based on statistics, in which focus functions are constructed using variance and correlation. Mendelsohn et al. summed the image values higher than a threshold [17]. Groen counted the number of pixels with values higher than a threshold [6]. Another small category of focus functions is histogram-based functions. Firestone et al. argued that a focused image contains more information than a defocused image and used the image entropy [18].

There are also focus functions that cannot be classified into the above categories. For example, Brázdilová et al. selected only areas of interest within an image and proposed further enhancements of existing methods [19]. Hamm et al. proposed a content-based focus searching method that uses a priori information on the observed objects by employing local object features and boosted learning [20]. Tsai et al. proposed a focus function that transforms the focus profile to the reciprocal domain [21]. Xu et al. proposed a focus detection criterion based on the phase contrast [22]. Ferraro et al. used an approximation of the Tamura coefficient to construct a focus function [23]. Gao et al. used the difference between the normalized amplitudes of red and green light [24].

Categorizing focus functions helps people understand how a focus function works. In addition, focus functions in one category always have one or more parameters, which enable researchers to construct new focus functions by tuning the parameters on the basis of the known focus functions. E.g., Elozory et al. proposed an indicator function that signifies whether the current pixel location is high contrast, a function that uses the binary median filter, and a function that returns the thresholded absolute gradient contrast strength in both the horizontal and vertical direction for a given pixel location [25].

However, no study has presented a focus function that both summarizes many known focus functions and has flexible and adjustable parameters. In this paper, we summarize 13 known focus functions [5–13] to produce the focus function Λ, which has two flexible and adjustable parameters: a convex function, and an image enhancement filter. For any given convex function and image enhancement filter, Λ is a focus function.

The proposed focus function Λ was first discovered using the observation that many of the known focus functions consist of a convex function and an image enhancement filter, as shown in Eq. (1), despite their differences.

Λ = \iint φ (I (x, y) * g) d x d y,

where I denotes the image, (x,y) denotes the coordinates of the image pixels, φ is a convex function, and g is an image enhancement filter.

A close observation of the focus functions that can be summarized by Eq. (1) yields two conclusions. First, the convex function φ can be a strictly convex function, such as the quadratic function $φ = x^{2}$ , or a non-strictly convex function, such as the absolute value function $φ = | x |$ . Second, it can be an axisymmetric function, such as the quadratic function $φ = x^{2}$ , or a non-axisymmetric function, such as the threshold absolute value function (e.g., $φ = 0$ when x < 0 and $φ = | x |$ when x ≥ 0). Further, the choice of the image enhancement filter g is rather arbitrary; it can be a one-dimensional image enhancement filter such as $g = [\begin{matrix} - 1 & 1 \end{matrix}]$ or a two-dimensional image enhancement filter such as $g = [\begin{matrix} 0 & - 1 & 0 \\ - 1 & 4 & - 1 \\ 0 & - 1 & 0 \end{matrix}]$ . Most image enhancement filters are sharpening filters, such as the Laplacian filter; however, we found that smoothing filters, for example, the Gaussian filter, also work.

The observations above imply that the construction of the convex functions and image enhancement filters is rather free. Thus, we wondered whether Eq. (1) succeeds for any convex function and image enhancement filter.

If the answer is yes, we would have by far the most flexible focus function, because the construction of the convex function and image enhancement filter is rather arbitrary. Furthermore, the new function will deepen our understanding of how focus functions work because so many known focus functions can be summarized by Eq. (1).

This paper is arranged as follows. First, we propose the new focus function Λ. Second, we create a model of the defocus problem. Then, using the model, we give a brief proof that Λ is a focus function for any convex function and image enhancement filter. We must note that the proof was based on the model; however, the result might also works in situations that do not fit the model. Finally, we constructed random Λs and experimentally applied them to both out-of-focus blurred images and motion-blurred images.

2. The proposed Λ

We found that 13 known focus functions can be expressed by Eq. (1), as shown in Table 1.

Table 1. List of Known Focus Functions that Can Be Expressed by Eq. (1).

View Table

Furthermore, we noticed that the construction of the convex functions and the image enhancement filters are rather free, as shown in Fig. 1. Therefore, we proposed that for any convex function and image enhancement filter, Λ is a focus function.

Fig. 1 Curves of convex functions φ in Table 1. Functions a, b, c, d, and e are convex; f and g are approximately convex. Function h is reversed convex because defocusing an image always shrinks the histogram, which reverses the smoothing process.

Download Full Size | PDF

The major challenge of proving that Λ is a focus function lies in how to model the defocus problems such that we can discuss it mathematically.

3. Modeling the defocus problem

In optical systems, an out-of-focus image is always described as the real image convolved by a point spread function (PSF). In polar coordinates, a diffraction-limited PSF has the form of a Bessel function of the first kind. For a wavelength λ with an f/# of F focusing radiation, it is

h (r) = 4 \frac{[J_{1} {(π r {(λ F)}^{- 1}]}^{2}}{{[π r {(λ F)}^{- 1}]}^{2}},

which can be approximated by the Gaussian function [26], as shown in Fig. 2. In addition, the Gaussian function is also always used to fit the modulation transfer function (MTF) [27–30]. This supports the idea that the PSF can be approximated by the Gaussian function, because the MTF and PSF are connected by a Fourier transform, and the Fourier transform of a Gaussian function is also a Gaussian function.

Fig. 2 PSF can be approximated by Gaussian function.

Download Full Size | PDF

Thus, in image processing, we can obtain a blurred image by convolving a less-blurred image using a Gaussian filter. This is a blurring model, However, in practice, this model is too simple. The Gaussian filter and PSF are both Dispersion Functions (in this paper, a Dispersion Function is defined as a real function with two properties: first, it is non-negative everywhere; second, its integral is equal to 1; this is shown in Eqs. (14) and (15) in the appendix). Therefore, we extended the model by modeling the blurred image as the less-blurred image convolved by a Dispersion Function, as shown in Eq. (3).

I_{2} = I_{0} * P S F_{2} = I_{0} * P S F_{1} * G = I_{1} * G,

where I₀ is the best focused image, I₁ is the less blurred image, I₂ is the more blurred image, PSF₁ is the PSF of I₁, PSF₂ is the PSF of I₂, and G is a Dispersion Function.

Further, when an image is convolved by an image enhancement filter, the input image values outside the bounds of the image are computed by implicitly assuming that the input image is periodic.

4. A brief proof

The proof of “Λ is a focus function” is equivalent to the proof of “Λ has a maximum value at the best-focused image and decreases as the defocus increases.”

Statements

In the proof, we will need the Dispersion Inequality [an inequality we proposed that describes the inequality in the dispersing process, as shown in Eq. (16) in the appendix].

Continuous form of the problem

Continuous form of the problem: Consider I₁ and I₂ to be images of the same object plane, where I₂ is more defocused than I₁. Let φ be a convex function in R² and g be an arbitrary function in R²; then we have

\iint φ ((I_{2} * g) (x, y)) d x d y \leq \iint φ ((I_{1} * g) (x, y)) d x d y .

Here, R² denotes two-dimensional vector space, and * is the convolution operator.

Proof

We start from the left side of Eq. (4). According to Eq. (3), we have

\begin{matrix} \iint φ ((I_{2} * g) (x, y)) d x d y = \iint φ (((I_{1} * G) * g) (x, y)) d x d y \\ = \iint φ ((I_{1} * (G * g)) (x, y)) d x d y, \end{matrix}

where G is a Dispersion Function.

On the other hand, according to the two-dimensional form of Eq. (25) in the appendix, we have

\iint φ ((I 1 * (G * g)) (x, y)) d x d y \leq \iint φ ((I 1 * g) (x, y)) d x d y,

where G is a Dispersion Function.

From Eqs. (5) and (6) we obtain Eq. (4). □

The proof is briefly illustrated in Fig. 3.

Fig. 3 Brief illustration of the proof.

Download Full Size | PDF

Discussion

The inequality might be the consequence of light energy dispersion during defocusing.

In real applications, images always have a finite height and width, in which condition Eq. (4) takes the form of

\iint_{Ω} φ ((I_{2} * g) (x, y)) d x d y \leq \iint_{Ω} φ ((I_{1} * g) (x, y)) d x d y,

where Ω is the image window.

The proof was based on a model, however, the result might also work in situations that do not fit the model, such as holography, which is a coherent imaging problem. Despite this limitation, the proof does suggest a focus function that always works under most situations. To study the performance of Λ in real situations, Λ was experimentally applied to blurred images.

5. Experimental results

The experiments were conducted in two parts. First, the experimental data, which contain the random Λs and the blurred images, were prepared. Then, we experimentally applied the constructed random Λs to the blurred images.

5.1 Preparation of experimental data

We began by constructing random Λs by generating random convex functions and random image enhancement filters.

First, random convex functions were constructed. In this paper, the convex function was modeled with a real polynomial in one variable. Because the second derivative of a polynomial is also a polynomial, and the second derivative of a convex function is non-negative everywhere, the problem of constructing a convex polynomial was reduced to the easier problem of constructing a positive polynomial. Furthermore, Positive polynomials have the property that “Every real polynomial in one variable is non-negative on R if and only if it is a sum of two squares of real polynomials in one variable” [31], which enables us to model a positive polynomial as

p_{m}^{2} + p_{n}^{2},

where p_m represents an mth-order polynomial, and p_n represents an nth-order polynomial.

Integrating Eq. (8) twice, we obtain the convex function

\iint (p_{m}^{2} + p_{n}^{2}) .

In this paper, we set m = 3, n = 2; thus, the convex function was an eighth-order polynomial.

Random convex functions can be constructed by randomizing the parameters in Eq. (9). In this paper, the gray values were normalized in the range of [0,1]. Consequently, to obtain convex functions significant in the range of [-1,1], the parameters were randomized in the range of [-1,1]. Examples of the obtained random convex functions are shown in Fig. 4.

Fig. 4 Shapes of 30 random convex functions generated by Eq. (9) in the range of [-1,1].

Download Full Size | PDF

Next, the random image enhancement filters g were constructed. They were constructed as 5 × 5 matrices, and the values of each matrix were randomized in the range of [-0.5, 0.5]. There are two types of image filters in the known focus functions. One type is smoothing filters, such as Gaussian filters, in which the sum of all the elements is equal to 1; the other type is sharpening filters, such as a Laplacian filter, in which the sum of all the elements is equal to 0. Here, the random filters g act as sharpening filters. Thus, the random filters g were normalized such that the sum of all the elements is equal to 0. The normalization was carried out by moving g vertically along the y-axis such that the sum of all the elements is equal to 0. Examples of the random image enhancement filters are displayed in Fig. 5.

Fig. 5 Meshes of four random image enhancement filters.

Download Full Size | PDF

Next, three series of blurred images were constructed: simulated out-of-focus blurred images, real out-of-focus images, and simulated motion-blurred images.

The simulated out-of-focus blurred images were constructed by convolving the focused images (as shown in Fig. 6) by the simulated PSFs. Four types of images were selected: a portrait (the Lena image), an object, an outdoor landscape, and a microscopic image. The PSFs were simulated using PSF Lab, which simulates the illumination PSF of a confocal microscope under various imaging conditions [32]. Most of the parameters in PSF Lab were set to the defaults except for the depth parameter, which denotes the distance from the microscope objective to the coverslip/sample interface. Adjusting the depth from −6 μm to 26 μm in 2 μm steps, we obtained 17 PSFs; examples are shown in Fig. 7. Then, convolving the focused image by the PSFs, we obtained the simulated out-of-focus blurred images, which were indexed from 1 to 17 with increasing depth; examples are shown in Fig. 8. The blurred images were best focused at index 8, and the defocus increased as the index increased or decreased.

Fig. 6 Original images: portrait (Lena), object, outdoor landscape, and microscopic image.

Download Full Size | PDF

Fig. 7 Examples of hot color maps of PSFs generated by PSF Lab.

Download Full Size | PDF

Fig. 8 Examples of blurred images. Blurred images were best focused at index 8, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

The real out-of-focus blurred images were captured by a microscope (XSP-06-1600). Here, a 5 × eyepiece and a 100 × objective were selected to observe a printed paper. The focal length was adjusted manually such that the object plane crossed the focal plane from the downside to the upside. Seventeen blurred images were captured and were indexed from 1 to 17 from the lowest to the highest object plane; examples are shown in Fig. 9. The blurred images were best focused at index 10, and the defocus increased as the index increased or decreased.

Fig. 9 Examples of real out-of-focus blurred images captured by a microscope. The images were best focused at index 10, and the defocus increased as the indexes increased or decreased.

Download Full Size | PDF

Motion blur was simulated by a filter to approximate rapid linear motion of a camera along an angle of 45 degrees with blurring degrees of 0, 2, 4, …, 16 pixels. Nine PSFs were constructed; examples are shown in Fig. 10. Then, by convolving the original image Lena by the PSFs as shown in Fig. 6, the motion-blurred images were obtained; examples are shown in Fig. 11. The images were indexed from 1 to 9 with increasing blurring degree. The blurred images were the least motion-blurred at index 1, and the blurring degree increased as the index increased.

Fig. 10 Examples of meshes of motion-blurred PSFs.

Download Full Size | PDF

Fig. 11 Examples of motion-blurred images. The images were the least motion blurred at index 1, and the blurring degree increased as the index increased.

Download Full Size | PDF

Third, three series of simulated blurred images in different sizes of windows were constructed by selecting different-sized windows on the simulated out-of-focus blurred Lena images, as follows: 64 × 64 small window, 128 × 128 median window, and 256 × 256 large window, as shown in Figs. 12 and 13. Each window contains both textured and smooth areas.

Fig. 12 Different-sized windows on Lena image.

Download Full Size | PDF

Fig. 13 Examples of blurred images in different-sized windows. Blurred images were best focused at index 8, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

Finally, three series of simulated blurred images with different noise were constructed by adding three types of noise to the simulated out-of-focus blurred Lena images: photon noise, thermal noise, and photon plus thermal noise. Photon noise is caused by photons and is usually represented by a Poisson distribution. Thermal noise is caused by electronic devices and follows a Gaussian distribution. The proposed Poisson–Gaussian model has the form [33]

I_{N} = I + n_{P} + n_{G},

where I_N represents the image with noise, I represents the original image, and n_P represents the Poisson distribution

\frac{1}{a} (I + n_{P}) ~ P (\frac{1}{a} I),

where a is the parameter of the Poisson distribution. The Gaussian distribution n_G is given by

n_{G} ~ N (0, b),

where b is the standard deviation of the noise.

Setting a = 0.064, b = 0; a = 0, b = 0.032; and a = 0.064, b = 0.032, three series of blurred images with noise were obtained, as shown in Fig. 14.

Fig. 14 Examples of blurred images with different types of noise. Blurred images were best focused at index 8, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

5.2 Experiments

The experiments were conducted as follows.

First, the constructed Λs were applied to the simulated out-of-focus blurred images. For convenience in comparing the performances of the Λs, their values were normalized in the range of [-1,1]. For each Λ curve, normalization was performed in two steps. First, the curve was moved vertically along the y axis such that $Λ' (I_{1}) = 0$ ; then, Λ′ was normalized using Eq. (13).

n o r m Λ' (I_{i}) = \frac{Λ' (I_{i})}{\max (a b s (Λ' (I_{i})))},

where I denotes the blurred image, i denotes the index of the blurred image, and norm indicates normalization.

The experimental results are displayed in Fig. 15, where the x axis denotes the image index, and the y axis denotes the Λ value. About 98% of the constructed Λs reach the maximum value at the best-focused image and most of them decrease as the defocus increases, which supports our proof in the previous section.

Fig. 15 Experimental results for 196 random Λs applied to simulated out-of-focus blurred images. The blurred images were best focused at index 8, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

Second, the constructed Λs were applied to the real out-of-focus blurred images. The focus value was calculated for the entire image. For convenient comparison, the same normalization method was used. The experimental results are displayed in Fig. 16. About 99% of the constructed Λs reach the maximum value at the best-focused image and almost all of them decrease as the defocus increases, which confirms that the focus function Λ can be used in real applications.

Fig. 16 Experimental results for 196 constructed Λs applied to real out-of-focus blurred images. The images were best focused at index 10, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

From Fig. 15 and Fig. 16, we can see that about 2% of the constructed Λs do not work well as focus functions. This is because the proof was constructed using a model, whereas the real situation is far more complicated. First, when an image is convolved by an image enhancement filter, input image values outside the bounds of the image are used, which causes errors. Second, real objects are three-dimensional rather than two-dimensional, so the image plane is not an ideal plane, which also causes errors. Third, errors result from noise in the optical system. The experimental results indicated that the best focus function varies with the application. In fact, researchers have been comparing and choosing focus functions for specific applications for years, and to date, the best focus functions for different applications have always varied.

Third, the constructed focus functions Λ were applied to the motion-blurred images. As we know, focus functions are widely used in many applications; therefore, we proposed that Λ might also work on motion blur. For convenient comparison, the same normalization was used. The experimental results are displayed in Fig. 17. About 82% of the constructed Λs reach maximum values for the image without motion blurring and decrease as the motion blur increases, which confirms that the proposed focus function Λ works on not only out-of-focus blur, but also motion blur.

Fig. 17 Experimental results for 196 random Λs applied to motion-blurred images. The images were the least motion blurred at index 1, and the blurring degree increased as the index increased.

Download Full Size | PDF

Then, The constructed Λs were applied to the simulated out-of-focus blurred images in different-sized windows. For convenient comparison, the same normalization was used. The experimental results are displayed in Fig. 18. About 99.9% of the constructed Λs reach the maximum value at the best-focused image, and most of them decrease as the defocus increases, which confirms that the focus function Λ can be used when window is applied.

Fig. 18 Experimental results for 196 constructed Λs applied to simulated out-of-focus blurred images in different-sized windows. The images were best focused at index 8, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

From Figs. 15 and 18, we can see that the Λs perform better for larger images or windows. That might be the result of the scale-space effect [34]. When an image is convolved by an image enhancement filter, input image values outside the bounds of the image are used, which causes errors, and for a fixed-size PSF, the smaller the image is, the larger the errors are.

Finally, the constructed Λs were applied to the simulated out-of-focus blurred images with different noise. For convenient comparison, the same normalization was used. The experimental results are displayed in Fig. 19. About 98% of the constructed Λs reach the maximum value at the best-focused image, and most of them decrease as the defocus increases, which confirms that the focus function Λ is robust to Poisson and Gaussian noise, such as photon noise and thermal noise.

Fig. 19 Experimental results for 196 constructed Λs applied to simulated out-of-focus blurred images with different types of noise. The images were best focused at index 8, and the defocus increased as the index increased or decreased.

Download Full Size | PDF

6. Conclusions

The known focus functions are summarized to create the flexible focus function Λ, which consists of a convex function and an image enhancement filter. Λ is rather flexible because for any convex function and any image enhancement filter, it is a focus function. A proof was also proposed using a model and Jensen’s inequality. It should be noted that the proof is built on a Gaussian or Gaussian-alike PSF assumption, which may be incompatible with some imaging systems. Nevertheless, the proposed focus function is still acceptable in wide variety of applications. A method of constructing random Λs was developed, and experimental results for both simulated and real blurred images proved its feasibility. The developed technology can be especially helpful in practical applications such as microscopy, holography, depth of focus estimation, deblurring, image refocusing, and iris recognition because it provides significant convenience in constructing focus functions. Future studies will address questions such as how to construct Λ for a specific application. Another question of interest is how to construct the convex functions by adjusting the parameters. Furthermore, studies on how the criteria for good focus functions can be improved would also be of remarkable value.

Appendix 1

The Dispersion Inequality

Statements

To prove that Λ is a focus function, we propose the Dispersion Inequality, which describes the inequality in the dispersion process.

First, we introduce two terms, the Dispersion Function and “disperse”:

The Dispersion Function: Let σ be an integrable function in R¹; then σ is a Dispersion Function if σ satisfies the following two formulas:

σ (x) \geq 0,

\int σ (x) d x = 1 .

disperse: Let f be an integrable function in R¹; then we say that f is disperse when f convolves by a Dispersion Function σ.

Then, the Dispersion Inequality is defined as follows.

The Dispersion Inequality

The Dispersion Inequality: Let φ be a convex function on [α,β], f be an integrable function in R¹, α ≤ f(x) ≤ β, x ∈ R¹, and σ be a Dispersion Function. Then the Dispersion Inequality is stated as

\int φ ((f * σ) (x)) d x \leq \int φ (f (x)) d x .

Here, R¹ is a one-dimensional vector space, and * is the convolution operator.

Proof

From the left part of Eq. (16), we get

\int φ ((f * σ) (x)) d x = \int φ (\int f (t) σ (x - t) d t) d x .

Then, we used Jensen’s inequality [35]. Jensen’s inequality states that if φ is a convex function on [α,β], f and p are integrable functions in R¹, α ≤ f(x) ≤ β, x ∈ R¹, and p satisfies the following two formulas:

p (x) \geq 0,

\int p (x) d x > 0 .

Then we have

φ (\frac{\int p (x) f (x) d x}{\int p (x) d x}) \leq \frac{\int p (x) φ (f (x)) d x}{\int p (x) d x} .

Obviously, σ is a subset of p. Replacing p with σ in Eq. (20), we get

φ (\int f (x) σ (x) d x) \leq \int σ (x) φ (f (x)) d x .

From Eq. (21), we have

\int φ (\int f (t) σ (x - t) d t) d x \leq \iint σ (x - t) φ (f (t)) d t d x .

From the right side of Eq. (22), we have

\begin{matrix} \iint σ (x - t) φ (f (t)) d t d x = \iint σ (x - t) φ (f (t)) d x d t \\ = \int φ (f (t)) \int σ (x - t) d x d t \\ = \int φ (f (t)) d t . \\ = \int φ (f (x)) d x \end{matrix}

From Eqs. (17), (22) and (23), we obtain Eq. (16). □