## Abstract

Fourier single-pixel imaging (FSI) is an efficient single-pixel imaging method of obtaining high-quality (resolution/signal-to-noise ratio) 2D images, which projects sinusoid patterns on the object and reconstructs the image through reflected light. The typical system of FSI consists of a single-pixel detector and a digital projector. However, the defocusing of the projector lens blurs the projected patterns, which results in reduced imaging quality. In this work, we propose the projector-defocusing rectification for FSI, which optimizes projector defocusing for the first time. The proposed method rectifies Fourier coefficients using the amplitude ratio between original and defocused patterns, which we can acquire through a controlled experiment on a whiteboard. The enhancement of imaging quality in imperfect circumstances is demonstrated by simulations and experiments.

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

Traditional photographic techniques mainly adopt charge-coupled devices or complementary metal oxide semiconductors, both of which utilize multi-pixel arrangement to obtain images. By contrast, the novel single-pixel imaging (SPI) only uses single-pixel detector, which is unrestricted by special resolution to reconstruct images. Therefore, SPI has been widely used in 3D [1–3], terahertz [4, 5], and multispectral [6] imaging areas.

SPI was first introduced in the “flying-spot” camera [7]; it used a perforated disk (a Nipkow disk) to modulate a light source for scanning light spots, but the result image exhibited a low signal-to-noise ratio (SNR). SPI has recently been developed into conventional ghost imaging [8] on the basis of probabilistic mathematics but with low reliability. Indirect measurement, such as computational imaging [9], is later on used in the method, which means the detection unit does not need a direct line of sight to the object. For instance, Zhang et al. [10] proposed Fourier single-pixel imaging (FSI) to reconstruct images with indirect light. In their work, sinusoid structured patterns were illuminated on the object by a digital projector, and the reflected light was collected by a single-pixel detector to reconstruct the image through the Fourier spectrum. This method can achieve high-quality images than those of other methods. However, many factors causing inaccuracy remain in the FSI scheme. For instance, the projector defocusing will decrease the image quality in pattern projection, which is an important step of FSI.

In the pattern projection of FSI, the defocusing caused by the projector lens results in differences between original and defocused patterns. Fourier coefficients calculated by reflected lights are inaccurate and cannot be used to reconstruct the original image. Li et al. [11] used binary pattern optimization, which needed to reach maximum iteration and did not apply to SPI, to solve the defocusing problem. Sun et al. [12] adopted digital microscanning to improve the SNR of SPI. In [13–16], defocusing was corrected by using images with two different depths, which was inapplicable to SPI due to its low efficiency. Therefore, the projector defocusing resulting in stripe changes remains unsolved yet.

In our previous work [17], we build up conventional SPI system and present adaptive regional single-pixel imaging (ARSI) method to improve imaging efficiency. In this study, we propose a projector-defocusing rectification (PDR) method to rectify projector defocusing in the frequency domain, which can reconstruct high-quality images. PDR is based on the conventional FSI method and uses Fourier spectrums to reconstruct images. We first build a model of projector defocusing according to defocusing theory and its effect on four-step phase-shifting sinusoidal patterns to solve the defocusing problem. In this model, defocusing only changes the amplitude of fringes for particular sinusoid patterns [18]; thus, the amplitude ratio between original and defocused patterns is definite at a specified frequency. We then design a new algorithm to rectify projector defocusing on the basis of the defocusing model and the FSI method. We prove in our algorithm that Fourier coefficients are inversely proportional to amplitude as other conditions are fixed. Thus, the Fourier spectrum can be rectified by amplitude ratio and then reconstruct high-quality images. This PDR method innovatively introduces the method of correcting defocusing and improves image quality effectively.

## 2. Principles

In conventional FSI, sinusoidal patterns are projected onto the object, and reflected lights are collected by a single-pixel detector. The image is then reconstructed by a computer through the inverse Fourier transform (IFT). Projector defocusing becomes a factor causing inaccuracy in this imaging process. The defocusing causes patterns to be projected on the object blurry, which means the interaction between the object and patterns cannot precisely represent the Fourier spectrum of the object. Therefore, we propose a novel SPI method based on defocusing rectification to improve image quality.

In this section, we first briefly introduce the conventional FSI method and its key formulas. A module of projector defocusing is then built according to defocusing theory and its effect on sinusoidal patterns. We finally design a rectification algorithm for projector defocusing on the basis of the defocusing model and the FSI method.

#### 2.1 Fourier single-pixel imaging

In the FSI, phase-shifting sinusoidal structured light patterns are projected on the scene, and reflected light is collected by a single-pixel detector [10]. The final image is reconstructed with the fast IFT algorithm.

The schematic of the FSI is shown in Fig. 1.

In [10], a 2D sinusoid pattern is determined by frequency (*f _{x}*,

*f*) and initial phase

_{y}*Ф*.

*x*,

*y*) represents the 2D Cartesian coordinates in the scene,

*B*(

*f*,

_{x}*f*) denotes the DC term equal to the average image intensity at frequency (

_{y}*f*,

_{x}*f*), and

_{y}*A*(

*f*,

_{x}*f*) is the amplitude of the pattern at frequency (

_{y}*f*,

_{x}*f*).

_{y}The total response of the single-pixel detector is

*k*depends on the size and location of the detector, and

*R*is the distribution of the surface reflectivity of the imaged objects in the scene.

After projecting patterns, i.e.,${P}_{0},{P}_{\pi /2},{P}_{\pi},{P}_{3\pi /2}$, we obtain Fourier coefficients in the form of

*j*denotes the imaginary unit, and F is the Fourier transform.

In this process, the defocusing of projector affects the accuracy of patterns projected on the object, which results in the actual pattern *P ^{’}*(

*x*,

*y*;

*f*,

_{x}*f*) and an inaccurate Fourier spectrum

_{y}*F*{

^{’}*R*(

*x*,

*y*)}. We build a defocusing model to establish the relationship between

*F*{

^{’}*R*(

*x*,

*y*)} and

*F*{

*R*(

*x*,

*y*)} to solve the issue.

#### 2.2 Model of projector defocusing

The defocusing comes from the projector lens, as shown in Fig. 2. When the object is not placed on the focal plane of projector, the projected pattern is defocused illustrated as [Fig. 2(a)]. We establish the relationship between original and defocused patterns by defocusing theory and its effect on sinusoidal patterns to correctly rectify defocusing.

### 2.2.1 Defocusing theory

Given (*f _{x}*,

*f*), the defocusing effect of projector can be expressed as

_{y}*P*(

^{’}*x*,

*y*) is the generated sinusoidal fringe pattern,

*P*(

*x*,

*y*) is the original pattern,

*G*(

*x*,

*y*) is the point spread function (PSF) [Fig. 2(b)], and ⊗ represents convolution.

### 2.2.2 Defocusing effect on Fourier single-pixel imaging

According to [18], the pattern *P*(*x*,*y*) is sinusoid; thus, defocusing *G*(*x*,*y*) only reduces the amplitude of *P*(*x*,*y*). The amplitude varies significantly with the sinusoid frequency, i.e.,

*A*(

*f*,

_{x}*f*) represents the amplitude of the original pattern at frequency (

_{y}*f*,

_{x}*f*),

_{y}*A*

^{’}(

*f*,

_{x}*f*) is the amplitude of the defocused pattern at frequency (

_{y}*f*,

_{x}*f*), and

_{y}*α*(

*f*,

_{x}*f*) denotes the amplitude ratio between the original and defocused patterns at frequency (

_{y}*f*,

_{x}*f*).

_{y}The average gray value of patterns does not change, which means

*B*(

*f*,

_{x}*f*) represents the average gray value of the original and defocused patterns at frequency (

_{y}*f*,

_{x}*f*).

_{y}The amplitude changing due to defocusing is shown in Fig. 3. The amplitude of the defocused pattern [Fig. 3(b)] is lower than that of the original pattern [Fig. 3(a)]. Curves of the central row extracted from the two patterns are compared in Fig. 3(c).

The response of single-pixel detector considering that the projector exerts the defocusing effect is

The inaccurate Fourier coefficient is

Equations (3) and (9) are combined, and we obtain

We obtain a defocusing model based on sinusoidal fringes by using the above principles. The amplitude is the only affected factor of defocusing. Hence, once we possess the amplitude ratio *α*(*f _{x}*,

*f*), we can rectify the defocusing and attain accurate Fourier coefficients.

_{y}#### 2.3 Rectification method for projector defocusing

We propose a PDR method on the basis of the conventional FSI method and the projector-defocusing model. We prove that the effect of defocusing on Fourier coefficients can be represented by the amplitude ratio between original and defocused patterns. Thus, we rectify the defocusing by taking Fourier coefficients multiplied by amplitude ratio as Eq. (10).

### 2.3.1 Amplitude ratio

From Eq. (5), the amplitude ratio can be calculated as

The values of *A*(*f _{x}*,

*f*) and

_{y}*B*(

*f*,

_{x}*f*) of

_{y}*P*(

*f*,

_{x}*f*) are set manually when generating image patterns. The amplitude

_{y}*A*

^{’}(

*f*,

_{x}*f*) of the defocused pattern

_{y}*P*

^{’}(

*f*,

_{x}*f*) is therefore required to compute the amplitude ratio

_{y}*α*(

*f*,

_{x}*f*).

_{y}We adopt a pixelated camera to capture the protected pattern on a whiteboard as *P _{c}*(

*f*,

_{x}*f*). Considering the influence of light propagation between the projector and the camera, we conclude that

_{y}*r*denotes the light propagation coefficient between the projector and the camera, including the whiteboard reflectivity.

Thus, for the captured pattern *P _{c}*(

*f*,

_{x}*f*), the amplitude and average value are

_{y}From Eqs. (13) and (14),

To eliminate the influence of *r*,

The amplitude and average value of the captured pattern *P _{c}*(

*f*,

_{x}*f*) can be calculated according to the features of the four-step phase-shift images as

_{y}*P*(

_{c}*f*,

_{x}*f*) is M × N.

_{y}### 2.3.2 Rectified Fourier coefficients with amplitude ratio

To acquire the correct Fourier coefficient, we rectify it as

We can then reconstruct an image of higher quality in the fast IFT algorithm.

### 2.3.3 Simulation

We simulate with an actual picture [Fig. 4], to validate the correctness of the rectification method.

The defocusing function is set to a Gaussian filter (3 × 3 pixels with a standard deviation of 0.6 pixels) in this simulation. The pattern size is 200 × 200 pixels, the average is 100 pixels, and the amplitude is set to 100 pixels as standard.

As to our rectification method, we first need to acquire the amplitude ratio of original and defocused patterns. As shown in Fig. 5, the ratio decreases with the frequency, which corresponds to the phenomenon that patterns with a high frequency have narrow fringes and are easily influenced by defocusing.

We then rectify the Fourier coefficients with the amplitude ratio and reconstruct the image with the inverse fast Fourier transform. The contrast among the original image, conventional FSI image without PDR, and FSI image with PDR is shown in Fig. 6. The image shown as Fig. 6(a) becomes blurred in the conventional FSI shown as Fig. 6(b), whereas it is almost the same as the original one in the FSI with PDR shown as Fig. 6(c).

We quantity these images’ quality by correlation coefficient *r* as shown in Eq. (20). This correlation coefficient increases with the resemblance of two images and is defined as:

*r*is the correlation of the two images;

*A*and

*B*are the image matrix with indexes

*m*and

*n*,

*‾A,‾B*denotes the mean value of matrix

*A*and

*B*.

The correlation coefficients of conventional FSI image and FSI image with PDR are 0.9867 and 0.9979 respectively shown as Fig. 6. Through comparing the correlations between conventional FSI and FSI with PDR, the image quality of FSI with PDR is evidently much higher than FSI without PDR.

We further compare the details of the images by zooming in on a patch. Figure 7 shows that the patch in the FSI image has a low sharpness. With PDR, the patch is almost the same as the original one. We select the middle row of the patch to compare. The red curve represents the original patch, the green curve represents the FSI patch, and the blue curve represents the patch of the FSI with PDR. The gradient apparently decreases in the FSI image, whereas the FSI with PDR perfectly reduces the image.

Taking noise into consideration, we conduct the simulation with black and white stripes as shown in Fig. 8. Compared to FSI stirpes, stripes of FSI with PDR has higher similarity with the original stripes. Because of noise, there are grey noise points in the background in Figs. 8(b) and 8(c). And PDR at present cannot deal with these noises, therefore bilateral filtering [19] is used to solve this problem in the experiments.

## 3. Experiments

Our goal in the experiment is to demonstrate the feasibility and efficiency of our proposed rectification method. For the setup [Fig. 9], a digital projector is selected as the spatial light modulator (SLM) to produce high-contrast structured patterns. The used digital projector contains a light source with a wavelength of 455 nm. A pattern that illuminates the scene is projected onto the scene by the digital projector every 0.1 s. In step 1, the reflected light is collected by a camera. In step 2, the reflected light is collected by a lens and detected by a single-pixel detector, which is actually a photodiode. The 455 nm bandpass filter is fixed in front of the single-pixel detector. The collected analog signal is converted into a digital signal by an analog-to-digital converter (ADC) and processed using a computer. The object is placed 0.5 m away from the lens of the digital projector. The defocusing coefficient is determined by the projector parameters. The pattern size is determined by the matching relationship between the projector and the camera and the size of result image. The average value of pattern is 100 pixels, and the amplitude is set to 100 pixels as standard. The experiment is conducted in black environment to eliminate the influence of environmental illumination.

We need to build the defocusing model by calibrating the amplitude ratio between original and defocused patterns in our method. A whiteboard is used as the control group in the experiment to improve reflectivity. Four-step phase-shifting sinusoidal patterns at all frequencies are projected on the whiteboard. The reflected patterns are captured by a camera. As for a specified frequency, the original pattern amplitude is set in the projector, and the defocused pattern amplitude can be calculated by the reflected pattern. The fundamental frequency is first measured on the basis of image size and symmetry of Fourier coefficients, and half of the coefficients are then measured. Image accuracy is affected by the uneven reflectivity of the printed paper surface, and the quantization noise exerts a certain influence on the correction of high-frequency information. Thus, we operate bilateral filter on the result of FSI with PDR.

As we adopt four-step phase-shifting sinusoidal structured light patterns, it needs four patterns to acquire one Fourier coefficient. And due to the symmetrical characteristic of Fourier coefficients, we only need to acquire half number of Fourier coefficients. Considering of the control experiment on the white board, it needs to project same number of patterns. But this control experiment only need to be done one time for one fixed experiment setup. In the first experiment, we project 16000 patterns and the whole illumination process will last around 4 hours. There are 64000 projected patterns in the second experiment and it takes around 16 hours.

We use three E letters in different sizes on a black background as the target object to quantitatively compare the imaging results of FSI and FSI with PDR as shown in Fig. 10. The widths of white bars are 2, 1.6, and 0.6 cm. The two reconstructed images have a solution of 80 × 50.

We use root mean square (RMS) contrast to compare image quality numerically [20]. This algorithm does not depend on the angular frequency content or the spatial distribution of contrast in the image and is defined as the standard deviation of the pixel intensities.

*C*is the RMS contrast of the image;

*M*and

*N*are the width and height of the image (in pixel), respectively; intensities

*I*are the

_{ij}*i*-th

*j*-th element of the two-dimensional image of size

*M*by

*N*.

*‾ I*is the average intensity of all pixel values in the image.

The RMS contrast of the FSI image is 22.6905, as calculated by the above algorithm. The RMS contrast of the PDR image is 27.1196, which is improved nearly 20 percent.

We select a normal printed picture as the object. After completing the control experiment, we formally image the object by FSI and FSI with PDR. We conduct an image-processing method on the FSI image to rectify the defocusing problem to prove the superiority of PDR. We select one of the best image restoration methods, Wiener filter, by comparing a series of methods and set the SPF manually to achieve the best reduction result. The contrast is shown in Fig. 11.

Intuitive perception of the two images shows that the details in FSI with PDR are more obvious, and the image is clearer. The calculation by the above algorithm presents that the RMS contrast of the original reference image shoot by a digital camera is 26.0981. The RMS contrast of the FSI image is 19.5656, the RMS contrast of the Wiener image is 27.3186, and the RMS contrast of the PDR image is 28.1790. The PDR achieves a better result. Similar to Wiener filter, for most image restoration methods, we need to determine the PSF of the defocusing, which means they are impractical in this circumstance. PDR can improve high-frequency information, but it does bring some noise at the same time, which results in several dark dots at the background.

The aforementioned experiments illustrate that the image quality is greatly improved with PDR compared with that using the conventional FSI method. The proposed PDR can be applied to any situation in which the defocusing is insufficient to lose all high-frequency information. The FSI accuracy can be significantly improved by rectifying projector defocusing.

## 4. Conclusions

We conduct FSI with PDR by adding a control experiment to acquire defocusing coefficients and optimize image quality of conventional FSI. The proposed PDR method achieves a high level of reconstructed image quality compared with that of the conventional SPI method.

The core of the proposed PDR is to calculate the influence of projector defocusing and rectify it via our newly proposed algorithm. In practice, we project patterns on a whiteboard and calculate amplitude ratio by reflected light, which can derive defocusing coefficients and make it work in the reconstruction of target object.

PDR is especially applied to conventional FSI. The influence of defocusing is significant in a high-frequency region; hence, the PDR method can reconstruct high-quality images with considerable details and specifics.

However, at high frequency, PDR cannot distinguish noises from effective signals and result in enhance of noises. How to eliminate high frequency noises is one direction of our future research although the bilateral filtering can reduce the image noises to some extent. . Besides, the objects for FSI in the experiments were 2D pictures and were placed at the constant focus distance in the experiments. When the scene to be imaged is 3D and complex, the PSF of projector will vary at different distances and PDR cannot work well. Thus, applying the PDR method to 3D scene is a challenging work, which is another direction of our future research.

FSI with PDR is experimentally capable of producing high-quality images of a 2D object compared with those of the conventional FSI method. The proposed method can be widely used in areas where projector defocusing needs to be considered and rectified to improve imaging accuracy. Moreover, PDR introduces a way to solve defocusing on the basis of Fourier spectrum, which can be applied to other frequency-domain methods.

## Funding

National Natural Science Foundation of China (61475013, 61735003 and 61227806), Program for Changjiang Scholars and Innovative Research Team in University (IRT_16R02).

## References and links

**1. **B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, “3-D Computational imaging with single-pixel detectors,” Science **340**(6134), 844–847 (2013). [CrossRef] [PubMed]

**2. **Z. Zhang and J. Zhong, “Three-dimensional single-pixel imaging with far fewer measurements than effective image pixels,” Opt. Lett. **41**(11), 2497–2500 (2016). [CrossRef] [PubMed]

**3. **M. J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel three-dimensional imaging with time-based depth resolution,” Nat. Commun. **7**, 12010 (2016). [CrossRef] [PubMed]

**4. **W. Chan, K. Charan, D. Takhar, K. Kelly, R. Baraniuk, and D. Mittleman, “A single-pixel terahertz imaging system based on compressed sensing,” Appl. Phys. Lett. **93**(12), 121105 (2008). [CrossRef]

**5. **C. Watts, D. Shrekenhamer, J. Montoya, G. Lipworth, J. Hunt, T. Sleasman, S. Krishna, D. Smith, and W. Padilla, “Terahertz compressive imaging with metamaterial spatial light modulators,” Nat. Photonics **8**(8), 605–609 (2014). [CrossRef]

**6. **L. Bian, J. Suo, G. Situ, Z. Li, J. Fan, F. Chen, and Q. Dai, “Multispectral imaging using a single bucket detector,” Sci. Rep. **6**(1), 24752 (2016). [CrossRef] [PubMed]

**7. **P. Sen, B. Chen, G. Garg, S. Marschner, M. Horowitz, M. Levoy, and H. Lensch, “Dual photography,” ACM Trans. Graph. **24**(3), 745–755 (2005). [CrossRef]

**8. **T. B. Pittman, Y. H. Shih, D. V. Strekalov, and A. V. Sergienko, “Optical imaging by means of two-photon quantum entanglement,” Phys. Rev. A **52**(5), R3429–R3432 (1995). [CrossRef] [PubMed]

**9. **J. H. Shapiro, “Computational ghost imaging,” Phys. Rev. A **78**(6), 061802 (2008). [CrossRef]

**10. **Z. Zhang, X. Ma, and J. Zhong, “Single-pixel imaging by means of Fourier spectrum acquisition,” Nat. Commun. **6**(1), 6225 (2015). [CrossRef] [PubMed]

**11. **X. Li and Z. Zhang, “High-quality fringe pattern generation based on binary pattern optimization with projector defocusing,” J. Opt. Technol. **84**(1), 32–40 (2017). [CrossRef]

**12. **M. J. Sun, M. P. Edgar, D. B. Phillips, G. M. Gibson, and M. J. Padgett, “Improving the signal-to-noise ratio of single-pixel imaging using digital microscanning,” Opt. Express **24**(10), 10476–10485 (2016). [CrossRef] [PubMed]

**13. **K. Fraser, D. V. Arnold, and G. Dellaire, “Projected Barzilai-Borwein Method with Infeasible Iterates for Nonnegative Least-Squares Image Deblurring,” in Proceedings of IEEE Conference on Computer and Robot Vision (IEEE, 2014), pp. 504–509. [CrossRef]

**14. **A. R. Barakat, “Dilute aperture diffraction imagery and object reconstruction,” Opt. Eng. **29**(2), 131–139 (1990). [CrossRef]

**15. **C. W. Helstrom, “Image Restoration by the Method of Least Squares,” J. Opt. Soc. Am. **57**(3), 297–303 (1967). [CrossRef]

**16. **M. E. Daube-Witherspoon and G. Muehllehner, “An Iterative Image Space Reconstruction Algorthm Suitable for Volume ECT,” IEEE Trans. Med. Imaging **5**(2), 61–66 (1986). [CrossRef] [PubMed]

**17. **H. Jiang, S. Zhu, H. Zhao, B. Xu, and X. Li, “Adaptive regional single-pixel imaging based on the Fourier slice theorem,” Opt. Express **25**(13), 15118–15130 (2017). [CrossRef] [PubMed]

**18. **M. Gupta, Y. Tian, S. G. Narasimhan, and L. Zhang, “A Combined Theory of Defocused Illumination and Global Light Transport,” Int. J. Comput. Vis. **98**(2), 146–167 (2012). [CrossRef]

**19. **S. Paris, P. Kornprobst, and J. Tumblin, “Bilateral Filtering,” Int. J. Numer. Methods Eng. **63**(13), 1911–1938 (2009).

**20. **E. Peli, “Contrast in complex images,” J. Opt. Soc. Am. A **7**(10), 2032–2040 (1990). [CrossRef] [PubMed]