1. Introduction

Digital cameras based on CCDs or CMOS sensors are key devices in most of the current imaging techniques. However, light sensors with a pixelated structure are not always necessary. In fact, imaging methods requiring raster scanning techniques, such as confocal or two-photon microscopy, use in general a single high sensitivity photosensor. Some other imaging techniques can benefit from light detectors with no spatial structure. This is the case of imaging with low light levels or with electromagnetic radiation in spectral ranges out of the visible region, particularly in the near IR and terahertz spectral bands, where it is more difficult to make a pixelated sensor [1–3]. Imaging with single-pixel detectors can be useful also when the purpose is to measure the spatial distribution of several optical parameters of the light beam simultaneously, not only the intensity but also the polarization state [4], the phase [5,6], the color [7–10], or the spectral content [11–16]. In particular, single-pixel recording strategies are well adapted to provide depth information and have been applied with success in many different 3D imaging techniques. For example, in time-of flight techniques [17,18], photometric stereo methods [19], binocular stereo approaches [8,20], and virtual fringe projection techniques [16,21]. Single-pixel recording techniques also provide extra flexible trade-off among SNR, frame-rate and resolution of the imaging system [22,23]. Some diffuse optical imaging (DOI) techniques [24] benefit also from single-pixel recording.

Single-pixel imaging techniques [25,26] are based on illuminating the scene with a sequence of light structured patterns while the light reflected or transmitted by the objects is recorded by a single photosensor such as a photodiode or photomultiplier tube (PMT). Alternatively to this active imaging system, where a spatial light modulator (SLM) modifies the input light beam, in the passive configuration the light coming from the object is sampled sequentially by a set of masks codified in the SLM. The image is then computed numerically by using different mathematical algorithms such as a simple linear superposition, a change of basis transformation, or a correlation operation. Aside from the ordinary raster scanning techniques, a common approach is to use light patterns codifying functions of a basis, such as Hadamard or Fourier components [26,27]. In this case, the image is obtained computationally by just an inverse transformation of the corresponding basis. The technique is very well adapted to apply compressive sensing algorithms by using different basis of functions, highly incoherent between them, to sample the object and to reconstruct the image [28].

Ghost imaging methods are similar to active single-pixel techniques, particularly in the case of computational ghost imaging. In these approaches the object is scanned with random light patterns and the image is obtained by correlation operations [29,30]. Active single-pixel imaging techniques are also related to dual photography, where the image is obtained by exploiting the Helmholtz reciprocity principle [31]. Temporal modulation imaging is based on similar principles. In this case, images are obtained by using a single-pixel detector and scanning the object with temporally multiplexing spatial frequency functions from orthogonal, time varying spatial line modulation gratings [32].

As we have mentioned above, the capabilities of single-pixel imaging systems have been verified in many different applications [1–24] since the first introduction of the technique by Decker [25] and the significant improvement made by Duarte with programmable SLMs [26]. However, two main limitations constrain further development. One is the time required to scan light patterns onto the object. The second is the cost of the spatial light modulators required to project light patterns with high flexibility. The first problem is being tackled by reducing the number of light patterns, for example using CS or adapting sensing techniques [26,28,33]. Hardware improvements have also allowed reducing the acquisition time by means of balanced detection [34] or by multimode detection [35,36]. Recently, a way to reduce cost has been proposed based on the use of LED panels to generate light patterns. LED arrays have been widely used in different imaging applications such as 4D light field capture [37], phase contrast imaging [38], and Fourier ptychography [39,40]. Also, it has been proposed to use LED arrays for single-pixel imaging in fluorescence application, achieving a very high pattern modulation frequency (250 kHz) although at a low resolution of 8 × 8 pixels [41]. A remarkable single-pixel imaging system to obtain 2D monochromatic images, working at a very high frequency with a resolution of 32 × 32 pixels, was developed very recently [42].

In this paper we propose a method to perform 2D color imaging with a single photodiode by using light structured illumination generated with a low-cost LED array. Moreover, the system is extended to 3D imaging by simply adding a low number of photodiodes in different locations. The LED array is used to generate a sequence of Hadamard patterns which are projected onto the object by a simple optical system. A field programmable gate array (FPGA) controls the LCD panel allowing us to obtain high refresh rates (up to 10 kHz). Color is obtained by generating a Hadamard pattern for each RGB chromatic component by controlling the color of the LEDs, which allows us to use a single monochromatic photodiode. Experimental results are provided for an LED frequency operation of 10 kHz.

Information of the 3D structure of the object is obtained by combining a low number of images generated from photodiodes located at different positions. We take advantage from the fact that shifting a photodiode in single-pixel imaging is equivalent to shifting the light source in conventional imaging [19,31]. We use a photometric stereo technique based on shape-from-shading [43], which obtains information about the 3D surface from a series of images of the object acquired while varying the illumination direction in the scene. There are two different methodologies to tackle the problem, called uncalibrated and calibrated photometric stereo. In this paper we use an uncalibrated method [44], which does not require to know the positions of the light sources in advance, allowing us to develop a very flexible 3D imaging system.

2. Single-pixel 3D imaging by using a color LED display

The layout of our single-pixel imaging system based on a color LED display is shown in Fig. 1. An LED panel controlled by an FPGA is used to generate a sequence of light patterns codifying the Hadamard functions. The FPGA also stores the binary patterns to be sequentially sent. The light patterns are projected onto the object by using a conventional optical system. The light reflected by the object is collected by a single photodiode and digitized by a DAQ system attached to a computer. To obtain a color image, a sequence of Hadamard patterns is projected for each RGB chromatic component. The image is generated in the computer by a simple Hadamard inverse transformation for each chromatic component.

 

Fig. 1 Single-pixel imaging scheme based on a color LED display. A sequence of Hadamard patterns are displayed on the LED panel and projected onto the object by the projection lens. The light is collected by a photodiode (PD) and digitized by a DAQ system controlled with a computer. For 3D imaging, the photodiode is shifted to different positions, as is shown in gray color in the figure.

Download Full Size | PDF

Let us consider an LED panel containing a total of N light emitters. In mathematical terms, the image of the object provided by photodiode k is considered as a column vector, O_k, containing the N components of the object, in lexicographical order, which will be sampled by the light patterns. The light intensity measured by the photodiode can be written as:

For the case of 3D imaging, a low number K of photodiodes is located at different positions around the object, as is shown in Fig. 1. Alternatively, a single photodiode is shifted to different positions and the sequence of patterns is sent again. As is well known, shifting the photodiode in single-pixel imaging techniques is equivalent to illuminate the object from a different direction in conventional imaging. This property has been used in dual photography [31] and in single-pixel 3D imaging by applying shape from shading algorithms [19]. In our system, with this approach we are able to obtain a set of different images to apply photometric stereo techniques without preliminary calibration.

Photometric stereo aims at inferring the normal vectors $S=\left[S_1,\dots ,S_N\right]\in {ℝ}^{3\times N}$ that define the surface of the object starting from K input images obtained with a conventional sensor in a specific position and illuminated by a light point source located at a distant position characterized by the direction vector $L_K\in {ℝ}^3$. In our case, the LED projector is at a fixed position and the photodiodes are moved to K different positions to obtain the different images. If we assume that surfaces in the scene have a Lambertian behavior, the photodiode has a linear response, and assuming an orthographic projection, the object provided by our single-pixel camera in Fig. 1 from photodiode k, can be written as $O_k=S^T·L_k$. Therefore, the image obtained for all the photodiodes is given as the result of the following matrix operation:

In principle, to solve the linear system in Eq. (2) for S, we need to know the photodiode directions. However, this is not always possible or convenient. Instead, we apply an uncalibrated method, which does not require to know the positions of the photodiodes in advance [44]. Firstly, an initial pair of surface normals, $\widehat{S}$, and photodiode directions, $\widehat{L}$, are estimated by minimizing a least square cost function and using a singular value decomposition (SVD) method [45]. The SVD method allows vectors S and L to be obtained from the initial estimation up to a linear transform ambiguity characterized by a matrix G, where G∈GL(3), in the following way [46]

3. Experimental results

Figure 2 shows a zenithal picture of the LED-based single-pixel camera configured by following the layout in Fig. 1. We use a 32 × 32 RGB LED matrix panel, with 4mm grid spacing (Adafruit). A simple lens, with a focal length f = 100 mm, located at a distance 35 cm far from the matrix, is used to project the patterns onto the object. The sensor is an amplified photodiode (Thorlabs PDA 100A-EC) and the FPGA module is an FPGA development KIT (Terasic, DE1-SoC board) with the Quartus Prime software used to codify the light patterns.

 

Fig. 2 Photograph of the optical system for single-pixel 3D imaging with a color LED display. The LED matrix and the FPGA are on the left. The photodiode and the object to be imaged are on the right end.

Download Full Size | PDF

The resolution of the Hadamard patterns is limited by that of the LED array. Therefore, the images reconstructed by our system have a resolution limited to 32 × 32 pixels. For the optical configuration in Fig. 2, the size of the scene illuminated by the light patterns is 4 × 4 cm. The pattern modulation frequency is limited by the maximum clock frequency of the LED array, which is 25MHz. Our 32 × 32 LED matrix is electronically composed by two matrices with 16 × 32 LEDs that can operate simultaneously. Then the maximum modulation frequency for patterns with 32 × 32 pixels is around 40 kHz if we take into account the multiplexing time and the control signals. However, the quality of the images in terms of the signal to noise ratio (SNR) was not satisfactory for frequencies larger than 10 kHz. Without applying CS, or similar optimization algorithms, we need 32 × 32 Hadamard patterns to obtain an image. Thus, at a pattern projection rate of 10 kHz, the image frame rate is almost 10 Hz.

We have performed two imaging experiments at a sampling rate of 10 kHz with the setup in Fig. 2. In the first one, we just record color images of the input scene. To this end, we send the full set of 1024 Walsh-Hadamard patterns associated to a space with 32 × 32 pixels, for each primary color, red, green and blue. The color image is reconstructed from the coefficients provided by the photodiode intensity signal by using Eq. (2). The result of this experiment is shown in Fig. 3. To quantify the quality of the image we measured the SNR for each chromatic channel RGB. The result of the averaged SNR for the three measurements is 53 dB. We measured also the SNR of the 2D monochromatic images used in the 3D imaging experiment shown in Fig. 4. As an example, the SNR of the central image in Fig. 4(a) is 62 dB. The SNR increases by using all LED colors at the same time to codify the light patterns.

 

Fig. 3 2D image of a color object obtained with the optical system shown in Fig. 2. The resolution is 32 × 32 pixels, the same as the number of elements in the LED array.

Download Full Size | PDF

 

Fig. 4 (a) 2D images of the object in Fig. 3 obtained with the single-pixel camera shown in Fig. 2 for different positions of the photodiode. The Photodiode was moved from left to right at arbitrary positions in the same horizontal plane. (b) 3D surface of the object, codified in color, obtained by the photometric stereo algorithm from the pictures in (a). (c) 3D surface of the object.

Download Full Size | PDF

We can compare the resolution, modulation frequency, and cost of our single-pixel camera based on LEDs with those of light projection systems based on spatial light modulators. The resolution of our system is much lower than that provided by off-the-shelf video projectors based on digital micromirror device (DMD) or liquid crystal on silicon (LCOS) spatial light modulators, which is typically 1280 × 1024 pixels. However, the modulation frequency of these video projectors is limited to 60 Hz, which is two orders of magnitude lower than the frequency of the LED system with a similar cost. Some commercial video projectors allow to operate at higher frequencies by modulating binary patterns instead of three grey-level chromatic channels. In this case the modulation frequency can arrive to 1200 Hz but at a higher cost. Finally, scientific spatial light modulators based on DMDs, when controlled with high quality drivers, can arrive to frequencies as high as 22 kHz with high resolution and flexibility. However, with a cost higher in one order of magnitude.

In our second experiment we obtain a 3D reconstruction of the surface of the object. As explained in the previous section, firstly we record a set of images of the object with our single-pixel camera in Fig. 2, by shifting the photodiode to different positions. In particular, we record 5 images by moving the photodiode in the same horizontal plane (see Fig. 1). This ensures us that we had enough LDRM points to obtain a robust GBR parameter estimation. The images are shown in Fig. 4(a). It is possible to see how the object appears illuminated from different directions. These images are used as inputs of the uncalibrated photometric stereo algorithm in [44]. The software code to apply this method is available in [48]. The resulting 3D shape is shown in Fig. 4(b) and 4(c). The axis labels in Fig. 4(b) show the x, y, and z coordinates of the object surface in mm. Please note that the surrounding of the figure looks uniform blue because the shape-from-shading algorithm do not provide information for low values of the intensity. Figure 4(c) shows just a 3D representation of the surface.

To estimate the quality of our result, we have measured relative depths at different locations in the object with a Vernier caliper appreciating 0.05 mm. The object has a size of 32.5 × 37.2 × 27.1 mm. The relative error of our surface, averaged from measurements in three locations of the object, is 3.1 ± 0.2%. Also, we have estimated the error of an uncalibrated shape-from-shading method in the bibliography performed with a conventional camera. In the experiment by Hayakawa [46], the average error is estimated to be 2.2%, which is of the same order of magnitude of our result. Finally, we have compared the result of our single-pixel uncalibrated 3D imaging method with the single-pixel calibrated one described in [19]. The root mean square error (RMSE) of our measurements compared to those performed with the caliper is 0,17 mm for an object with a total depth of 27.1 mm, while the RMSE of the measurements performed with the system in [19] compared with those provided by a stereo photogrammetric camera system is 4 mm for an object with a total depth of 250 mm. The RMSE error is lower in our case, but it should be taken into account that we measured the error only from a few measurements, while in [19] the comparison was performed for all the surface. Besides, in our case the total depth is much lower. The relative error in terms of the total depth is of the same order of magnitude in both cases.

4. Conclusions

We have proposed a method to perform 2D color imaging and 3D imaging with a single photodiode by using light structured illumination generated with an LED array. Our system allows us to record low-resolution images of color objects with a resolution limited to 32 × 32 pixels, the number of LEDs of the array. Despite the maximum modulation frequency of the LED panel is 40 kHz, we need to operate at a frequency of 10 kHz to obtain a satisfactory SNR. Therefore, the image frame rate is almost 10 Hz. With this frequency we obtained 2D images with SNR equal to 53 dB for color images and 62 dB for monochromatic ones.

The imaging system has a cost similar to off-the shelf video projectors based on DMD or LCOS spatial light modulators, which have a higher resolution but are limited to operate at lower frequencies in single-pixel imaging applications. Spatial light modulators based on DMDs and controlled with high quality drivers provide both larger resolutions and modulation frequencies but at a higher cost. The resolution of our system is low because of the limited size and pitch of the LED array. It could be possible to improve the resolution by combining several LED panels operating in parallel at the same frequency. In our case, however, the characteristics of our FPGA board would limit the total number of panels to three.

For 3D imaging we have used a non-calibrated photometric stereo technique avoiding a precise calibration of the position of the photodiodes. The accuracy of the depth profile was determined by direct comparison with measurements using a Vernier caliper. We estimated a depth RMSE of 0,17 mm and an averaged relative error of 3.1%.

We have used the Walsh-Hadamard basis for the sampling operation, but other different basis such as Fourier or Morlet wavelet could also be used. In this prove of concept experiment we did not performed CS but we plan to do it in the next applications of the camera. By using CS and optimizing the system to obtain higher frequency rates, we expect to get color 2D images at a frame rate of 40 fps.