1. Introduction

Since Dennis Gabor invented holography in 1947, electro-holography has been a promising technology for application in three-dimensional (3D) displays. Further, 3D television is arguably the most anticipated application of electro-holography; however, several issues have to be overcome before such applications become commercially viable. The main issues are (i) the enormous amounts of data that have to be transmitted and (ii) the extremely high computational power required for the generation of computer-generated holograms (CGHs), which constitute a digital medium to record 3D data for subsequent display. According to one estimation, the bandwidth that is required to transmit holography is 14 Tb/s (= 1.75 TB/s) [1], which is 14,000 times the bandwidth of the Gigabit Ethernet that is currently extensively used.

Based on the data flowing in the transmission path, the electro-holography transmission system can be classified as either a distributor-intensive system or a receiver-intensive system. Figure 1 schematically depicts both the aforementioned types of systems. A distributor-intensive system calculates and transmits a CGH (or the wave front information) from the distributor side, indicating that the computational load is far higher on the distributor side when compared to that on the receiver side. However, because the reconstruction conditions of the display device (e.g., the reconstruction position of the 3D images and the wavelength of the light source) are usually unknown while creating the CGH, a distributor-intensive system suffers from drawbacks such as the fact that it is compatible with only certain display devices and the additional computational load required for converting the CGH. In addition, the amount of data associated with a CGH is considerably large for transmitting via a network because extremely large CGHs are required for obtaining practical electro-holography (e.g., 58 gigapixels for 10 cm$\times 10$ cm display [1]) and because no effective compression scheme has yet been obtained for holography.

 

Fig. 1 Comparison of the two types of electro-holography transmission systems.

Download Full Size | PDF

In contrast, a receiver-intensive system transmits only 3D information (e.g., the coordinates of point light sources (PLSs)) and further results in the creation of CGHs on the receiver side. In this case, the calculation load is highest on the receiver side, and the reconstruction requirements of the display device can be satisfied without the requirement of any additional computation. In terms of the amount of data flowing through the transmission path, a receiver-intensive system is considered to be superior to a distributor-intensive system because the data associated with the 3D information are fewer and/or more compressible than those associated with the CGHs in majority of the practical situations. Therefore, we targeted a receiver-intensive system in this study.

Several studies have investigated CGH compression [1–7]. A popular approach for investigating CGH compression is to use an existing image or video compression scheme, such as high efficiency video coding (HEVC, H.265) [2, 3], advanced video coding (AVC, H.264) [3], or JPEG2000 [4, 5], for natural images. However, this leads to insufficient compression rates because of the different nature of the frequency domain and the motion compensation being less effective. A recent video compression scheme has reduced the amount of data in an image by eliminating the redundant data from both the spatial and temporal domains using spatial frequency conversion (e.g., discrete cosine transform) and motion compensation. However, those approaches are ineffective for holography because the information in the entire frequency domain is required for modulating the reference light and any tiny movement or alteration in a 3D image will change the pattern of the associated hologram in the whole area, i.e., the correlation between the frames is usually very low. Therefore, conventional video compression is not readily applicable to holograms.

Some studies have considered suitable approaches for distributor-intensive systems. Bernardo et al. proposed a compression scheme based on the fact that the complex amplitude distribution (CAD) around the target object between adjacent frames is highly correlated [8]; however, this approach is ineffective for 3D images having long depth. Meanwhile, Blinder et al. proposed a motion compensation scheme for a 3D model by modeling the relation between the global motion of the 3D model and the signal changes in the hologram [9], and Viswanathan et al. proposed a ray-based compression scheme based on the Gabor wavelet transformation [10]. Various studies have also investigated schemes to compress 3D information for electro-holography using receiver-intensive systems, such as the multiview encoding scheme based on depth-map coding as proposed by Senoh et al. [11].

Herein, we propose an electro-holography compression scheme with vector quantization (VQ) encoding for a PLS-based 3D model and a fast CGH calculation method based on lookup tables (LUTs). Instead of compressing a hologram, our scheme encodes PLS data with VQ on the distributor side and further creates a CGH using the encoded PLS data on the receiver side. Further, we have developed the VQ-LUT method that calculates the CGH directly from the encoded PLS data to reduce the computational load associated with the creation of CGH on the receiver side. Some studies have applied VQ to compress the hologram [6, 7]; however, this approach quantizes and compresses the wavefront information about a hologram or related data (e.g., differences of intensities between the phase-shifted holograms), whereas our method compresses the PLSs with VQ. Therefore, the proposed method is different from other methods which apply VQ in the hologram compression.

The remainder of this study is organized as follows. CGHs are briefly described in Section 2, and the details of our system are presented in Section 3. Further, we evaluate the performance of our system in Section 4 and present our conclusions in Section 5.

2. Computer-generated holograms

A CGH is a digital medium that is used to record 3D data for displaying in electro-holography. In the present study, we have targeted a 3D model expressed by PLSs, which can be referred to as a PLS model. Further, the CGH calculation of the PLS model [12] is described by

Further, 3D images are reconstructed by modulating the incident light using a spatial light modulator that displays the CGH as a modulation pattern, i.e., a 3D video can be reconstructed by sequentially altering the CGH. However, it is extremely challenging to reconstruct a 3D video interactively using electro-holography because of the enormous amount of computation required for calculating Eq. (1).

3. Proposed method

The proposed method comprises an encoding scheme on the distributor side and a decoding scheme on the receiver side, as schematically depicted in Figs. 2 and 3, respectively. Further, we proceed to describe the details of each scheme.

 

Fig. 2 Block diagram of the encoding scheme of the proposed method.

Download Full Size | PDF

 

Fig. 3 Block diagram of the decoding scheme of the proposed method.

Download Full Size | PDF

3.1. Encoding scheme based on vector quantization

The encoding scheme encodes the PLSs for subsequent display on the receiver side by applying VQ-based compression. The PLSs are encoded using three steps. In step 1, the PLSs are classified into several groups based on their depth (z_j). Here, each separate depth range can be defined as a group of layers (GOL), with each layer being a plane that is parallel to the x–y plane. When a GOL contains L layers, the kth GOL includes the PLSs for which $kL\le z\le \left(k+1\right)L-1$, i.e., jth PLS belongs to kth GOL, where

In step 2, the presence or otherwise of PLSs in each layer of a GOL is converted into a bit string for each $\left(x,y\right)$. For example, as depicted in Fig. 2, the $\left(X_1,Y_1\right)$ position in the GOL with k = 6 has PLSs in layers 1, 2, 3, and5; therefore, it is converted into the bit string 10111. Given this bit string, their positions $\left(X_i,Y_i\right)$, and k, all the PLSs can be completely reconstructed.

Further, steps 1 and 2 are implemented by sorting the coordinates of the PLSs in the $z,x,y$ order because the encoding method encodes PLSs of the same k, x_j, and y_j using the same bit strings. Figure 4 depicts the encoding scheme as a flowchart, and Table 1 presents an encoding example. After sorting the PLS coordinates in the $z,x,y$ order, we calculate k using Eq. (2) to determine whether adjacent PLSs are present in the same GOL. Further, we determine whether x_j and y_j correspond to each other. PLSs with the same k, x_j, and y_j are encoded as a bit string in which the $l_j=\mathrm{mod}(z_j,L)$ bit (counting from the least significant bit) is 1 and the remaining bits are 0. However, the bit strings are immediately determined if the adjacent PLSs have different k, x_j, and y_j. Further, the bit strings are assembled into a file together using the values of k, x_j, and y_j. In step 3, the encoded PLSs are compressed using the existing Zstandard file compression scheme [13] and are further transferred to the receiver side. Because the encoding scheme of the proposed method can contain multiple PLSs in one code, the data capacity of the 3D model is reduced, especially when the 3D object comprises PLSs in high density.

 

Fig. 4 Flowchart for determining the bit string.

Download Full Size | PDF

Table 1. Example of the encoding scheme.

View Table

3.2. Decoding scheme using the lookup-table method based on vector quantization

The decoding scheme comprises three steps. In step 1, the existing file compression scheme is used for decompressing the received data. In step 2, a one-dimensional (1D) CAD is read from an LUT corresponding to the encoded PLSs. In step 3, the CGH is created by circularly expanding the 1D CAD using the run-length-encoding LUT (RLE-LUT) method [14] and by synthesizing it according to the $\left(X_i,Y_i\right)$ position in the decompressed data. We call the CGH creation process which is described in step 2 and step 3 as VQ-LUT method.

The LUT method is an effective methodology for reducing the amount of computation associated with CGH calculation [14–21]. This method pre-calculates the CAD that corresponds to the coordinates of each PLS, and synthesized according to each coordinate of PLSs. According to Eq. (1), PLSs with the same z_j have corresponding CADs, which are synthesized using the LUT method by the sliding position on a hologram when the 3D model contains two or more PLSs with the same z_j. Although the LUT method reduces the computational load associated with Eq. (1), the amount of required memory is often a significant obstacle to its usage in practice.

In this study, for reducing the amount of required memory on the receiver side, we compressed the CAD of each PLS to 1D data. Figure 5 schematizes the manner in which an LUT is created for the VQ-LUT method. First, we pre-calculate the CAD of zone plates whose depth corresponds to the depth of each layer in a GOL; in other words, we calculate L pattern of CAD of a zone plate (STEP 1). Then, we compress those CADs to 1D CAD using its radial symmetry (STEP 2). Next, we combine the 1D CADs based on all possible patterns of bit string; thus, the LUT of each GOL contains $2^L$ elements (STEP 3). The combined 1D CAD is stored in a LUT (STEP 4). For example, a 1D CAD in the LUT corresponding to a bit string "01001" is synthesized using 2nd and 5th 1D CAD from the left in Fig. 5 which is illustrated as switching (SW) the data path before inputting the 1D CAD to the adder in Fig. 5.

 

Fig. 5 Overview of the creation of a lookup table.

Download Full Size | PDF

The VQ-LUT method exhibits two advantages: it requires less time to read and synthesize the LUT data and the usage of circular symmetry reduces the amount of memory required for storing the LUTs. With respect to the first advantage, because the VQ of the PLSs in the encoding scheme allows simultaneous synthesis of the CADs of multiple PLSs, few readouts and syntheses of LUT data are required, thereby accelerating the CGH calculation. With respect to the second advantage, all the synthesized CADs in the LUT exhibit circular symmetry because the CAD of each PLS exhibits circular symmetry and because the CADs in the LUT are synthesized from the PLSs that are aligned on the z axis. Thus, it can be compressed to 1D data along the radial direction, which are easily decompressed using the RLE-LUT method with relatively little computational load [14]. Overall, the proposed method requires V amount of memory; further,

4. Results and discussion

To evaluate the proposed method, we emulated both a distributor-intensive system with conventional video coding (HEVC, JPEG2000) and a receiver-intensive system using the proposed method. In case of HEVC, we used HEVC Intra only because it is more effective than HEVC Inter [2]. The evaluation indicators were (i) the data rate saving (DRS), (ii) the image quality, and (iii) the computational load on both the distributor side and the receiver side. Further, the evaluation conditions were (i) a CGH resolution of $2,048\times 1,024$ pixels, (ii) a CGH bit depth of 8 bits, and (iii) monochrome (λ = 520 nm). The calculation environment comprised an Intel Core i7 8700K 3.7 GHz CPU (single core only), DDR4-2400 32 GB memory, the Windows 10 Professional 64-bit operating system, and the Microsoft Visual C++ 2017 compiler.

We prepared two 3D models in which the PLS coordinates are within the range of $0\le x_j<2,048$, $0\le y_j<1,024$, and $0\le z_j<512$, including a fountain with 320,000 PLSs [Fig. 6(a)] and a merry-go-round with 90,000 PLSs [Fig. 6(b)]. The lattice spacings are 8 μm on the x and y axes and 512 μm on the z axis. To evaluate the VQ efficiency, we prepared other 3D models that were randomly extracted from the original models. In general, a VQ-based compression scheme is efficient as more allocation to elements containing multiple elements. Here, we expect the compression efficiency to improve when multiple PLSs are continuously aligned in the z direction, which is likely to occur when PLSs are distributed with high density. Therefore, we evaluated the performance for different PLS densities.

 

Fig. 6 Three-dimensional models: (a) fountain and (b) merry-go-round.

Download Full Size | PDF

4.1. Data rate saving

DRS quantifies the reduction in data rate on the transmission path and can be defined as

For performing comparison, we set the UDR as the data size of an uncompressed CGH, i.e., 2M Bytes. The data size of $x_j,y_j$ in the vector quantized data is 4 Bytes, k and encoded bit-string is 2 Bytes.

 

Fig. 7 Data-rate savings: (a) fountain and (b) merry-go-round.

Download Full Size | PDF

Figure 7 compares the DRS among the proposed method with $L=1,4,8$, HEVC Intra, and JPEG2000. In case of the HEVC encoding, we used ffmpeg 3.4 for Windows x64 [22] with a compression rate factor (CRF) of 28, which is the default value for ffmpeg. In case of the JPEG2000 encoding, we used ffmpeg and set the compression level as 1 or 2; in Fig. 7, these appear as JPEG2000 (cl1) and JPEG2000 (cl2), respectively. The CGHs used for the HEVC and JPEG encoding were created using the VQ-LUT method (L = 1).

As depicted in Fig. 7, the DRS of the proposed method is observed to be superior to those of HEVC and JPEG2000 for almost all number of PLSs in both the models. For example, in case of the fountain model, the DRS is 75.8% in the proposed method (L = 4), 61.2% for HEVC, 50% for JPEG2000 (cl1), and 75.0% for JPEG2000 (cl2). Furthermore, the DRS of the proposed method deteriorates with more PLSs whereas those of HEVC and JPEG2000 exhibit a minor change, which is reasonable because the amount of data to be compressed in HEVC and JPEG2000 is dependent on the CGH resolution, whereas that in the proposed method is dependent on the number of PLSs. However, the degree of DRS deterioration obtained using the proposed method decreases with increasing number of PLSs because the PLS density increases, thereby increasing the probability of the allocation of multiple PLSs to one bit string and increasing the VQ efficiency.

Further, the DRS of the receiver-intensive system does not change with the CGH resolution using the proposed method, whereas that of the distributor-intensive system is considered to increase as the square of the CGH resolution. Therefore, the higher the CGH resolution, the larger will be the merit of the proposed method with respect to the DRS.

Meanwhile, the optimal value of L with respect to the DRS is observed to differ for each 3D model using the proposed method. For example, as depicted in Fig. 7, L = 4 is optimal for the fountain, whereas L = 1 is optimal for the merry-go-round. Further, the DRS is mainly determined by three factors: (i) the effect of reduction in the number of PLSs that is to be calculated using the VQ (ii) the length of the encoded bit string and (iii) the effect of the file compression scheme. Because the Zstandard file compression scheme used in this study is based on dictionary compression, its compression efficiency increases when the similarity of the target data increases. When L = 1, the length of the bit string is the shortest. Besides, because the bit string is always the same, the similarity in the data becomes higher; therefore, Zstandard is more effective than it is with other values of L. However, when the reduction in the number of times for reading and synthesizing the CADs in the LUT using VQ exhibits a considerable effect, this exceeds the compression effect of (ii) and (iii). Therefore, the optimal value of L is dependent on the 3D object.

To verify the above assumption, we analyzed the effect of VQ. Figure 8 compares the histograms of the number of “1” bits in the bit strings of the encoded data of the two objects using the proposed method when L = 4 and $N=90,000$. Approximately 13 % of the PLSs are allocated bit strings with multiple “1” bits in the fountain, whereas approximately 8 % of the PLSs are allocated bit strings with multiple “1” bits in the merry-go-round. This indicates that the compression effect of VQ is higher in the fountain than in the merry-go-round. Therefore, the optimal value of L for the fountain is 4 and that for the merry-go-round is 1. In practical situations, L could be switched based on the analysis of 3D models.

 

Fig. 8 Comparison of the histograms of the number of “1” bits in the bit string (L = 4 and N = 90,000).

Download Full Size | PDF

4.2. Image quality

To evaluate the image quality in terms of the visual performance of the 3D images, we measured the peak signal-to-noise ratio (PSNR) of the numerically reconstructed images of the CGHs that were created or decoded on the receiver side. The reference imageused for calculating the PSNR was obtained from the CGH that was created by solving Eq. (1) directly with double precision. To obtain the numerically reconstructed images, we simulated the wave propagation from the CGHs to around 0.3 m, where the 3D objects exist, by the angular spectrum method using the CWO++ library [23]. Further, we obtained 128 images of the intensity distribution at an interval of 2,048 μm from a distance of 0.3 m away from a CGH and measured the average PSNR of each image. In the VQ-LUT method, we applied the RLE-LUT method [14] without error compensation while creating the CGHs, and we prepared the LUT data with float precision. Note that the original CGHs encoded by JPEG2000 and HEVC were created usingthe VQ-LUT method (L = 1).

 

Fig. 9 Examples of the numerically reconstructed images.

Download Full Size | PDF

 

Fig. 10 Average peak signal-to-noise ratio (PSNR): (a) fountain and (b) merry-go-round.

Download Full Size | PDF

Figure 9 depicts the examples of the images that were numerically reconstructed from the CGHs, and Fig. 10 compares the PSNR of various methods. Further, the numerically reconstructed images in Fig. 9 were obtained from the CGHs that were created using 320,000 PLSs for the fountain and 90,000 PLSs for the merry-go-round. The number in each rectangle of Fig. 9 is the PSNR between the direct calculation of Eq. (1).

As depicted in Fig. 10, the PSNRs of JPEG2000 and HEVC are lower than those in all the versions of the VQ-LUT method because JPEG2000 and HEVC are lossy compression schemes. The PSNR of each version of the VQ-LUT method remains identical regardless of the value of L because the PLSs are completely reproduced using the decoding scheme of the VQ-LUT method. Further, the image quality obtained using the VQ-LUT method is poorer than that obtained by directly solving Eq. (1) because of a quantization error that occurs while expanding the 1D LUT data during the CGH calculation. Because the proposed method circularly expands a 1D CAD in the LUT to form a CAD corresponding to a zone plate, this quantization error must be included out of the area on the axis [14]. However, we conclude that the proposed method can result in favorably good 3D images because it is generally difficult for the human eye to distinguish between two images whose PSNRs exceed 30 dB [24].

 

Fig. 11 Calculation time of the encoder: (a) fountain and (b) merry-go-round.

Download Full Size | PDF

 

Fig. 12 The computer-generated hologram (CGH) calculation times using the VQ-LUT method: (a) fountain and (b) merry-go-round.

Download Full Size | PDF

4.3. Computational load

Here, we evaluate the computational loads for both the distributor (encoder) and the receiver (decoder) sides by measuring the calculation time required for encoding a frame of PLSs and decoding a frame of the CGH.

Figure 11 compares the calculation time on the distributor side. The encoding scheme of the proposed method comprises two steps: VQ of the PLS data and Zstandard compression; further, image compression is the encoding scheme of the distributor-intensive system for the CGH created using the VQ-LUT (L = 1) method. Note that the encoding time of the distributor-intensive system does not include the CGH calculation time. As depicted in Fig. 11, the calculation load for the encoding of the proposed method is lower than that of the distributor-intensive system using HEVC and JPEG2000. Further, the encoding time of the proposed method gradually increases with the number of PLSs, which is reasonable because the amount of PLS data increases with the number of PLSs. In the proposed method, the Zstandard compression requires up to 25% of the encoding time.

Figures 12 and 13 denote the calculation time on the receiver side. The decoding scheme of the proposed method comprises two steps, Zstandard decompression and CGH creation using the VQ-LUT method; further, image compression using HEVC or JPEG2000 is the decoding scheme of the distributor-intensive system. The computational load of the distributor-intensive system is clearly lower than that of the proposed method because of CGH generation using the proposed method.

Figure 12 depicts the calculation time for the generation of CGHs using the proposed method. For reference, we measured the calculation time for creating CGHs by solving Eq. (1) directly and further obtained the speed-up ratio when compared to the calculation time using the proposed method. Although the calculation could not be completed in a reasonable time for practical use (e.g., video rate) in the present experiment, we succeeded in speeding up the calculation by a factor of 115 for the fountain (L = 8, $N=320,000$) and a factor of 85 for the merry-go-round (L = 1, $N=80,000$) when compared to solving Eq. (1) directly. Furthermore, in case of the fountain, the proposed method with L = 8 is at most 1.37 times faster than that with L = 1, which is considered as the conventional LUT method. Therefore, the VQ-LUT method is superior to the conventional LUT method in terms of calculation speed.

Considering the results in Fig. 12 in terms of the value of L, the most effective value of L differs according to the choice of 3D model, L = 8 for the fountain and L = 1 for the merry-go-round. This is because the CGH calculation speed is mainly dependent on the number of times that CADs are read from the LUT in the proposed method. Further, the computational load for CGH calculation in the proposed method concentrates on the expansion of 1D CADs that are read from the LUT. There are two cases for reducing the number of reads from the LUT: (i) when multiple points are encoded by VQ and (ii) when one expanded CAD is reused by multiple PLSs (e.g., when there are vector-encoded elements with the same bit string but different $x,y$). The probability of case (i) arising increases with the PLS density, whereas that of case (ii) decreases with L because the probability of matching the bit string of the encoded data in the same GOL decreases with the length L of the bit string. Therefore, we can assume that the calculation time of the fountain was strongly affected by the former and that the calculation time of the merry-go-round was strongly affected by the latter. In practical use, L should be switched according to the features of the3D model.

 

Fig. 13 The decoder calculation times: (a) fountain and (b) merry-go-round.

Download Full Size | PDF

5. Conclusion

In this study, a compression scheme has been proposed for electro-holography with the VQ of PLSs. The proposed method is targeted for application in a receiver-intensive system that transmits 3D data (e.g., PLSs) and further creates CGHs on the receiver side; subsequently, the data amount on the transmission path and the computational load of the generation of CGHs on the receiver side are calculated. For example, the proposed method achieves approximately 76% of DRS, which is 1.24 times greater than HEVC intra coding; the proposed method calculates CGH 1.37 times faster than the LUT method without VQ. Further, the image quality of the proposed method is sufficient and highest in the evaluation. Although the computational speed of the generation of CGHs is currently insufficient for reconstructing 3D video at video rates, it would be improved by implementing the proposed method on a dedicated computer system such as a system based on field-programmable gate arrays [25].