1. Introduction

Nowadays, with the rapid development of 3D technology, people increasingly demand more and more realistic 3D imagery. Therefore, the commercialization of 3D technology is becoming more and more mature, such as with 3D games [1], 3D TVs [2], or even virtual reality technology [3]. These 3D technologies have become commonplace in our daily lives, and there is a lot of research on these technologies, making our daily lives more and more convenient. However, although they can easily show beautiful 3D scenes, if humans watch scenes generated by these technologies for a long time, some bad phenomena could occur, such as fainting, nausea or cold sweats. Therefore, it is very difficult to show a very natural 3D scene. However, holography is considered to be the ultimate 3D technology that can satisfy all physiological factors of the human eye (such as binocular parallax, focus adjustment and convergence etc.) [4].

Holography is a technique that uses patterns of interference and diffraction to record and reconstruct real three-dimensional images of objects. "Hologram", that is, "all information" refers to the entire information of light emitted by an object being recorded and reconstructed by projection. Holographic technology is also commonly referred to as virtual imaging technology or holographic imaging. The imaging pattern is to record the phase and amplitude of the light wave of an object by means of light wave interference. At the same time, the light wave information of an object is displayed by means of the diffraction pattern to achieve the effect of 3D imaging. The computer-generated hologram (CGH), is a technology that simulates the recording part of traditional optical holography by using a computer, and it saves light information as electronic data called interference pattern. This technology has a lot of advantages, such as being able for making animation. However, CGH also has many disadvantages, and one of them is the large amount of calculation. Therefore, to solve the problem of the huge amount of CGH calculation, we optimize the CGH calculation by using saccades.

A saccade is one of a very rapid movement of the human eye, and it is an ability of the eye to quickly and accurately move from one target to another. This is very critical for reading and involves very precise and specific eye movements. A saccade happens very quickly and cannot be controlled by conciously. It usually takes about 200 ms to happen (which is the average human reaction time), and will be last for about 20 - 200 ms, which depends on the amplitude of the saccade [5]. The amplitude of a saccade means the angular distance that our eyes travel from one object to another. A saccade normally happens at a frequency of 2 - 8 times per second in daily life, but it can also happen due to an outside stimulus, such as the jumping movement of an object. During a saccade, the human eyes see almost nothing because visual information from our eye to our brain is suppressed. Due to this characteristic of the human eye, since the eye does not receive any visual information during a saccade, we can use this fact to detect a saccade of the human eye and reduce the resolution of CGH animation during the saccade to significantly reduce the CGH calculation amount, so that we can achieve CGH high-speed calculation without affecting observers watching CGH animation.

In this study, we propose a new method for CGH high-speed calculation using saccade suppression that lowers the resolution based on the Fresnel zone plate limitation. With this method, we can generate a CGH at a much faster calculation speed than the conventional CGH method without any negative effect on those viewing CGH reconstruction images.

2. Conventional method of CGH

2.1 Point-based method

For computer-generated holograms, it is very important to calculate object light (diffracted light field from the object). There is a lot of research dedicated to calculating object light, which can be approximately divided into the following methods: holographic stereogram methods [6,7], Fourier transform methods [8], wavefront recording plane method [9], Look-up table method [10], and the point-based method [11], which is used in our proposed method. We use the point-based method for calculating object light, and then, we use the ray-tracing method for generating a CGH interference pattern.

First, we will introduce details on the point-based method in this section. The point-based method is very important for computer-generated holograms. When calculating the light of an object, the core idea of the method is to use a very dense point light source to cover the entire surface of the object. One of the biggest benefits of this is that even if an object is very complicated, you can use a large number of point light sources to express the entire object in great detail. However, there is also a very obvious disadvantage to this method, that is, as the density of point light sources increases, the calculation amount of the point-based method will also increase in linear, which will seriously increase the burden on the computer. This will make the calculation amount become very huge and the calculation time become very long.

The details of point-based method are shown in Fig. 1. When we use point lights to display an object, we assume that the coordinates of the point light are represented by $P_{L}$. Therefore, the light wave information from the point light to the hologram plane can be represented by $H_{L}$, and the equation is shown in Eq. (1).

 

Fig. 1. Point-based method.

Download Full Size | PDF

2.2 Ray-tracing method

Ray-tracing method [12] is a special color rendering algorithm in computer graphics. It traces light emitted from the eyes instead of a light source. The rendering results obtained with this method are very similar to those of the ray-casting method and scanline rendering method, but this method prodeces better optical effects, such as more accurate analog effects for reflection and refraction, and is very efficient. Therefore, it is often used when pursuing high-quality rendering.

In practical applications, various electromagnetic waves or tiny particles can be regarded as ideal beams (lights). Based on this assumption, people use the ray-tracing method to calculate the propagation of light in a medium. First, the method calculates the distance, direction, and new position of a ray in a medium before it is absorbed by the medium or changes its direction, and then a new ray is generated from this new position, using the same processing method. Finally, a complete light propagation path in the medium is calculated pixel by pixel and shown on the display [13,14].

In CGH, the ray-tracing method can also be used as shown in Fig. 1. The rays are emitted from the center of a coordinate (we define the center coordinate as the viewpoint) at a specific angle and are in contact with all objects in an entire scene. When the rays collide with an object, we record the coordinate information and light information of the point light at the collision place, and record it on a hologram plane. By using the ray-tracing method for CGH [15], we can render CGHs at very high quality, create the reflections, refractions, shadows and shades, and remove the hidden surfaces to make CGH scenes more realistic.

3. Application of saccade suppression

We already know that during a saccade, our brain suppresses visual information from the eye, and we call this characteristic of human eye "saccade suppression". Therefore, in the field of VR, much research has used this characteristic to improve or solve problems in VR and have been published. One of them uses saccade suppression in the VR world to create a wider activity space than the real world [16]. When a saccade happens, it will last for some time, and this time is used to slightly shift the scenery in VR. Users move unconsciously diagonally because they move their bodies based on the information they see. Although the range of movement is small at one time a saccade happens, by repeating the movement while the saccade happens, it is possible to create a situation in which even if a user himself is walking straight toward the wall, by bending the VR world, the users can avoid collision with the wall. Therefore, the range of the room scale can be virtually expanded. Also, a route for guidence is calculated by a GPU in real time. Therefore, it is possible to avoid obstacles such as furniture in an area and to avoid objects that have entered the area later. Although this is still in the research stage, it can be put to practical use because it can be used without any awareness of the function by the user and can be used regardless of the type of content.

Through this research, it is very clear that, when a saccade happens, the brain really cannot receive almost any visual information from the eye. In addition, we can use this feature of the human eye to optimize various scenarios and algorithms to achieve unexpected results. Therefore, our study is also dedicated to making use of this characteristic of the human eye to develop a new system to make the calculation of CGH faster, without affecting observers watching CGH animation.

4. Proposed method

4.1 Details of saccade suppression

In the previous sections, we introduced the principle of the saccade and the current application of the saccade. As we know, every time our eyes move very quickly, a saccade occurs. Since the human eye can hardly obtain any visual information at this time, we can use this characteristic of the human eye to optimize the calculation of CGH. The method of using visual information being suppressed from our eyes to our brain is called "saccade suppression".

As we mentioned, one disadvantage of CGH is the massive calculation. To solve this problem, we must reduce the amount of CGH calculations. Human eyes almost see nothing when a saccade occurs. Therefore, the method we use is to reduce the resolution of CGH to reduce the amount of calculation when a saccade occurs to achieve high-speed calculation of CGH. Figures 2 and 3 show the details of our proposed method. We assume that the observer is watching CGH animation. Ideally, when we detect a saccade, the sensitivity of human eye will dramatically drop and we can reduce the resolution of CGH during the saccade, so that we can significantly reduce the amount of CGH calculation. However, even if the saccade takes up to hundreds of milliseconds, it is still difficult for CGH to reduce the resolution of CGH in real time while saccade suppression is performed. Therefore, we decided to prepare a set of CGH animations in advance. The animations contain random jumping motions of an object to cause a saccade, and then, we render several frames in low resolution after a jumping motion of the object. In subjective experiments, observers needed to watch CGH animations and to evaluate whether they feel the animations were blurred or not.

 

Fig. 2. Fast calculation with saccade suppression.

Download Full Size | PDF

 

Fig. 3. Concept of saccade suppression.

Download Full Size | PDF

Another problem is that if we only detect saccades and do not know their duration time, we still cannot know exactly for how long we can reduce the resolution of CGH animation, which will affect observers watching the CGH animation, and thus, we will fail to reach our aim. Therefore, it is very important to know what the duration time of a saccade is related to. Many pieces of research have proved that the duration time of a saccade is very stable and is almost linear with the viewing amplitude [17]. Equation (4) shows the relationship between the duration time of a saccade and the viewing amplitude.

 

Fig. 4. Relationship between the amplitude and the duration time.

Download Full Size | PDF

With saccade suppression, we can determine very accurately when a saccade will occur and how many frames of CGH animation we can render at very low resolution after the saccade occurs without any negative effect for observers watching CGH animation.

4.2 CGH resolution control based on Fresnel zone plate limitation

In our proposed method, saccade suppression is used to perform high-speed calculations of CGH. As mentioned in the previous section, we pre-render a few sets of CGH animations, which contain the random jumping motion of an object to cause a saccade. Then, during a saccade, we reduce the resolution of CGH animation to achieve high-speed calculation of CGH, so as to have no negative effect on observers watching CGH animation.

However, in the method of reducing resolution, the resolution of CGH cannot be reduced by reducing the number of pixels like CG (Computer Graphics). Therefore, to reduce the resolution of CGH, we have to use other methods. In our study, we used the point-based method to display an object and used the ray-tracing method to generate CGH reconstruction images. Therefore, we can reduce the resolution of CGH by expanding the ray angles that are traced to an object as shown in Fig. 5. However, if we just increase the angle between rays, when generating CGH, there will be very obvious gaps between point lights, which is undesirable for watching CGH animation. Therefore, to solve this problem, we must fill these gaps while expanding the angle of the rays.

 

Fig. 5. The way to expand the ray angle.

Download Full Size | PDF

In CGH, we record all the optical information of an object through the interference between object light and interference light. These generated interference patterns are made up of light and dark rings, and we call these rings "Fresnel zone plates". A whole CGH scene is made up of all Fresnel zone plates coming from all point lights of all objects. Due to the Rayleigh theory, those zone plates decide the size of the reconstruction point lights, which means deciding the resolution limit of CGH [18]. Equation (5) shows the relationship between the resolution limit of CGH and the size of Fresnel zone plate. Figure 6 shows the details of the Fresnel zone plate limitation.

 

Fig. 6. Fresnel zone plate limitation.

Download Full Size | PDF

With this equation, we can clearly understand that the size of the reconstruction point light is inversely proportional to the radius of the Fresnel zone plate. Therefore, we can increase the size of the reconstruction point lights by reducing the radius of the zone plate, so that the purpose of filling the gaps can be achieved [19,20]. As a result, when the angle of the rays becomes larger, the number of rays will become lower, and the amount of CGH calculation will be reduced dramatically. Therefore, we can greatly increase the calculation speed of CGH without affecting the observers watching the CGH animation with our proposed methd.

4.3 High-speed calculation of CGH based on Fresnel zone plate limitation and saccade suppression

The previous section introduced the principle of controlling CGH resolution by limiting the radius of the Fresnel zone plate. Here, we will introduce the principle of the high-speed calculation of our proposed method. The principle of the high-speed calculation of our proposed method is mainly caused by the following reasons: 1. the reduction of the amount of rays caused by the expansion of the angle of the rays, 2. the reduction of the Fresnel zone plate radius, and 3. the frequency and duration time of a saccade.

Equations (7) and (8) show the principle of the high-speed calculation of 1 and 2.

Based on Eqs. (7) and (8), we can understand that with the change in the angle of the rays and the change in the radius of a Fresnel zone plate, we can reduce the amount of CGH calculation in the fourth power of change. Therefore, by controlling the angle of the rays and the radius of the zone plate, our proposed method can significantly improve the calculation speed for CGH.

Another reason that can make CGH high-speed calculations is due to saccade suppression. The effect of saccade suppression on the high-speed calculation of CGH animation depends on the frequency of saccades and the duration time of each saccade. We know that in daily life, the frequency of a saccade in the human eye is around 2 - 8 times per second, and the duration of each saccade is affected by the amplitude. Therefore, we can come up with a equation as shown in Eq. (9) to express the impact of the three reasons on the speed of CGH calculation.

With Eq. (9), we can understand that with the increase in the frequency and duration time of a saccade and the reduction in CGH resolution based on Fresnel zone plate limitation, our proposed method will greatly improve the calculation speed of CGH.

5. Experiment design and results

In this section, we will introduce the design and results of our subjective experiments on our proposed method. In the subjective experiments, we mainly tried to understand the following things: 1. whether as we expected, a saccade occurred correctly after subjects observed the random jumping motion of an object and 2. whether the saccade suppression had any observable effect on the subjects while watching the CGH animation.

5.1 Experimental environment

First, to capture eye movements, we connected our holographic device [21] to a head-mounted device called ET-3D10. ET-3D10 is a device that can capture eye movements in real time. It has two infrared cameras on the top, which can accurately capture eye movements even in a dark environment. Then, we pre-rendered the CGH animations needed for the experiment with our computer and then played them through our holographic device. Subjects needed to wear our ET-3D10 when conducting experiments so that we could capture their eye movements in real time while they were watching our CGH animation. The parameters of our computer are shown in Table 1 and all calculation is by our GPU [22]. And details on our device are shown in Table 2, Figs. 7 and 8.

 

Fig. 7. Holographic device in concept.

Download Full Size | PDF

 

Fig. 8. Holographic device in real.

Download Full Size | PDF

Table 1. Parameters of computer

View Table | View all tables in this article

Table 2. Parameters of holographic display

View Table | View all tables in this article

5.2 Subjective experiments

5.2.1 Design of subjective experiments

Here, we are going to talk about the design of the subjective experiments.

In our experiments, each subject needed to watch all animations and evaluate them. The animations used in the experiment (each animation was in 60 FPS) contained an object randomly jumping 5 times. Figure 9 shows one example of the jumping motion. All animations were mainly divided into three major parts, each representing a different resolution lowered to 0% (which means a pure black frame), 10% and 30% in the duration time of a saccade. Figure 10 shows the example of low-resolution CGH patterns and reconstruction images. After watching each CGH animation, the subjects needed to tell us how many times they experienced unusual feelings (including blurring, flickering, etc.) for all 5 times. Then, we recorded the results of the subjects’ reports. We subtracted points according to the number of times the subject felt unusual. For example, if the subject experienced an unusual feeling one time, we subtracted 5 points from 1 to get 4 points, and 4 points was the final recorded result. The evaluation results are shown in detail in Table 3.

 

Fig. 9. Example of the jumping motion of object.

Download Full Size | PDF

 

Fig. 10. Example of the CGH patterns and reconstruction images.

Download Full Size | PDF

Table 3. Evaluation method

View Table | View all tables in this article

Another thing is that, we know that humans will respond to external stimuli, for example, drivers will brake when they see obstacles. However, human response to external stimuli is not instantaneous but takes a certain amount of time. As we know, the average human response time is about 0.2 seconds (approximately 12 frames in our 60 FPS CGH animation, and we call this the "delay frame"), so when we cause a saccade through the random jumping motion of an object, we must consider the response time of the subject. Although the average human response time is 0.2 seconds, for individuals, there is definitely a difference in response time. Therefore, in the experiments, we set the response time between the random jumping motion of the object and the saccade suppression to be 0.1 seconds to 0.2 seconds, which is about 6 to 17 delay frames.

Our experiments were set to the 3 different resolution during the saccade suppression and 12 different delay frames. Therefore, each subject needed to watch a total of 36 different CGH animations and evaluate the results in turn.

5.2.2 Results of experiments

Here, we are going to show the results of our subjective experiments. We first recruited 20 subjects, in 19 - 22 years old. However, at the time of the final result statistics, there were 5 subjects who had misunderstood the evaluation method. Therefore, we excluded the data of these 5 subjects. For all results, we counted the data of 15 subjects.

First of all, to ensure that each subject saw our CGH animations correctly and to make sure that each subject had a correct saccade at the right time, we used ET-3D10 to record the entire eye movements of each subject (which is equivalent to detecting a saccade), and the amplitude of the movements of each subject were recorded at a frequency of 30 Hz. Figure 11 shows the movement of the object in 3 of the 12 different categories and Fig. 12 shows the results of the saccades, the vertical axis in Fig. 12 represents the amplitude of the saccades in degree, and the horizontal axis represents the number of frames of eye movement detected by ET-3D10 in 12-second CGH animation. Because ET-3D10 scans the movement of eye in 30 Hz, the number of frames detected by ET-3D10 were 360 frames. In Fig. 12, we can see that the 15 subjects, for different CGH animations, all correctly observed the random jumping motion of the object, and all of their saccades happened at a similar time (some places in the figure where the amplitude changes greatly in a very short period of time).

 

Fig. 11. Movement of the object.

Download Full Size | PDF

 

Fig. 12. Part of results of the saccade in different categories.

Download Full Size | PDF

Next, we recorded the results of each subject’s evaluation of each CGH animation. Figure 3 is the evaluation method of our experiment, and Fig. 13 shows the average value of the evaluation results of all of the subjects in detail. The average human reaction time is 0.2 seconds, which means for our 60 FPS animation, it is about 12 frames. The result shown in Fig. 13 is that what effects will have on observers’ evaluation of CGH animation by setting different delay frames (reaction time). Therefore, the closer the delay frame is set to 12 frames (0.2 seconds), the harder it is for the observer to feel the blur, flicker, etc. of the CGH animation, which means that the score of this evaluation is higher. Therefore, the evaluation result is basically the highest score when the delay frame is 12, and the farther away from the 12 delay frame, the lower the evaluation score would be. As a result, in Fig. 13, we can see that when we set the delay frame to be close to the average human response time (about 12 frames in Fig. 13), the subjects could hardly sense any blurring or flickering in the CGH animations in the duration time of saccade at almost any resolution. However, as the delay time gradually increased or decreased from the average human response time (shown as larger or smaller than 12 delay frames in Fig. 13), the subjects could gradually see that the CGH animation was becoming blurry or flickering many times.

 

Fig. 13. Evaluation result in average.

Download Full Size | PDF

From the evaluation results, we can see that when the delay frames were between 8 and 9 or between 14 and 15, the evaluation result of the CGH animation with a 30% resolution was significantly higher than the other two resolutions, so we performed a t-Test on them. Table 4 shows the results. We can see that in the case of 9 delay frames, the evaluation for 30% resolution was significantly higher than the other two. However, although the results for the 30% resolution were not significantly higher than the other two under other conditions, the average value of the evaluation results was still higher than 3, that is, 5 random jump motions of the object could only be confirmed twice on average. Therefore, in the range of delay frames from 8 to 9 or 14 to 15, we believe that the CGH animation with 30% resolution can also make it possible for blurring or flickering of the animation to be hardly seen by observers.

Table 4. Results of t-Test

View Table | View all tables in this article

Finally, we calculated the CGH calculation time of the proposed method and the CGH calculation time of the conventional method and compared with them. For the improvement of calculation speed, we performed two parts of the calculation. 1. For the single CGH reconstruction image after the resolution is reduced, how much faster is our method compared with conventional method. 2. For saccades with a frequency of 2 - 4 times per second and a duration time of 20 - 200 ms (up to 12 frames), how many times the speed of the entire CGH animation can be increased by using saccade suppression. Here, we are going to show the results for the calculation time.

We have seen that the high-speed calculation of our proposed method is caused by a decrease in CGH resolution and saccade suppression. So, first of all, we will show the change in calculation time caused by the decrease in CGH resolution. Figure 14 shows the calculation time of the conventional CGH method of character A (which is shown in Fig. 9) and the calculation time after the resolution reduction. The ray angle with the conventional method is $0.034^\circ$, and the 30% resolution is $0.11^\circ$, the 10% resolution is $0.34^\circ$. Based on this result, we can conclude that even with a CGH resolution of 30% with delay frames set between 8 and 9 or 14 and 15, we can still get 4 times the calculation speed of traditional CGH calculation during saccade suppression. Although we only calculate the change of calculation time of character A, based on Refs. [19,20], we can know that even for more complex scenes, we can still get the corresponding reduction of calculation time.

 

Fig. 14. Calculation time of character A.

Download Full Size | PDF

Next, we will show the relationship between calculation speed and the duration time and frequency of saccades. Figure 15 shows the detailed relationship between the calculation speed and the frequency and duration time of a saccade. As we know, in daily life, the frequency of saccade in human eyes is about 2 - 8 times per second, and the average duration time of saccade is about 20 - 200 ms. Therefore, in CGH animation at 60 FPS, we calculated the magnification of the calculation speed of the saccade frequency at 2 - 4 times per second and the saccade duration time in 20 - 200 ms. And based on the results, we determined that with 4 times per second saccade frequency and 200 ms duration time, the calculation speed can be about 4.64 times faster than the conventional CGH method.

 

Fig. 15. Calculation speed.

Download Full Size | PDF

Due to the above experimental results, we can conclude that our proposed method can perform CGH calculation more quickly based on Fresnel zone plate limitation and saccade suppression and will not affect observers watching CGH animations.

6. Conclusion

In this study, we proposed a new method for the high-speed calculation of CGH using saccade suppression by lowering the resolution based on the Fresnel zone plate limitation. In our experiments, we demonstrated that by causing a saccade and setting the delay frame to an appropriate time, it is difficult for observers to sense any blurring or flickering of CGH animation in the duration time of the saccade even if we reduce the resolution of the CGH animation. Moreover, as based on the results of the calculation time, even if the range of delay frames is between 8 and 9 or between 14 and 15 and only a resolution of 30% can be used, we can still get 4 times the speed of the conventional CGH method by Fresnel zone plate limitation. Additionally, with a saccade frequency of 4 times per second and a saccade duration time of 200 ms, we can get a calculation speed of 4.64 times faster than the conventional CGH method by saccade suppression. In the future, we look forward to using our proposed method to enable the real-time calculation of CGH to be realized earlier. Also, we hope that our proposed method can be applied to CGH head-mounted displays [23] through eye tracking.