Open Access
Add to BibSonomy
Abstract
Incomplete optical distortion correction in VR HMDs leads to spatial dynamic distortion, which is a potential cause of VIMS. A perception experiment is designed for the investigation with three spatial distortion levels, with the subjective SSQ, five-scale VIMS level rating, and objective postural instability adopted as the evaluation metrics. The results show that the factor of spatial distortion level has a significant effect on all metrics increments (p<0.05). As the spatial distortion level drops off, the increments of VIMS symptoms decrease. The study highlights the importance of perfect spatial distortion correction in VR HMDs for eliminating the potential VIMS aggravation effect.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
The development of technology has changed society and the way people interact with each other. With the definition of Metaverse, a virtual twin of the actual world can be established and people can play, learn, work, and trade within the virtual environment [1]. As an essential technology for building an alternative world in the virtual, Virtual Reality (VR) is currently the most prominent consumer technology [2–4]. With the booming global demand for the complete virtual ecosystem, the study of VR consumer experiences is entering a new era.
Referring to the experiences of VR consumers, Visually Induced Motion Sickness (VIMS) has become a serious obstacle for VR applications [5]. VIMS is taken as an umbrella term used to describe the psychophysiological discomfort driven by visual stimuli with limited or absent physical movement [6]. Resulted from different application scenes with various devices, the VIMS can be labeled as cybersickness [7], gaming sickness [8], or simulator sickness [9]. VIMS symptom usually includes fatigue, headache, nausea, cold sweating, and even severe pain and vomiting [10].
Although numerous studies have focused on the causes of the VIMS, a fully adequate theory is not presently available. The theory of sensory conflicts between different motion signals is the mainstream explanation of the VIMS [11]. It’s been found that different signals are usually from the vestibular and visual systems and spontaneous postural stability is a pre-cursor of the VIMS symptom according to the study of Palmisano et al. [12], so people with greater postural instability are more likely to report VIMS when exposed to the VR environment with motion stimulus [13]. The eye movements theory announced that optokinetic nystagmus evoked by the motion stimulus may innervate the vagal nerve, which leads to the VIMS [14]. The sensory rearrangement theory has made a comparison between current sensory input signals with expected information based on previous experience [15]. A mismatch between the present inputs and the exposure history may trigger the VIMS. During the VR content acquisition, display, and perception procedures, geometric distortions may be introduced. The static geometric distortions fused with the motion characteristic, which results in dynamic distortion. The spatial dynamic distortions in the virtual world conflict with the expected stability and rigidity of the real world, leading to the VIMS symptoms [15].
As an important device to present three-dimensional (3D) VR content, Head-Mounted Displays (HMDs) need a precise optical design to realize a wide field of view, wide eye box, ergonomic eye relief, and so on [16]. To achieve the compact form factor of sunglasses, the binocular images need to be displayed within a limited distance. Therefore, the magnifying optics are required to move the focal plane of the display to a focusable distance for the HMDs. The thick lenses introduce undesired pincushion distortions, which are normally corrected by applying an algorithmic barrel pre-distortion. As incomplete correction may affect the reconstruction of 3D virtual space, including distance estimation [17] and slant perception of surface shape [18] in both static and dynamic images, and the monocular image distortions caused by insufficient pre-distortion directly contribute to the binocular fused virtual space, so it is important to correct the optical distortions sufficiently in VR HMDs for the veridical perception of surface attitude [19–21]. With the time dimension involved, the virtual space created by the HMDs is expressed with the connection and interaction between time and spatial perception [22], and the spatial dynamic distortions in HMDs become a more important issue in the perception and viewing experience aspects. As a potential factor affecting the symptom of VIMS, the 3D dynamic distortions caused by imperfect HMDs’ optical design have not been explicitly investigated.
As a pilot study, we designed a direct comparison perceptual experiment with a maximum of 45 minutes of 3D video watching for each session, a dynamic content long enough to induce the VIMS. The comparison study simulated three different spatial distortions in VR HMDs, with different level barrel pre-distortions for correcting a fixed level of pincushion distortion. The applied virtual space reconstruction model integrates the optical distortions of VR HMDs based on our previously proposed 3D virtual space model [23]. Subjective [24] and objective [25] evaluation methodologies are adopted in the experiment to record the VIMS-related discomfort levels. This study verified the theory of dynamic distortions aggravating VIMS in virtual space with a comprehensive experimental investigation, which is based on the geometric distortion induced by an incomplete optical correction in VR HMDs.
2. Spatial dynamic distortions in VR
2.1 Content acquisition, display, and perception in VR
The 3D content acquisition, display, and perception are three fundamental processes in VR technology, in which the real world or computer-generated scene space is transferred to the perceptual virtual world of the observer. There are two types of stereo shooting configurations for 3D content acquisition, parallel and toed-in camera configurations. Since the parallel camera configuration maintains the linearity during the conversion [26], and it is the most commonly used one, the parallel camera configuration is taken as an example for the conversion and geometric distortion analysis (see Fig. 1(a)). To make the optical axes intersect while the cameras are parallel, the camera sensors are shifted horizontally [27]. The real world or computer-generated scene space is defined as object space ($X_o$, $Y_o$, $Z_o$), which takes the midway of two cameras as the origin of the coordinates. The 3D content display and perception processes are illustrated in Fig. 1(b) [28]. The perceptual virtual world of the observer is defined as image space ($X_i$, $Y_i$, $Z_i$), which takes the midway between two eyes as the origin of the coordinates.
Fig. 1. VR content acquisition, display, and perception (Plan View). (a) Real-world or computer-generated 3D content acquisition; (b) virtual image display and perception. See Table 1 for the symbols’ meanings.
Table 1. Symbols used in the 3D content acquisition, display, and perception model.
There are three corresponding geometric transformation steps to demonstrate the space conversions. For the content acquisition step, a specific point $P_o$ in object space is mapped to the camera sensor coordinates as $O_l$ and $O_r$ respectively. In the camera sensor coordinate planes, the centers of the camera sensors are taken as the origins respectively. The conversion is accomplished through Eq. (1),
The content display step is converting the point in the camera sensor coordinates to the display screen coordinates. In the display screen coordinate planes, the centers of the display screen are taken as the origins respectively. $O_l$ and $O_r$ in the camera sensor coordinate planes are displayed on the display screen coordinate planes as $S_l$ and $S_r$ accordingly. The conversion can be taken as simple magnifying without considering any display distortion here. Equation (2) demonstrated the process,
The last content perception step is the binocular single vision process, in which the observed left and right views are fused into a single vision. In the display screen coordinate planes, $S_l$ on the left view is seen by the left eye, and $S_r$ on the right view is seen by the right eye, and they are fused to form the observed virtual point $P_i$ in the image space. The process is achieved through Eq. (3),
In this illustrated parallel camera configuration-based model, the values in the Y coordinates are identical for the left and right views on the camera sensors and display screen. Besides the specific geometric calculation using the adopted symbols in Eqs. (1) to (3), there is another way to evaluate the matching degree between object space and image space. The method is based on some important parameters that affect the geometry of constructed space. Including the introduced $b_c$, $b_e$, and $V_d$, $C_d$ is the convergence distance of the 3D camera system; $\alpha$ is the single camera’s field of view; $\beta$ is the single eye’s field of view. With the equal configuration of the baseline length ($b_c$ = $b_e$), the convergence distance ($C_d$ = $V_d$), and the field of view ($\alpha$ = $\beta$) between VR content acquisition, display, and perception, the perceived image space can match the object space perfectly without any geometric distortion [29].
2.2 Distortions caused by the display procedure of HMDs
The virtual space reconstruction model is based on the processes of VR content acquisition, display, and perception [23]. Each process can induce distortions, and the proposed simulation model here only focused on the display procedure with VR HMDs introduced geometric distortions. An HMD utilizes screens around 2 inches [4] to display images, and the near-eye optical lenses are needed to focus and place images at a desired optical distance. The magnification process leads to radial distortions, which are called pincushion distortion. Normally, the mature HMD product adopts the inverse distortion in its driver to counteract the radial distortion, which is the barrel distortion. The optical pincushion and pre-warping barrel distortion are both non-uniform and related to the eccentricity from the optical axis. The following simple model expressed in Eq. (4) can be used to describe such radial distortion,
where r is the radial distance from the optical axis to the point in the original image; R is the corresponding point’s radial distance from the optical axis in the processed image. With the same expression, the coefficient k determines the distortion magnitude and types (negative value for pincushion and positive value for barrel).Figure 2 illustrates the distortion simulation caused by the display process in HMDs. The distortions are in the two-dimensional (2D) planes when discussing the content display process. The checkerboard and natural image are adopted for the comparison between the original 2D image (see Fig. 2(a)), 2D barrel distortion (see Fig. 2(b)), and 2D pincushion distortion (see Fig. 2(c)).
Fig. 2. Distortions simulation caused by the display process in HMDs. (a) Original 2D image for comparison; (b) 2D barrel distortion; (c) 2D pincushion distortion.
The 2D distortions caused by the display process in HMDs are mapped to the 3D space in the content perception and 3D fusion process. Equation (5) demonstrated the process with Hadamard Product, and the values in the Y coordinates are kept identical for the left and right views. Figure 3 illustrates the 3D distortion simulation caused by the display process in HMDs and perceived by the observer. 3D barrel distortion (see Fig. 3(a)) and pincushion distortion (see Fig. 3(b)) are compared with the origin, in which the red color squares represent the non-distorted original distributed objects, and the blue color squares represent the distorted objects.
Fig. 3. 3D distortion simulation caused by the display process in HMDs and perceived by the observer. (a) 3D barrel distortion compared with the origin; (b) 3D pincushion distortion compared with the origin. The red color squares represent the non-distorted original distributed objects, and the blue color squares represent the distorted objects. The 3D coordinates here are consistent with the image space coordinates of the 3D content acquisition, display, and perception model.
3. Perception experiments setup
3.1 Experimental contents and display hardware
In the proposed simulation model, the distortion coefficients of optical pincushion distortion produced by the lenses and pre-warping barrel distortion are defined as $k_1$ and $k_2$, respectively. As a pilot study, three rough distortion levels are designed for the investigation. The pincushion distortion level is fixed by setting the coefficient $k_1$ as −0.4, and the coefficients of the barrel pre-distortion $k_2$ are set to 0.0, 0.2, and 0.4. The simulation results in three distortion correction levels: no correction, intermediate correction, and full correction, which are corresponding to the virtual spatial distortion levels: severe, intermediate, and barely, respectively. Figure 4 illustrates the 3D perception effects, comparing the original contents (red color) with different level barrel pre-distortions of pincushion distortion correction (blue color). The grids in the space can be relied on for better comparison. According to the objective spatial distortion evaluation method based on the Elastic Potential Energy Similarity [30], the distortion index of "barely distortion" (Fig. 4(c)) is only 1$\%$ of the "severe distortion" (Fig. 4(a)) distortion index. Therefore the "barely distortion" resulting from the full correction can be taken as the baseline of the investigation.
Fig. 4. Simulation of spatial distortions with different level barrel pre-distortions of pincushion distortion correction. (a) Severe distortion without correction; (b) intermediate distortion with intermediate correction; (c) barely distortion with full correction. The red color represents the non-distorted original objects, and the blue color represents the distorted objects.
As VIMS is a symptom accumulated over time, the perception experiment should be designed as long as possible to draw forth the VIMS symptom. Fifteen 3D video clips with each clip of 3 minutes are selected for the perception experiment, resulting in 45 minutes of 3D video content (see Fig. 5). The contents are classical movie and video game clips with high-velocity motion. With designed three virtual spatial distortion levels, the original 45 minutes of 3D video content are processed to generate three distortion levels of experimental video content. The perception experiment is divided into three sessions, and each with one distortion level that has a maximum of 45 minutes of viewing time. The experiment sessions have at least 48 hours of separation from each other to avoid the influence.
Fig. 5. 45 minutes 3D dynamic video consisting of 15 video clips with each clip of 3 minutes, which is processed to generate three distortion levels experimental video contents with each a maximum of 45-minute viewing time.
Nowadays, most of the leading HMD companies implanted the pre-warping processing procedure in their VR device drivers according to the distortion estimation of their specific optical design [31]. To avoid the influence introduced by the specific HMDs device, the 3D video contents are rendered with a desktop polarized stereoscopic display (PHILIPS, 278G4DHSD), which is 27 inches with a resolution of 1920$\times$1080 pixels. The investigation is focused on the impact of spatial dynamic distortions on VIMS, so the precise simulation of the perceived spatial distortion level is the priority. The desktop polarized stereoscopic display without extra spatial distortion is the preferred perception experiment rendering device.
3.2 Evaluation methods
The VIMS symptom measurement and evaluation are complicated tasks because there are no acknowledged guidelines to indicate the level of VIMS [32]. Usually, the subjective and objective evaluation methodologies are adopted simultaneously for VIMS investigation [33–35].
Subjective evaluation is the most natural method for estimating the participant’s discomfort level as it is a subjective phenomenon by nature [36]. The most widely used subjective VIMS evaluation methodology is the Simulator Sickness Questionnaire (SSQ) [5,24,37,38]. The SSQ is a self-report method that assesses the level of 16 discomfort symptoms that are divided into three distinct symptom clusters, which are defined as Oculomotor, Disorientation, and Nausea with distinct combinations of 16 symptoms, and each symptom has four rating levels: none, slight, moderate, and severe. The total score of the SSQ is used to indicate the subjective VIMS level, which is the sum of the weighted totals of the three symptom clusters. It takes some time for the observer to report the discomfort level of 16 symptoms, so the SSQ is normally reported only before and after the presentation of the VIMS-inducing stimuli. The increment of the SSQ score is adopted as the VIMS level index of the whole stimuli. During the VIMS-inducing stimuli presenting procedure, a quick subjective VIMS rating scheme is needed for pseudo-real-time evaluation. The VIMS level (VIMSL) rating [33,39–41] is adopted for quick rating that does not disturb the viewing procedure as much as possible. The VIMS level rating is an efficient five-scale method reporting the level of symptom from 0 (no VIMS) to 4 (extremely severe VIMS), which can be completed quickly during the experiment. The pseudo-real-time subjective evaluation method makes the subjective process study possible. Both SSQ and five-scale VIMS level rating methods are adopted in the proposed perception experiment.
The subjective evaluation methodologies are depending on the observer’s experience and current status, which are easily influenced by the individual’s bias. The objective evaluation methods rely on visual and cognitive fatigue-related physiological information [36]. There are several intensively used objective measurement ways, such as the Electroencephalographic (EEG) [5,42], functional magnetic resonance imaging (fMRI) [43,44], the postural instability measurement based on the balance board [45,46], and so on. Due to the validity, reliability, convenience, and economy [25,47,48], the objective evaluation methodology adopted in our proposal is the postural instability measurement using the Nintendo Wii Balance Board.
The balance board recorded the Center of Pressure (CoP) in the anterior-posterior (AP) and mediolateral (ML) directions, at 30 Hz. As conditions of eyes open and closed may have different responses on the postural instability measurement [49,50], two situations were tested in the experiment. During the postural instability measurement, the subjects were asked to stand still on the balance board for 1 minute with their eyes open and another 1 minute with their eyes closed. Figure 6 illustrates an example of recorded CoP points.
Fig. 6. Center of Pressure (CoP) recording for the postural instability evaluation with eyes closed, in the anterior-posterior (AP) and mediolateral (ML) directions.
Among the various metrics that can be derived from the analyzed data, the root-mean-square distance (RMSD) and 95$\%$ confidence ellipse area (Area) are used as the metrics indicating the VIMS-related postural instability [47,49,51]. The RMSD estimates the standard deviation of the vector amplitude from the mean CoP to each pair of points in AP and ML directions. The RMSD is calculated according to Eq. (6),
The 95$\%$ confidence ellipse area (Area) is the area of 95$\%$ bivariate confidence ellipse that enclose approximately 95$\%$ of the CoP points in the analysis. The calculation procedure is expressed in Eq. (7),
3.3 Experimental procedure
The experiment was carried out in a general lighted room with an air condition maintaining the temperature around 25$^{\circ }$C. The sequence of three sessions is random for each participant, and at least a 48-hour interval was arranged for recovery. The procedure of the perception experiment for each session is illustrated in Fig. 7. The participants completed the Informed Consent Form before the experiment, and they were told to have the right to terminate the experiment at any time. Before the specific 3D content watching, the participant was asked to complete the pre-experiment VIMS symptom status measurements, SSQ, and postural instability measurement. The postural instability was measured in two forms: eyes open and closed. The initial VIMS level was also recorded pre-watching. All the pre-watching recordings assess the initial level of VIMS-related symptoms, which were taken as the benchmarks for the analysis. During the 3D video watching, the participant was asked to sit 3 times the display height away from the 3D display, with a comfortable posture facing the center of the display. The VIMS level was rated after each 3 minutes of video clip within seconds. The participant can drop out of the experiment at any time. After the 3D video watching, the participant was asked to complete the SSQ and the postural instability measurement again.
Fig. 7. Perception experiment procedure for the three repeated sessions. The SSQ and postural instability measurements were completed before and after the 3D video watching. During the video watching, participants were asked to report the statuses through a five-scale VIMS level rating every 3 minutes (before and after each video clip).
3.4 Participants
A total of twenty-three participants, including five females, volunteered for the experiment. Participants’ ages ranged from 19 to 36 years, with an average age of 23.5 years. All subjects had a normal or corrected-to-normal visual acuity (20/20 or better on Test Chart 2000 Pro, Thomson Software Solutions, London) and normal stereo acuity $(\leq 70'')$ tested with a clinical stereo vision testing chart (Stereo Optical Co., Inc., Chicago, IL). All the participants administered the Motion Sickness Susceptibility Questionnaire (MSSQ); however, nobody was excluded due to the score of it.
4. Spatial distortion-related VIMS investigation
4.1 Subjective evaluations
The subjective evaluation data were analyzed using SPSS (Statistical Product and Service Solutions) software (IBM SPSS statistics 26.0). Table 2 listed the statistical SSQ and VIMSL scores for all participants, mean and standard deviations at different spatial distortion levels (severe, intermediate, and barely distortions), each with pre- and post-values. Based on the pre- and post-watching subjective evaluation data measurement, the evaluation metric value increment was calculated for each participant and distortion level. To investigate the difference in distortion level impacts among the three SSQ distinct symptom clusters, the Oculomotor, Disorientation, and Nausea are also included in the analysis of variance (ANOVA). The increments of the SSQ and its three distinct symptom clusters are calculated with the post-watching measured scores subtracting pre-watching measured scores. The increment of the VIMS level is calculated with the last VIMS level rating of the participant subtracting the initial VIMS level value. The results of an ANOVA are illustrated in Table 3. It is found that the investigated factor "spatial distortion level" has a significant effect on the subjective metric SSQ (p<0.05), while the factor has a highly significant effect on the subjective metric VIMS level (p<0.01). Referring to the three distinct symptom clusters of SSQ, it is found that the "spatial distortion level" has a significant effect on the increment of Disorientation and Nausea (p<0.05), while it has no significant effect on that of Oculomotor (p>0.05).
Table 2. SSQ and VIMSL scores, mean and standard deviations under different spatial distortion levels (severe, intermediate and barely distortions), each with pre- and post-values.a
Table 3. The results of ANOVA analysis, investigating the effect of factor "spatial distortion level" on the increments of subjective visually induce motion sickness (VIMS) evaluation metrics, simulator sickness questionnaire (SSQ), VIMS level and three distinct symptom clusters of SSQ (Oculomotor, Disorientation, and Nausea).a
As shown in Fig. 8, the subjective evaluation results illustrate the trend of evaluation value affected by the spatial distortion level. The increments of SSQ score and VIMS level all drop off with the decrease of the spatial distortion level as expected, and the significances of the difference are with a significant and a highly significant level, respectively. Adopted two subjective evaluation methodologies give the consistent result that the spatial distortion factor has a significant effect on the subjective perceptual VIMS symptom. With the same other characteristics of the 3D dynamic video content, the more spatial distortion will lead to a more severe VIMS symptom level according to the subjective report of the participants.
Fig. 8. Subjective evaluation results, mean and standard deviations. The significance of the difference is indicated. (a) Increment of the SSQ score; (b) increment of the VIMS level.
The subjective VIMS level rating methodology not only recorded the pre- and post-watching status but also the VIMS status during the watching period. The VIMS level was reported every three minutes, so including the pre- and post-evaluation values of each video clip, there are 16 records for most of the testing sessions. There are 5 sessions within 69, in which participants have not finished watching all the 15 video clips because the VIMS symptom reached an extremely severe level (recorded value is 4). The whole VIMS level rating results are illustrated in Fig. 9, with mean values and standard deviations before and after each video clip watching under each experiment session (severe, intermediate, and barely distortions). The results are analyzed based on the data from all participants, and to ensure reasonableness, the lacking data of the 5 sessions are topped up with the last value (an extremely severe level value 4). It is demonstrated in Fig. 9 that as time goes on, the rated VIMS level is continuously increasing, and the difference between different spatial distortion level sessions is enlarging too.
Fig. 9. VIMS level rating result analysis, mean values, and standard deviations before and after each video clip watching under each experiment session (severe, intermediate, and barely distortions).
To analyze the subjective rating differences between different spatial distortion level sessions more objectively, the ANOVA analysis is applied to investigate the effect of the factor "spatial distortion level" on subjective VIMS level rating for each time node. It is found that within the first 10 time nodes before 30 minutes, the factor "spatial distortion level" has no significant effect on the subjective metric VIMS level (p>0.05). Starting from the time node of 30 minutes, the significance of the difference arises. Figure 10 illustrates the difference analysis results of three spatial distortion levels based on the VIMS level ratings 30 minutes later. The mean values, standard deviations, and significances of the difference are all indicated in the figure. The results of significance show that after 30 minutes, the factor "spatial distortion level" has a significant effect on the subjective metric VIMS level (p<0.05) except for the time node of 33 minutes. However, the p-value of 33 minutes time node is 0.052, which is already a critical value. As time goes on, the significance is improving and the result of the last three time nodes shows a highly significant effect (p<0.01).
Fig. 10. The difference analysis of three spatial distortion levels based on the VIMS level rating result. The significance of the difference is indicated by the p-value.
4.2 Objective evaluations
The Center of Pressure (CoP) was recorded using the Nintendo Wii Balance Board for postural instability estimation, and the pre- and post-3D video-watching conditions were recorded with eyes open and closed. One minute recording with the frequency of 30 Hz leads to 1800 points for analysis of each trial. Figure 11 illustrates the typical CoP recording under different spatial distortion levels (severe, intermediate, and barely distortions), and eyes-closed situations with each pre- and post-values.
Fig. 11. Typical Center of Pressure (CoP) recording for the postural instability measurement, under different spatial distortion levels (severe, intermediate, and barely distortions), eyes closed situation with each pre- and post-values. (a) Pre-watching of severe distortion content; (b) post-watching of severe distortion content; (c) pre-watching of intermediate distortion content; (d) post-watching of intermediate distortion content; (e) pre-watching of barely distortion content; (f) post-watching of barely distortion content.
Based on the recorded CoP data and the evaluation methodologies, the two objective evaluation metrics can be calculated. Table 4 listed the postural instability evaluation metric based on the root-mean-square distance (RMSD), with mean and standard deviations under different spatial distortion levels (severe, intermediate, and barely distortions). The eyes open and closed situations are calculated separately with each pre- and post-values. Table 5 listed the postural instability evaluation metric based on the 95$\%$ confidence ellipse area (Area), with mean and standard deviations under different spatial distortion levels (severe, intermediate, and barely distortions). The eyes open and closed situations are calculated separately with each pre- and post-value. According to the analysis, there is no sign showing a significant difference between eyes open and closed situations. So, the averaged values of eyes open and closed are used as the final objective evaluation metrics.
Table 4. Postural instability evaluation based on the root-mean-square distance (RMSD, $cm$), mean and standard deviations under different spatial distortion levels (severe, intermediate and barely distortions), eyes open and closed situations are recorded with each pre- and post-values.
Table 5. Postural instability evaluation based on the 95% confidence ellipse area (Area, $cm^2$), mean and standard deviations under different spatial distortion levels (severe, intermediate and barely distortions), eyes open and closed situations are recorded with each pre- and post-values.
With the pre-tested benchmarks, the increment of the postural instability evaluation metrics is taken to estimate the VIMS symptom introduced by the 3D video watching. The increment of the postural instability evaluation metric here employs the way of values’ difference divided by two times their average, which can normalize the increment to the range between 0$\%$ and 100$\%$. For the non-zero initial value metrics, the normalized increment calculation method is more reasonable. With the ANOVA analysis, the effect of the factor "spatial distortion level" on the objective VIMS evaluation metrics, root-mean-square distance (RMSD), and 95$\%$ confidence ellipse area (Area) are investigated (see Table 6). The investigated factor "spatial distortion level" has a very highly significant effect on the objective metric RMSD (p<0.001), while the factor has a significant effect on the objective metric Area (p<0.05).
Table 6. The results of ANOVA analysis, investigating the effect of factor "spatial distortion level" on the increments of objective visually induced motion sickness (VIMS) evaluation metrics, root mean square distance (RMSD) and 95% confidence ellipse area (Area).a
As shown in Fig. 12, the objective evaluation results illustrate the trend of evaluation value affected by the spatial distortion level. The increments of postural instability based RMSD and Area all drop off with the decrease of the spatial distortion level as expected, and the significance of the difference with a very highly significant and significant level, respectively. Adopted two objective evaluation metrics give the consistent result that the spatial distortion factor has a significant effect on the objective perceptual VIMS symptom. With the same other characteristics of the 3D dynamic video content, a more spatial distortion will lead to a more severe VIMS symptom level according to the objective measurement of the participant’s status.
Fig. 12. Objective evaluation results, mean and standard deviations. The significance of the difference is indicated. Increments of the postural instability evaluation metrics are adopted as the objective evaluation of VIMS. (a) Increment of the root mean square distance (RMSD); (b) increment of the 95$\%$ confidence ellipse area (Area).
4.3 Correlation analysis
The correlation relationship was also analyzed using linear correlation and expressed in Fig. 13. The results show that each two of the four-evaluation metrics have a very high linear correlation. Comparing the correlation coefficients of subjective metrics, the value of VIMSL is always lower than that of SSQ when correlated with objective metrics (see Fig. 13(b) vs. Figure 13(c) and Fig. 13(e) vs. Figure 13(f)). The subjective SSQ evaluation method reporting 16 discomfort symptoms reveals the real condition of the participants better than simple five-scale VIMS level rating. Comparing the correlation coefficients of objective metrics, the value of RMSD is always higher than that of Area when correlated with subjective metrics (see Fig. 13(b) vs. Figure 13(e) and Fig. 13(c) vs. Figure 13(f)). The objective RMSD is a better calculation method for CoP-based postural instability estimation than Area for reflecting the participant’s condition of VIMS. Within all linear correlation analyses, the increments of SSQ and RMSD have the highest linear correlation, and the increments of VIMSL and Area have the lowest correlation.
Fig. 13. Correlation results between the increments of evaluation metrics. (a) SSQ correlated with VIMS level; (b) SSQ correlated with RMSD; (c) VIMS level correlated with RMSD; (d) RMSD correlated with Area; (e) SSQ correlated with Area; (f) VIMS level correlated with Area.
5. Discussion
According to the sensory rearrangement theory, besides the low-level inter-sensory conflicts between visual and vestibular signals, the visual-to-visual intra-sensory conflicts may be a higher-level inducement of the VIMS symptoms during the exposure to motion VR contents [15]. The spatial dynamic geometric distortions in the 3D virtual space [52] as a present visual input will conflict with the expected stability and rigidity in the real-world space. The proposed experimental study verified this hypothesis systematically. Due to the enormous individual variation, the individual case study is not proper for this investigation. The size of the participants is recommended to be more than fifteen for statistical analysis [53]. Since the particularity of VIMS symptoms, the exposure time of the design is another critical parameter. To induce the possible significant difference, the time of exposure should be more than half an hour with a certain high-speed motion content. The MSSQ that estimates the participants’ sensitivity to motion sickness was applied to all participants. No participants were excluded due to the MSSQ screening because the designed difference between distortion levels is big enough to induce the significant difference between symptom levels, and the post-analysis showed that there is no significant difference between MSSQ grouped participants.
The adopted subjective SSQ and objective postural instability evaluation methodologies only qualified the changes between pre- and post-watching. The pseudo-real-time VIMS level rating evaluation method [39] enables the deep investigation of VIMS symptoms invoked during watching. Based on the temporal VIMS level rating, the difference between the three spatial distortion levels is found to enlarge over time. There is no significant difference between the three spatial distortion levels only after half an hour of watching. According to the rising slope of the temporal VIMS level rating, the deteriorating trend of VIMS symptoms is mitigated a little bit around 20 minutes (see Fig. 9). This may result from the self-regulatory mechanism of the human brain working after around 20 minutes under the specifically designed strength of the content. The temporal VIMS evaluation methodologies are necessary for comprehensive investigation and a better understanding of VIMS, and objective temporal VIMS evaluation methods such as EEG and fMRI will be used in our further study on spatial dynamic distortion-related VIMS symptoms. The adopted rendering device is a desktop polarized stereoscopic display, which has limitations on validating the effects in HMDs. It will consider the replication in specific consumer HMDs in our further study.
As a pilot study to validate the impact of spatial distortions on VIMS, three rough distortion levels (barely, intermediate, and severe) based on different optical corrections are designed. This investigation is introduced with the optical distortion of VR and proposes a comprehensive 3D optical design-based perception model, but the significance of this research is not limited to this. The protagonist of this investigation is spatial dynamic distortion, which is emphasized by establishing a comprehensive model to present the 3D content acquisition, display, and perception processes. Although the HMD companies implanted the pre-warping processing procedure in their VR device drivers according to the distortion estimation of their specific optical design, the derived optical distortion is not small enough, which lit the interest in similar studies of distortion impact on distance estimation [17] and slant perception of surface shape [18]. Even though the HMD companies can make the perfect correction of the lens distortion, distortion caused by the display procedure is only one of the situations inducing spatial dynamic distortions. Besides the investigated display procedure, the 3D VR content acquisition and perception procedure may also introduce spatial dynamic distortions. The spatial distortions correcting in 3D imaging should consider all possible factors, but the device manufacturers and content creators are not cooperating well enough with individual observers presently. The human factor, which is the critical factor for the application of VR in the future metaverse, has not been paid enough attention to for the moment. Individual factors such as the cyclopean eye [23,54,55] are key elements to establishing a perfect virtual twin of the actual world model for a more immersive viewing experience and more accurate interaction.
6. Conclusion
The compact form factor of sunglasses chasing VR HMDs leads to the optical distortions induced by the magnifying optics, which is hard to compensate for. The incomplete correction, which may lead to geometric distortions in 3D virtual space, will produce spatial dynamic distortions if combined with dynamic content. Spatial dynamic distortion is a potential factor affecting the symptom of VIMS, which was investigated with a comprehensive experimental study. Three rough spatial distortion levels were simulated and applied to 15 successive 3D video clips as the content for the perception experiment. The SSQ, five-scale VIMS level rating, and postural instability were adopted as the evaluation metrics to indicate the increments of VIMS-related symptoms. The results show that the factor spatial distortion level has not only a significant effect on the increments of SSQ and 95$\%$ confidence ellipse area-based postural instability (p<0.05), but also a highly significant effect on the increment of VIMS level rating (p<0.01), and has a very highly significant effect on the increment of root mean square distance based postural instability (p<0.001) as well. As the spatial distortion level drops off, the VIMS-related evaluation metrics all decrease. Besides, the linear correlations are very high between different evaluation metrics. The study may highlight the importance of perfectly correcting the distortion in VR HMDs for eliminating its potential VIMS aggravation effect.
Funding
National Natural Science Foundation of China (61876121, 62002254); Natural Science Foundation of Jiangsu Province (BK20200988).
Acknowledgments
The authors are grateful to the volunteers for participating in the subjective experiments.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. S. Abbate, P. Centobelli, R. Cerchione, E. Oropallo, and E. Riccio, “A first bibliometric literature review on Metaverse,” 2022 IEEE Technol. Eng. Manag. Conf.254–260 (2022).
2. C. P. Chen, Y. Cui, Y. Chen, S. Meng, Y. Sun, C. Mao, and Q. Chu, “Near-eye display with a triple-channel waveguide for metaverse,” Opt. Express 30(17), 31256–31266 (2022). [CrossRef]
3. D.-I. D. Han, Y. Bergs, and N. Moorhouse, “Virtual reality consumer experience escapes: preparing for the metaverse,” Virtual Real. 26(4), 1443–1458 (2022). [CrossRef]
4. D. Cheng, Q. Hou, Y. Li, T. Zhang, D. Li, Y. Huang, Y. Liu, Q. Wang, W. Hou, T. Yang, Z. Feng, and Y. Wang, “Optical design and pupil swim analysis of a compact, large EPD and immersive VR head mounted display,” Opt. Express 30(5), 6584–6602 (2022). [CrossRef]
5. K.-M. Jang, M. Kwon, S. G. Nam, D. Kim, and H. K. Lim, “Estimating objective (EEG) and subjective (SSQ) cybersickness in people with susceptibility to motion sickness,” Appl. Ergon. 102, 103731 (2022). [CrossRef]
6. B. Keshavarz, B. E. Riecke, L. J. Hettinger, and J. L. Campos, “Vection and visually induced motion sickness: How are they related?” Front. Psychol. 6, 472 (2015). [CrossRef]
7. J. J. LaViola, “A Discussion of Cybersickness in Virtual Environments,” SIGCHI Bull. 32(1), 47–56 (2000). [CrossRef]
8. O. Merhi, E. Faugloire, M. Flanagan, and T. A. Stoffregen, “Motion Sickness, Console Video Games, and Head-Mounted Displays,” Hum. Factors 49(5), 920–934 (2007). [CrossRef]
9. E. Igoshina, F. A. Russo, R. Shewaga, B. Haycock, and B. Keshavarz, “The relationship between simulator sickness and driving performance in a high-fidelity simulator,” Transp. Res. Part F Psychol. Behav. 89, 478–487 (2022). [CrossRef]
10. J. R. Lackner, “Motion sickness: more than nausea and vomiting,” Exp. Brain Res. 232(8), 2493–2510 (2014). [CrossRef]
11. J. E. Bos, W. Bles, and E. L. Groen, “A theory on visually induced motion sickness,” Displays 29(2), 47–57 (2008). [CrossRef]
12. S. Palmisano, B. Arcioni, and P. J. Stapley, “Predicting vection and visually induced motion sickness based on spontaneous postural activity,” Exp. Brain Res. 236(1), 315–329 (2018). [CrossRef]
13. J. Munafo, M. Diedrick, and T. A. Stoffregen, “The virtual reality head-mounted display Oculus Rift induces motion sickness and is sexist in its effects,” Exp. Brain Res. 235(3), 889–901 (2017). [CrossRef]
14. S. M. Ebenholtz, “Motion Sickness and Oculomotor Systems in Virtual Environments,” Presence Teleoperators Virtual Environ. 1(3), 302–305 (1992). [CrossRef]
15. A. D. Hwang and E. Peli, “Instability of the perceived world while watching 3D stereoscopic imagery: A likely source of motion sickness symptoms,” i-Perception 5(6), 515–535 (2014). [CrossRef]
16. K. Bang, Y. Jo, M. Chae, and B. Lee, “Lenslet VR: Thin, Flat and Wide-FOV Virtual Reality Display Using Fresnel Lens and Lenslet Array,” IEEE Trans. Visual. Comput. Graphics 27(5), 2545–2554 (2021). [CrossRef]
17. F. Kellner, B. Bolte, G. Bruder, U. Rautenberg, F. Steinicke, M. Lappe, and R. Koch, “Geometric Calibration of Head-Mounted Displays and its Effects on Distance Estimation,” IEEE Trans. Visual. Comput. Graphics 18(4), 589–596 (2012). [CrossRef]
18. J. Tong, R. S. Allison, and L. M. Wilcox, “Optical distortions in VR bias the perceived slant of moving surfaces,” 2020 IEEE Int. Symp. Mix. Augment. Real. 73–79 (2020).
19. S. Lee and H. Hua, “A Robust Camera-Based Method for Optical Distortion Calibration of Head-Mounted Displays,” J. Disp. Technol. 11(10), 845–853 (2015). [CrossRef]
20. Y.-H. Lin, T.-W. Huang, H.-H. Huang, and Y.-J. Wang, “Liquid crystal lens set in augmented reality systems and virtual reality systems for rapidly varifocal images and vision correction,” Opt. Express 30(13), 22768–22778 (2022). [CrossRef]
21. Z. Luo, J. Zou, E. Zhao, Y. Rao, and S.-T. Wu, “Correcting the wavelength-induced phase deviation of Pancharatnam-Berry lenses,” Opt. Express 30(20), 36644–36650 (2022). [CrossRef]
22. P. Morat, A. Schwerdtfeger, and F. Heidmann, “In Case You Don’t Know What to Play: Framework for a VR Application that manipulates Time Perception through Spatial Distortion.,” Proc. Mensch und Comput. 2021, 609–613 (2021). [CrossRef]
23. Z. Xia and A. Hwang, “Self-position awareness-based presence and interaction in virtual reality,” Virtual Real. 24(2), 255–262 (2020). [CrossRef]
24. R. S. Kennedy and N. E. Lane, “Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness,” Int. J. Aviat. Psychol. 3(3), 203–220 (1993). [CrossRef]
25. R. A. Clark, A. L. Bryant, Y. Pua, P. McCrory, K. Bennell, and M. Hunt, “Validity and reliability of the Nintendo Wii Balance Board for assessment of standing balance,” Gait Posture 31(3), 307–310 (2010). [CrossRef]
26. H. Yamanoue, “The Differences Between Toed-in Camera Configurations and Parallel Camera Configurations in Shooting Stereoscopic Images,” 2006 IEEE Int. Conf. Multimed. Expo1701–1704 (2006).
27. S. Kishi, N. Abe, T. Shibata, T. Kawai, M. Maeda, and K. Hoshi, “Stereoscopic camera system with creator-friendly functions,” Proc. SPIE 7237, 72371M (2009). [CrossRef]
28. H. Urey, K. V. Chellappan, E. Erden, and P. Surman, “State of the Art in Stereoscopic and Autostereoscopic Displays,” Proc. IEEE 99(4), 540–555 (2011). [CrossRef]
29. A. Woods, T. Docherty, and R. Koch, “Image Distortions in Stereoscopic Video Systems,” Proc. SPIE 1915, 36–48 (1993). [CrossRef]
30. Y. Xu, Q. Yang, L. Yang, and J.-N. Hwang, “EPES: Point Cloud Quality Modeling Using Elastic Potential Energy Similarity,” IEEE Trans. Broadcast. 68(1), 33–42 (2022). [CrossRef]
31. J. Tong, R. S. Allison, and L. M. Wilcox, “The Impact of Radial Distortions in VR Headsets on Perceived Surface Slant,” J. Imaging Sci. Technol. 63(6), 60409-1–60409-11 (2019). [CrossRef]
32. T. Porcino, D. Trevisan, and E. Clua, “A cybersickness review: causes, strategies, and classification methods,” J. Interact. Syst. 12(1), 269–282 (2021). [CrossRef]
33. R. Liu, S. Cui, Y. Zhao, X. Chen, L. Yi, and A. D. Hwang, “VIMSNet: an effective network for visually induced motion sickness detection,” Signal, Image Video Process. 16(8), 2029–2036 (2022). [CrossRef]
34. J. Wu, Q. Zhou, J. Li, X. Kong, and Y. Xiao, “Inhibition-related N2 and P3: Indicators of visually induced motion sickness (VIMS),” Int. J. Ind. Ergon. 78, 102981 (2020). [CrossRef]
35. J. Kim, H. Oh, W. Kim, S. Choi, W. Son, and S. Lee, “A Deep Motion Sickness Predictor Induced by Visual Stimuli in Virtual Reality,” IEEE Trans. Neural Netw. Learning Syst. 33(2), 554–566 (2022). [CrossRef]
36. M.-C. Park and S. Mun, “Overview of Measurement Methods for Factors Affecting the Human Visual System in 3D Displays,” J. Disp. Technol. 11(11), 877–888 (2015). [CrossRef]
37. A. Jasper, N. Cone, C. Meusel, M. Curtis, M. C. Dorneich, and S. B. Gilbert, “Visually Induced Motion Sickness Susceptibility and Recovery Based on Four Mitigation Techniques,” Front. Virtual Real. 1, 582108 (2020). [CrossRef]
38. D. Stelling, M. Hermes, G. Huelmann, J. Mittelstädt, D. Niedermeier, K. Schudlik, and H. Duda, “Individual differences in the temporal progression of motion sickness and anxiety: the role of passengers’ trait anxiety and motion sickness history,” Ergonomics 64(8), 1062–1071 (2021). [CrossRef]
39. A. D. Hwang, H. Deng, Z. Gao, and E. Peli, “Quantifying Visually Induced Motion Sickness (VIMS) During Stereoscopic 3D Viewing Using Temporal VIMS Rating,” J. Imaging Sci. Technol. 61(6), 60405-1–60405-9 (2017). [CrossRef]
40. A. S. Fernandes and S. K. Feiner, “Combating VR Sickness through Subtle Dynamic Field-Of-View Modification,” 2016 IEEE Symp. 3D User Interfaces 201–210 (2016).
41. R. Liu, M. Xu, Y. Zhang, E. Peli, and A. D. Hwang, “A Pilot Study on Electroencephalogram-Based Evaluation of Visually Induced Motion Sickness,” J. Imaging Sci. Technol. 64(2), 20501-1–20501-10 (2020). [CrossRef]
42. S. A. A. Naqvi, N. Badruddin, M. A. Jatoi, A. S. Malik, W. Hazabbah, and B. Abdullah, “EEG based time and frequency dynamics analysis of visually induced motion sickness (VIMS),” Australas. Phys. Eng. Sci. Med. 38(4), 721–729 (2015). [CrossRef]
43. V. Napadow, J. D. Sheehan, J. Kim, L. T. Lacount, K. Park, T. J. Kaptchuk, B. R. Rosen, and B. Kuo, “The Brain Circuitry Underlying the Temporal Evolution of Nausea in Humans,” Cereb. Cortex 23(4), 806–813 (2013). [CrossRef]
44. J. Miyazaki, H. Yamamoto, Y. Ichimura, H. Yamashiro, T. Murase, T. Yamamoto, M. Umeda, and T. Higuchi, “Inter-hemispheric desynchronization of the human MT+ during visually induced motion sickness,” Exp. Brain Res. 233(8), 2421–2431 (2015). [CrossRef]
45. R. Li, H. Walter, C. Curry, R. Rath, N. Peterson, and T. A. Stoffregen, “Postural time-to-contact as a precursor of visually induced motion sickness,” Exp. Brain Res. 236(6), 1631–1641 (2018). [CrossRef]
46. J. Faubert and R. Allard, “Effect of visual distortion on postural balance in a full immersion stereoscopic environment,” Proc. SPIE 5291, Stereosc. Displays Virtual Real. Syst. XI 5291, (2004).
47. M. Salavati, M. R. Hadian, M. Mazaheri, H. Negahban, I. Ebrahimi, S. Talebian, A. H. Jafari, M. A. Sanjari, S. M. Sohani, and M. Parnianpour, “Test-retest reliabty of center of pressure measures of postural stability during quiet standing in a group with musculoskeletal disorders consisting of low back pain, anterior cruciate ligament injury and functional ankle instability,” Gait Posture 29(3), 460–464 (2009). [CrossRef]
48. F. Kinoshita, M. Miyao, M. Takada, and H. Takada, “Expression of balance function during exposure to stereoscopic video clips,” Adv. Sci. Technol. Eng. Syst. 2(1), 121–126 (2017). [CrossRef]
49. T. E. Prieto, J. B. Myklebust, R. G. Hoffmann, E. G. Lovett, and B. M. Myklebust, “Measures of Postural Steadiness: Differences Between Healthy Young and Elderly Adults,” IEEE Trans. Biomed. Eng. 43(9), 956–966 (1996). [CrossRef]
50. B. A. Springer, R. Marin, T. Cyhan, H. Roberts, and N. W. Gill, “Normative Values for the Unipedal Stance Test with Eyes Open and Closed,” J. Geriatr. Phys. Ther. 30(1), 8–15 (2007). [CrossRef]
51. D. Lafond, H. Corriveau, R. Hébert, and F. Prince, “Intrasession Reliability of Center of Pressure Measures of Postural Steadiness in Healthy Elderly People,” Arch. Phys. Med. Rehabil. 85(6), 896–901 (2004). [CrossRef]
52. A. D. Hwang and E. Peli, “Stereoscopic Three-dimensional Optic Flow Distortions Caused by Mismatches between Image Acquisition and Display Parameters,” J. Imaging Sci. Technol. 63(6), 60412-1–60412-7 (2019). [CrossRef]
53. ITU, “Methodology for the subjective assessment of the quality of television pictures,” Itu-R Bt.500-11 211(BT.500-11), 1–48 (2002).
54. M. Khokhotva, H. Ono, and A. P. Mapp, “The cyclopean eye is relevant for predicting visual direction,” Vision Res. 45(18), 2339–2345 (2005). [CrossRef]
55. Z. Xia and E. Peli, “Cyclopean Eye based Binocular Orientation in Virtual Reality,” SID Symp. Dig. Tech. Pap. 49(1), 381–384 (2018). [CrossRef]
