1. Introduction

Displays are an interface that bridges the two ends of an information delivery system: the electronic signal supplied to the display and the viewer receiving the information converted to light by the display. For effective communication and a realistic visual experience, the interface should be tailored to the capabilities of the human visual system. Therefore, display manufacturers have worked to improve the design of their products—desktop monitors, projectors, televisions, cinema, and virtual-reality (VR) and augmented reality (AR) headsets—with the goal of achieving a realistic dynamic experience.

On a display the motion of an object is presented as a sequence of static views. A proper design would ensure that the viewer perceives smooth motion free of artifacts [1,2]. Unfortunately, this goal is often not achieved. Rather, a variety of motion artifacts can occur including flicker, judder (unsmooth motion), edge banding, motion blur, color breakup, and depth distortion [1,3–6]. To predict when artifacts will appear and how to minimize them, one needs to consider the properties of the input image sequence, the spatial and temporal parameters of the display, and the spatio-temporal visual sensitivity of the viewer as he/she is positioned to view the display. Our software tool aims to make accurate predictions of the above motion artifacts, given a set of user-defined parameters, in order to predict performance for modern display designs.

To model the visual processing of the input stimulus we employ the contrast sensitivity function (CSF). The CSF characterizes the visual system’s response to luminance variation in space and time. The CSF plots the reciprocal of the just-visible contrast at various spatiotemporal frequencies. Said another way, the CSF delineates how the visual system’s sensitivity to contrast varies across frequency bands, determining stimuli visibility. The visible domain has been called the window of visibility [1,2]. The motion on a display creates an output spatio-temporal frequency spectrum after filtering by the CSF. As long as that output spectrum is identical to the output spectrum of continuous real-world motion, the displayed object will be free of motion artifacts [1]. Luminance has a significant effect on the CSF, so a prediction tool should also incorporate that parameter. Our model includes the effects of spatial and temporal frequency as well as luminance while remaining computationally simple for practical use.

We developed, for the first time to our knowledge, a ’click-and run’ toolbox to guide the design of modern displays. We call it the Binocular Perceived Motion Artifact Predictor (BiPMAP). It enables prediction and visualization of motion artifacts. In this version of the toolbox, the input stimulus is a bright object moving horizontally at constant speed across a dark background [1,3]. Our main contributions are:

We note that other efforts along these lines have been presented [1,7–13]. But ours is the first, to our knowledge, to incorporate such a broad range of stimulus, display, and viewing parameters and to include the effect of eye movements and to include the phenomena of color breakup and depth distortions.

2. Background and related work

2.1 Window of visibility for the perception of a time-sampled moving stimulus

A displayed moving stimulus is a time-sampled version of the corresponding real continuous motion. A useful method for representing and analyzing sampled motion was developed by Watson and colleagues [1]. Here, we review this development, which forms the framework for our toolbox.

For simplicity, consider an infinitely long vertical line with unit contrast moving horizontally at constant speed. The resulting contrast distribution along the vertical axis is uniform, while along the horizontal axis $x$ is:

If the stimulus is illuminated for the full duration of each frame (sample and hold), the sampled stimulus now has a temporal staircase presentation:

Watson and colleagues approached the problem of predicting motion artifacts in the Fourier domain. The frequency spectrum of the continuous stimulus is:

The stroboscopic sampled stimulus has a spectrum of:

For the staircase representation of a sampled stimulus, the Fourier transform yields the same function as Eqn. (5) multiplied by a temporal sinc function (sin$\pi \omega /\pi \omega$) whose effect is to modulate the amplitudes of the replicates:

To examine how the above frequency spectra are altered by the visual system, we use the CSF to quantify the resolution limits of the visual system in space and time. Watson and colleagues [1] approximated the boundary between visible and invisible frequencies as a rectangle in the spatial and temporal frequency domain, and called this the window of visibility [1]. As we said earlier, this conceptualization is very useful because it proposes that a discrete stimulus on a display will appear the same as a continuous stimulus whenever the frequency spectra after filtering by the CSF are the same.

2.2 Motion artifacts

We consider several motion artifacts:

3. Overall design of BiPMAP

The BiPMAP toolbox has the following design goals. It should allow for a comprehensive list of user inputs, including the velocity of the moving stimulus, and nearly all display and viewing parameters. It should adopt a simple but accurate CSF model. The toolbox should present continuous and sampled stimuli side-by-side, so that motion artifacts caused by the discrete sampling that accompanies displays can be identified, investigated, and possibly eliminated by adjusting design parameters. Finally, it should allow visualization of a variety of motion artifacts in non-stereoscopic and stereoscopic displays.

BiPMAP consists of an interactive user interface and computational pipeline (Figs. 1 & 2). The executable toolbox is available at: https://github.com/CIVO-BiPMAP/executable/releases

 

Fig. 1. Overview of BiPMAP. (A) Front end with user-defined parameters divided into two components: Setup (configuration inputs) and Parameter Selection (simulation variables). (B) Back-end pipeline for artifact predictions containing two streamlines—Non Stereo (top) and Stereo (bottom)—and a figure-generation step. When "Compare Mode" in the Setup component of (A) is enabled, the Comparison Processor integrates information from past runs and outputs a comparison figure.

Download Full Size | PDF

 

Fig. 2. Input parameters in the user interface. The three panels on the left show the interfaces for selecting, respectively, stimulus, display, and viewing parameters. Within each panel, only parameters of the specified group can be defined. The panel on the right shows the interface for stereoscopic displays.

Download Full Size | PDF

3.1 Inputs and running modes

3.2 Motion-artifact predictions on non-stereoscopic displays

The pipeline consists of four steps to compute and visualize artifacts (Fig. 1(B)):

The tool outputs and presents results from all four of the aforementioned steps, including both continuous and sampled representations of the stimuli, to enhance user interpretation.

3.3 Input parameters & stimulus configuration

There are three sets of input parameters: Stimulus properties, display parameters, and viewing parameters (Fig. 2).

Stimulus parameters (Fig. 2, Stimulus):

Display parameters (Fig. 2, Display):

Viewing parameters (Fig. 2, Viewing)):

3.4 Mathematical representation of user-defined stimuli

Equations (2–6) set up the basic framework for our analysis. We next describe how we incorporated several additional parameters. Specifically, we provide details on how we expanded Watson’s development to enable a comprehensive list of user-defined parameters. If the reader wishes to skip the mathematical development, go to Section 3.5.

We first consider the situation in which no eye movement occurs. Starting from Eqn. (1), the contrast distribution from a narrow moving object with unit contrast can be expressed as:

Temporal sampling of Eqn. (9) generates a stroboscopic representation of the moving stimulus:

We can use a generalized staircase function to incorporate the hold interval $h$ and pixel response $\tau$:

Applying the Fourier transform to Eqn. (13), the spectrum of the configured stimulus is:

$Z_g(\omega )$ is the Fourier transform of the generalized staircase function (Eqn. (12)):

Plug Eqns. (15 and 16) into Eqn. (14):

Thus far all frames are treated as black and white. The case for $RGB$ frames is similar, but with a few modifications. Each frame has three channels, and each channel has its own contrast distribution function in the same format but with different shifts in space and time. For the contrast distribution for all channels, the fill factor $f$ in $c_z$ in equations below should be replaced with $f_{RGB} = f/3$. In the simultaneous presentation mode of RGB colors:

In the sequential $RGB$ mode, the frame period $\Delta t$ in the staircase function $z_g(x,t)$ is replaced with $\Delta t/3$, and sampling frequency $\omega _s$ in its Fourier transform $Z_g(u, \omega )$ is replaced with $3\omega _s$. There is an additional shift in time:

The resulting frequency spectrum is:

Next, we consider the situation in which the viewer makes a smooth eye movement to track the stimulus. When tracking occurs, all captured frames produce the same contrast distribution on the retina at the frame onset. However, when the hold interval $h$ is greater than 0, the eye’s motion introduces opposite motion on the retina during a frame. Therefore:

The trajectory is continuous during frame intervals given the smooth motion of the eye. The spectrum of the stimulus is:

Similarly, the representation of $RGB$ stimulus incorporates the additional shifts and a different pixel fill factor. For the contrast distribution for each channel, the fill factor $f$ is now $f_{RGB} = f/3$. In the simultaneous presentation mode of $RGB$ colors:

The sequential $RGB$ mode is more complicated with object tracking. The additional time shift is accompanied by spatial offsets for different colors, and the staircase widths along both temporal ($\Delta t$ in $z_g(x,t)$) and spatial axes are reduced by a factor of 3:

The resulting frequency spectrum is:

3.5 CSF model

A binary rectangular window, i.e., the "window of visibility" proposed by Watson [1], is an oversimplification of the frequency response of the visual system. Other factors that affect sensitivity, such as luminance, should be considered as well. The "pyramid of visibility" is an upgraded model, also proposed by Watson and colleagues, in which high-frequency sensitivity was fit with linear models (in log space). The pyramid defines the CSF as a 3D surface with a height that increases with luminance (Fig. 3), and is only valid for high-frequency bands [8]. The stelaCSF model, in contrast, is more thorough but complicated with five dimensions [12]. In our CSF model, we incorporated three dimensions: spatial frequency, temporal frequency, and luminance. Previous work has shown that the CSF is approximately separable at high frequencies, so we assumed separability to simplify computation. Our CSF model is:

 

Fig. 3. Model of human contrast sensitivity function (CSF). (A) Empirical temporal CSF model (solid lines) obtained from fitting experimental data (round markers), plotted as a function of temporal frequency for different retinal illuminances (in trolands). (B) Dependence of spatiotemporal CSF on luminance. Log contrast sensitivity is plotted as a function of temporal ($\omega$) and spatial frequency ($u$) at two luminances: 0.5cd/m² (inner cone) and 160 cd/m² (outer cone).

Download Full Size | PDF

From Eqns. (28–30), we obtained our spatiotemporal CSF model. As expected, peak contrast sensitivity grows with increasing luminance. For example, it increases by a factor of 4 as luminance increases from $0.5$ to $160\mathrm {cd/m^2}$ (Fig. 3(B)). The specific CSF profile for the configured stimulus is determined based on the average luminance across the viewer’s fovea:

The predicted stimulus based on the input’s Weber contrast is then determined using the CSF model mentioned above.

3.6 Depth distortion in stereoscopic displays

In the stereoscopic mode, the user inputs parameters involved in calculating disparity. Binocular disparity is the position of a feature in the right eye relative to the same feature in the left eye. Specifically,

The disparity error for a variety of stimulus speeds, capture rates, and flash numbers is:

The right panel in Fig. 2 shows the user inputs.

Parameters in binocular disparity calculation:

4. Task-based functionality of BiPMAP

4.1 Judder

From Eqns. (1–6), increasing speed produces counterclockwise rotation in the space-time domain and horizontal shear in the frequency domain. Figure 4 shows this for speeds increasing from $1$–$10$cm/s. Slope in space-time is greater for the faster speed (Fig. 4, left column), and shallower in the Fourier domain (Fig. 4, second column). The third and fourth columns show, respectively, the spectra after filtering by the CSF, and perceived stimuli after filtering. If we assume a capture rate of 120Hz, only one component of the frequency spectrum for the slower sampled stimulus is filtered through the CSF (Fig. 4, second row, filtered spectrum), and that component is the same as the one in the continuous stimulus. In other words, the output frequency spectra of the continuous and sampled stimuli are identical (Fig. 4, first and second rows, filtered spectrum), which means that the sampled stimulus should be perceived as moving smoothly (Fig. 4, first and second rows, Reconstruction). In contrast, the faster stimulus produces more than three replicates that fall within the window of visibility (third and fourth rows). As a result, the reconstructed stimulus has gaps, which means that the perceived motion will be discontinuous: i.e., judder will occur. Note that the filtered spectra for the slower and faster stimuli only intersect the temporal frequency axis at the origin, which means that there will be no visible flicker despite the presence of judder.

 

Fig. 4. Effect of stimulus speed on judder. Stimulus column: The stimulus is a bright line whose position is plotted as a function of time. The user specifies line width and speed. Input spectrum column: 2D discrete Fourier transform of the stimulus. Spatial frequency and temporal frequency are plotted on the ordinate and abscissa, respectively. Brightness represents amplitude. Output spectrum column: Spectrum filtered by the CSF. Reconstruction column: Perceived stimulus reconstructed using inverse Fourier transform of the output spectrum. Top two rows: Results for continuous (first row) and sampled stimulus (second row) with a speed of 1cm/s (1.15°/s). No motion artifacts perceived from the sampled stimulus. Bottom two rows: Results for continuous (third row) and sampled stimulus (fourth row) with a speed of 10cm/s (11.42°/s). Pixel density: 300dpi. Capture rate: 120Hz. Hold interval: 0.5. Viewing distance: 50cm.

Download Full Size | PDF

The occurrence of judder and not flicker is also revealed in Fig. 5. In this set of examples, the speed is 1cm/s while the capture rate increases from 30Hz (Fig. 5, second row) to 120Hz (bottom row). Greater capture rates—i.e., sampling rates—push the replicates in the sampled frequency spectra farther from one another until only one remains inside the window of visibility. Thus at sufficiently high capture rates, the reconstructed stimulus (Fig. 5, bottom row) should appear to move smoothly (top row).

 

Fig. 5. Effect of capture rate on judder. Columns in same format as Fig. 4. Top row: Results for continuous stimulus. Second row: Output from sampled stimulus with $30$Hz capture rate. Third row: Same but with capture rate of $60$Hz. Fourth row: Same but with capture rate of $120$Hz. Stimulus speed: 1cm/s (1.15°/s). Pixel density: 300dpi. Hold interval: 0.5. Viewing distance: 50cm.

Download Full Size | PDF

When the hold interval is increased from a small value ($\sim$0) to a large one ($\sim$1), the temporal sinc function attenuates higher frequencies in the replicates, but that attenuation occurs mostly outside the window of visibility. Thus increasing the hold interval does little to suppress judder.

4.2 Motion blur

When viewers track a moving continuous stimulus, the velocity on the retina is zero, but because the brain measures the eye motion through extra-retinal signals, the stimulus will appear to be moving and doing so smoothly. When viewers track a sampled stimulus of the same velocity, they perceive the motion but also often experience motion blur [14]. The hold interval is crucial here. When the hold interval is large, blur is experienced because the static image of the stimulus in one frame is smeared across the retina as the eye keeps moving [28]. This is illustrated by Fig. 6, bottom row. Notice the diagonal lines in the left panel, which is the smearing across the retina during the hold interval. The limited spatiotemporal bandwidth of the visual system acts like a low-pass filter and causes a sharp contour to appear blurred. Reducing the hold interval causes less retinal smear and therefore motion blur is minimized or even eliminated (Fig. 6, second row).

 

Fig. 6. Effects of hold interval and tracking eye movement on motion blur. Columns in same format as Figs. 4 and 5. Stimulus position and spatial frequency are in retinal coordinates. Viewer makes a tracking eye movement to fixate the moving stimulus. The continuous stimulus (row 1) is therefore stationary on the retina. Hold interval in the sampled stimulus is 0.1 and 1 in the second and third rows, respectively. Motion blur occurs with large but not small hold interval. Stimulus velocity on display: 20cm/s (22.62°/s). Pixel density: 300dpi. Capture rate: 120Hz. Viewing distance: 50cm.

Download Full Size | PDF

4.3 Color breakup in field-sequential RGB displays

Color breakup is another artifact that affects the viewing experience. It occurs when the display presents colors sequentially as with most DLP projectors [29]. Breakup is most noticeable when viewers track the displayed stimulus with smooth eye movements (Fig. 7, middle row), but it can also be seen when the eye is stationary and the stimulus moves past [15]. When red, green, and blue are presented sequentially within a frame and the viewer tracks the stimulus, each color will fall on a slightly different part of the retina. As a result, a white line on a black background will be imaged on the retina as three spatially separated lines: red, green, and blue; red will appear farther in the direction of motion than green, which will appear farther than blue. Thus, color breakup is an appearance of spatial color fringing caused by offsets in time. One can apply spatial offsets to eliminate the artifact [19]. The spatial offset required for each color channel is:

 

Fig. 7. Color breakup with tracking eye movement. Left column: Stimulus moving at 20cm/sec (22.62°/s) with tracking eye movement. Position on the retina is plotted as a function of time. Right column: Reconstructed stimuli. Top row: Continuous stimulus with red, green, and blue presented simultaneously. Middle row: Sampled stimulus presented with color-sequential display. Bottom row: Sampled stimulus with spatial offsets to counter color breakup. Stimulus speed: 20cm/sec (22.62°/s). Pixel density: 300dpi. Capture rate: 120Hz. RGB mode: sequential. Hold interval: 1. Viewing distance: 50cm.

Download Full Size | PDF

These predictions from BiPMAP are confirmed by experimental data from Johnson and colleagues [19]. They showed that when one applies the spatial offsets specified by Eqn. (35), color breakup is completely eliminated when the viewer tracks a moving object and is greatly diminished when the viewer does not track a moving object.

4.4 Depth distortion in stereoscopic displays

When filming or generating content for stereoscopic displays, the left and right images are usually captured simultaneously. Many stereoscopic displays then present the left and right images in temporally alternating fashion to the two eyes [30]. As a consequence, horizontally moving objects can appear to be displaced in depth; this is a depth distortion [31]. To understand the cause of this effect, we must look into how the brain estimates binocular disparity. Consider a situation in which the disparity of a moving object is intended to be zero (meaning the left- and right-eye images are the same and the object is meant to appear at the depth of the display screen) and the left- and right-eye images are presented in alternation in a given frame. The brain must pair the left and right images in order to compute the binocular disparity. The problem for a given left-eye image is whether to pair it with the following right-eye image or the preceding one. If the left-eye image occurs in the first half of the frame, the pairing with the following image is correct and the pairing with the preceding one is incorrect. But the brain has no way to know which pairing is the correct one, so it matches using an average of the two [23,32]. Specifically, the left-eye image is paired with both the preceding and following right-eye images and the disparity is derived from the average of those two pairings. The estimated disparity is therefore incorrect, which produces an apparent displacement of the object in depth. BiPMAP allows the user to determine what the estimated disparity is likely to be for different stimuli and display protocols.

Consider an object moving from left to right with zero disparity (i.e., it should be seen in the plane of the display). When the left-eye image is presented before the right-eye image in each frame, the estimated disparity will be the average of 0 (pairing with the following image) and -${\Delta }x$ (pairing with preceding image). So instead of obtaining the correct estimate of zero disparity, the brain obtains an incorrect estimate of -${\Delta }x/2$, which in the example shown in the left column of Fig. 8 produces an average disparity error of 5.65minarc, which is readily visible because it is more than an order of magnitude greater than the disparity threshold. The error can be derived from Eqn. (34) by substituting $-{\Delta }x/{\Delta }t$ for $v$, $1/{\Delta }t$ for $r$, and $1$ for $f$:

Because of this, the object’s perceived distance is nearer than it should be: i.e., a depth distortion. If the motion was right to left, the object would appear farther than it should. These predictions of BiPMAP have been confirmed experimentally by Hoffman and colleagues using the same model [3]. They showed that when simultaneously captures images are displayed alternately to the two eyes, the depth distortion predicted by BiPMAP is observed. And they showed that the distortion can be eliminated by inserting a nulling disparity that has the opposite sign from the disparity distortion predicted by BiMPAP.

 

Fig. 8. Estimated disparity with different display protocols. Labels for each column indicate that capture ($C$) is simultaneous and presentation ($P$) is sequential. The number followed by $X$ is the flash number. Top row: Simultaneous capture and sequential presentation with different flash numbers for a stimulus with a nominal disparity of zero moving left to right. Position on the display is plotted as a function of time. Red lines indicate the left-eye presentation and blue lines the right-eye presentation. The first, second, and third panels show the displayed stimulus for single-, double-, and triple-flash, respectively. Bottom row: Estimated disparity for these protocols. Estimated disparity is plotted as a function of time. Green dots represent the disparities associated with left- and right-eye pairings in which right eye leads or lags the left eye. Dashed line indicates the estimated disparity. Stimulus speed: $10$cm/s (11.31$^{\circ }$/s). Capture rate: $60$Hz. Viewing distance: 50cm.

Download Full Size | PDF

This depth distortion can be eliminated, of course, by presenting the images to the two eyes simultaneously or by capturing the left- and right-eye image data sequentially at the same alternation rate as will be used in the presentation. This BiPMAP prediction has been confirmed experimentally [3].

Field-sequential presentation is commonplace in stereoscopic cinema and is usually accompanied by multi-flash. For example, in the RealD protocol each pair of images is presented three times (’triple flash’) before updating to a new pair of images [33]. This reduces the magnitude of the expected depth distortion by a factor of 3 as shown by Eqn. (34) and the right panel of Fig. 8. The expected effect with double flash is smaller and illustrated by the middle panel of the figure.

When the viewer tracks the horizontally moving object with a smooth eye movement, the depth distortion created by alternating presentation is the same as that without the eye movement. This can be shown by running BiPMAP in stereo mode with eye tracking enabled (Fig. 9). This prediction has been confirmed experimentally [3].

 

Fig. 9. Effect of eye motion on estimated disparity. Left panel: Stimulus (top) and estimated disparity (bottom) with no tracking eye movement. Right panel: Stimulus (top) and estimated disparity (bottom) with tracking eye movement. Green dots represent the disparities associated with left- and right-eye pairings in which right eye leads or lags the left eye. Dashed line is the estimated disparity. The estimated disparity is the same with and without eye tracking. Stimulus speed: $10$cm/s (11.31$^{\circ }$/s). Capture rate: $60$Hz. Viewing distance: 50cm.

Download Full Size | PDF

5. Conclusions and future direction

We developed a toolbox—BiPMAP—for predicting and visualizing motion artifacts that are often seen when viewing digital displays. The toolbox enables users to input parameters including the stimulus configuration, the properties of the display, and viewing parameters. They can then determine whether motion artifacts will be seen and, if they are, which ones will be seen. By adjusting input parameters, the user can determine how best to minimize or eliminate the artifacts. Hopefully, this tool will aid the development of future displays.

BiPMAP could in future be further extended to include other parameters such as retinal eccentricity, chromatic vs luminance variation, and refractive error [12,24,34,35]. The model of the visual system could incorporate additional nonlinear mechanisms, more sophisticated chromatic models, and more sophisticated eye movement models [36]. In addition, more complicated stimuli could be added at the front end: i.e., natural video. Finally, image-quality metrics could be included so that the user can obtain a quantitative estimate of the quality of the display being prototyped [37–39]. Current predictions of judder and blur could be extensively calibrated via further experiments. We intend to add these features in future additions of the toolbox.