Open Access
Add to BibSonomyAbstract
We introduce a new concept to improve the contrast and peak brightness of conventional data projectors. Our method provides a non-homogenous light source by dynamically directing fractions of the light from the projector lamp before it reaches the display mechanism. This will supply more light to the areas that need it most, at the expense of the darker parts of the image. In effect, this method will produce a low resolution version of the image onto the image-forming element. To manipulate the light in this manner, we propose using an intermediate array of microelectromechanical system (MEMS) mirrors. By directing the light away from the dark parts earlier in the display chain, the amount of light that needs to be blocked will be reduced, thus decreasing the black level of the final image. Moreover, the ability to dynamically allocate more light to the bright parts of the image will allow for peak brightnesses higher than the average maximum brightness of display.
©2008 Optical Society of America
1. Introduction
1.1 Background
The two dominant types of data projectors available today are based on either Digital Light Projection (DLP), or Liquid Crystal Displays (LCD). Both employ different types of light valves – elements that selectively block light – to construct their images. A lamp provides a uniform illumination of the image, and the light valves of the LCD, for example, selectively reduce illumination of a pixel on the screen in order to form the dark parts of the image. The Digital Multimirror Device (DMD), inside a DLP projector functions in a similar manner. A uniform light source limits the maximum brightness for any image. For most images, however, only a fraction of the total area is illuminated at peak brightness. A conventional projector simply blocks the light which is not necessary for a scene, thereby wasting this fraction of light while it could be used to further illuminate the bright parts of the image.
Furthermore, the currently available light valves are “leaky” and cannot block all the light for black image areas [1]. The dynamic range of a projector is the ratio between the brightest and darkest levels it can display. Merely increasing the illumination through the projector lamp in a conventional projector does not necessarily increase the dynamic range, as both the darkest and brightest level rise. Current projector technology could benefit from an improvement in both dynamic range and peak brightness of the projected image.
1.2 Concept of an improved data projector
This paper details a new concept that addresses the contrast shortcomings of currently available projectors by adding a low-resolution intermediate mirror array to provide a non-homogenous light source. This intermediate mirror device is capable of directing the uniform light from the projector lamp incident on its surface to different areas on the light valve, in effect projecting a low-resolution version of the original image onto the light valve as shown in Fig. 1. Adding this intermediate mirror device will improve the dynamic range in two ways: by directing the light to the bright parts of the image, the achievable peak brightness will be increased. Simultaneously, the amount of light that needs to be blocked in the dark regions of the image will be reduced, thus decreasing the brightness of the black level.
Fig. 1. (a). Schematic of a conventional DLP projector, and (b) schematic of an enhanced DLP projector with second MEMS mirror array (AMA).
This intermediate device can be realized with a low-resolution analog micromirror array (AMA), made using microelectromechanical system (MEMS) technology. The tip and tilt angle (two degrees of freedom) of the micromirrors in the array can be set continuously in order to direct light to an arbitrary location on the light valve such as the DMD. Micromirrors have been an active area of research for over 20 years [2, 3]. The optical efficiency of micromirrors can be very high, limited only by the gaps between the individual mirrors in the array. The ratio of optical area to overall mirror area is referred to as fill-factor, and arrays have been demonstrated with a fill factor of 99% [3].
2. Related work
Contrast, or the dynamic range of a display device can be defined as the ratio cn:1=Ib/Id between the brightest Ib and darkest Id pixel generated by a projector. A technique developed by Seetzen et al. to improve the dynamic range of a display involves modulating two displays – or arrays of light valves – in series [4]. With the dynamic range c1:1 of the first display, and the dynamic range c2:1 of the second display, the theoretical contrast of the combined system is c1c2:1. Using two light modulators also results in a multiplicative increase in the number of different brightness levels.
Seetzen, et al., have developed an algorithm that generates images for each of two sequential light modulators from a given high dynamic range (HDR) image [4, 5]. The authors demonstrated the reconstruction of the original HDR image with a corresponding display prototype. In their most current HDR display, a low resolution array of light emitting diodes (LEDs) is used as heterogeneous background illumination of a conventional LCD screen, where the LCD provided the high resolution image correction [4]. In the algorithm, the original HDR image is first downsampled to the resolution of the LED array, then the actual brightness distribution provided by the LEDs is determined by taking into account the point-spread function (physical brightness distribution) of an LED through the display optics. The high-resolution LCD correction is then generated to correct for any perceivable differences between the LED illumination and the target image.
Some work has been reported on applying the concept in [4] to projectors. Pavlovych and Stuerzlinger [6] added a second light modulator by focusing the light from a regular DLP projector onto an LCD panel, and then projecting the image of the LCD panel onto a display screen. Overall contrast of this projection system is improved due to the reduction in dark level by the second LCD modulator, but at the cost of a significant reduction in overall brightness, as LCD panels typically transmit only 10% or less of unpolarized incident light [4]. Also, the extra dynamic range available in the dark regions will only be perceivable in a very dark room; the dark level increases regardless as ambient room illumination increases. In an even moderately-lit room, the effect of the lower dark levels will thus be lost, resulting in a net decrease in overall perceivable contrast compared to an unmodified projector [6].
Damberg, et al., [7] similarly employ two LCD panels in series. Their prototype includes the second LCD inside the projector, and uses six panels in total -- two for each colour channel. The authors report an order of magnitude increase in contrast relative to the original projector, but as in [7], this comes at the expense of the overall brightness of the image due to the light losses from the second LCD panel.
3. HDR projector image processing
3.1 Virtual, mobile light sources
In the approaches to dual-modulation described above, both modulators control light intensity in fixed regions, whether the first modulator is a set of LEDs or an LCD panel. A key aspect of improving dynamic range by using two modulators is that one of them can be of much lower resolution than the final displayed image. The light distribution of the low-resolution modulator is then corrected by the high-resolution modulator to achieve the desired image. However, due to the fixed location of the light sources from the first modulator, the spatial distribution of luminance in the final image may not be optimized through the spatial distribution of the low-resolution modulator. As an example, take the LED modulator detailed in [4]. A bright spot in the image may occur in between two adjacent LEDs. In that case, the neighboring LEDs need to provide higher light intensities than the local maximum of the image. The excess light from these LEDs would then have to be blocked by the high-resolution LCD.
In an AMA approach, the intensity of light of each virtual light source (PSF) is not changed, but the location on the high-resolution light valve is variable. This means that every PSF can be targeted to exactly where it is needed.
3.2 Greyscale levels
Besides peak brightness and contrast, a key attribute of any display is how many discrete steps of luminance in between the brightest and darkest settings are achievable. Conventional displays usually offer 8-bit control (256 steps) over luminance for each of the three color channels red, green and blue. While this is an adequate number of steps for conventional image formats, the much larger range of luminance of HDR displays also requires more brightness levels.
Visual psychologists such as Barten [8] have charted the number of steps necessary to cover a given range of luminance, given the limits of the human visual system. From Barten, 962 distinct steps are sufficient for a display that can reach from 0.05cd/m2 to 2700cd/m2. With two modulators as proposed by Seetzen et al. [4], the number of steps possible in each modulator is multiplied to get the total number of luminance steps for the combined display. Two linear 8-bit devices in series theoretically provide 2562 steps, which easily exceeds the requirement calculated by Barten.
Additional grey levels in an AMA-enhanced DLP projector will come from two main sources. First, multiple virtual light sources from different AMA mirrors can be directly overlapped. If only one micromirror area on the DMD is illuminated, all n AMA mirrors could divert their light to that spot, resulting in n possible grey levels from the AMA. The DMD could further modulate this light, so the maximum number of grey levels is 256n.
Also, the light from each AMA mirror will have an approximately Gaussian distribution on the DMD. The resulting continuously-varying intensities from the AMA mirrors will also provide more brightness levels than would be possible using just the DMD.
4. Mirror allocation algorithm
4.1 Coarse-to-fine representation of an image (Gaussian pyramid)
Similar to the approach by Seetzen, et al., [4] we first address the low-resolution light modulator, then determine the light distribution caused by the low-resolution light modulator, and finally, correct the image using the high-resolution second light modulator. Light from an individual micromirror of the AMA can be described by a point spread function (PSF) on the DMD, where it has an approximately Gaussian distribution as shown in Fig. 2(a).
Finding the optimal light distribution for the AMA corresponds to choosing the locations for each of the n PSFs from the array of n analog micromirrors. In our current approach, the original image and PSF image are represented as Guassian Pyramids, first described in [9]. We use an implementation by Simoncelli [10] to form a pyramid in which the original image is iteratively low-pass-filtered with a binomial filter to reduce its high-frequency content, while the image is simultaneously downsampled to reduce each of its dimensions by a factor of two. The PSF is treated similarly, as shown in Figs. 2(a) and 2(c). At lower resolutions, each pixel is a weighted sum of those around it at higher resolutions, so exploring the solution space at a coarse image representation allows easier recognition of the global luminance distribution in the image. Once an optimum solution for PSF placement is found at one level, it is refined at successively higher-resolution levels of the pyramid.
Fig. 2. Gaussian pyramid representation of a PSF (a) and (c) and an image (b) and (d). Each successively filtering step reduces the horizontal and vertical resolutions by factor of 2, from (a) to (c) and from (b) to (d).
4.2 Mirror allocation algorithm
Initially, the mirrors of the AMA are in a non-actuated, flat state so that all PSFs are equally distributed on the AMA. A “greedy” algorithm is used to optimize the placement of the PSFs by iterative redistribution. This algorithm uses the locally optimum choice at each iteration stage, assuming that in most cases this will lead close to the global optimum. The algorithm described below is not suitable for real-time applications, and is not guaranteed to find the optimal placement of mirrors, but is sufficient for a proof of concept to show the benefits of the AMA projector. The algorithm for the initial PSF placement progresses as follows:
on each pixel (in row r and column c) of the high resolution light valve is estimated from the sum of the contributions from all PSFs of the AMA micromirrors, where (ai,bi) is the center location of the PSF from mirror i.
- The illumination contribution from one mirror (PSF) is removed from Ip. (Id=Ip-PSFi)
- The relative per-pixel intensity in Id compared to the original image Im is found (Ir=Id/Im).
- The available PSF is placed so that its peak covers the minimum of Ir. (Ip’=Ip+PSFinew) This procedure is repeated with all other PSFs until there is no further change in the location of any of the PSFs after one full loop of all PSFs, or until a preset maximum number of iterations is reached.
To refine the solution at each higher level, the local area around each PSF is examined as a potentially improved location.
Finally, the placement of the PSFs will provide a heterogeneous illumination Pim for the second modulator, in this case the DMD. We must then find the high-resolution correction Idmd for the DMD, which will produce an image that should be close to the original HDR image with boost in contrast and brightness compared to a representation using a uniform illumination of the DMD. The final result for the projected image will be an optical multiplication of the two modulators: c*Im=Idmd*Pim, where c represents the improvement in brightness. Idmd is thus simply equal to c*Im/Pim, quantized to the 256 possible greyscale states of the DMD.
The maximum possible improvement factor c will depend on the image’s luminance distribution. If we wish to guarantee that all displayed pixels have sufficient luminance, we need to require c=min(Ip/Im). Above that, some pixels cannot be reproduced at the luminance specified by c*Im. For a completely white image, c would be 1, the minimum. This would still give the same brightness as a non-AMA projector. The improvement factor c may also be fixed beforehand. For instance, c could be specified to ensure that a stream of video has a constant relative brightness over time.
There may also be situations where the maximum brightness available is not needed – for instance, when rendering very dark images. In this case, the AMA could direct some light away from the DMD, discarding it before it reaches the DMD and affects the dark level.
5. Results
We have simulated the operation of an AMA projector with 100 micromirrors on an 800×600 pixel image. The shape of the PSF has been generated by convolving an 80×60 pixel region of the final image with a binomial filter to obtain an approximately Gaussian distribution with a width at half brightness of 130 pixels. The PSF is scaled so that the total luminance is preserved -- in an image where all AMA mirrors are in their original state, every pixel in PIm would have a value of 1. Figure 3 shows the results of the algorithm described above applied to the sample image memorial.hdr [11] in Fig. 3(a).
Fig. 3. (a). Original grayscale HDR image [9], (b) pictorial representation of the placement of the 100 PSFs, including a scale bar showing the number of mirrors at a particular location, (c) luminance map of the PSF image and scale bar showing brightness improvement from the base level of 1, and (d) PIm, the corrected image sent to the DMD Idmd, with scale bar showing the DMD greyscale level.
The locations of the PSFs from the 100 micromirrors are shown in Fig. 3(b). The corresponding luminance distribution that results from this PSF placement is shown in Fig. 3(c), and the high resolution DMD correction image in Fig. 3(d). The final image generated by the simulated AMA projector will be Fig. 3(c) multiplied by Fig. 3(d). For this example image, the AMA concept allows increasing the image brightness by 2.6 times compared to the non-AMA version of the same projector with the same projector lamp.
The original image in Fig. 3(a) is in HDR radiance format [12]. The dynamic range of HDR images is usually much wider than conventional, low-dynamic range (LDR) image formats can represent. PIm is a collection of all of the mirrors’ analog, continuously-variable PSFs, and therefore the luminance gradations are in PIm are continuous. With an HDR image, we can demonstrate that along with added brightness and efficiency, an AMA projector can also display many more greyscale levels than conventional 8-bit projectors by modulating the continuously-varying image PIm with the 8-bit image sent to the DMD.
6. Analysis – similarity metrics
The improvement factor, c, sets the brightness boost of the AMA projector for a given image. If c is set too high, the image cannot be reproduced perfectly because the total luminance available is constrained by the projection lamp. Choosing c thus becomes a tradeoff between maximizing total image brightness and maintaining image accuracy. Small brightness inaccuracies are also introduced by quantization, because the DMD can only be addressed to 256 discrete states.
To quantify the effect of these errors, we compare our simulated projected image with the original image. To avoid extensive psychophysical user testing, we instead use an automated visual difference predictor (VDP), [13] which has been extended for application to high-dynamic range images [14]. The VDP algorithm provides a metric to distinguish the subset of differences between two images that a standard human observer would be able to detect. The algorithm filters the images through various stages that mimic the light scattering in the cornea, lens, and retina, and our non-linear response to luminance and contrast. As a result, many physical differences between images are not detectable by a human observer.
Fig. 4. Output from the Visual Difference Prediction algorithm of the simulated projector image compared to the original image, scaled to the same brightness. Small areas marked in black are predicted to be distinguishable if both images were to be compared side by side. These areas total only 0.03% of the total pixels in the image.
Figure 4 shows the result from the VDP algorithm after comparing the simulated AMA projector image Idmd*Pim to that of an idealized projector with sufficient dynamic range and brightness to perfectly display the image at this luminance level (or the original image, scaled to identical brightness). The VDP implementation produces a comparison map in which the areas are marked that are predicted to be noticed by a human observer as difference when comparing the two images side by side. For this particular example, only 0.03% of the pixels are estimated to be detectable as having a perceivable difference with a probability of over 75%, and none with a probability of over 95%.
8. Conclusions
We have shown through simulations that the dynamic range and peak brightness of conventional projectors can be improved by employing a secondary MEMS mirror array. This additional light modulator diverts light away from the dark regions where it reduces the overall contrast, and to the bright parts of the image, where the light can be used to increase the brightness of the picture. While the algorithm used to determine the mirror locations does not necessarily find their optimal locations, it can find solutions that improve the overall brightness of the image significantly.
Recently, a class of “pocket projectors” has been developed and marketed for mobile applications. The projectors typically use LEDs, which can easily be dimmed, and they are often equipped with batteries, which limits their time of operation. Instead of using the AMA mirrors to increase the peak brightness, they could be used to reduce the power consumption of such a projector to 1/3 while retaining the original picture brightness.
The net benefit of using the AMA will vary depending on the image content, but also opens up many intriguing areas of study for adaptive image rendering. It is possible to use the AMA at a conservative level that always maintains perfect image reconstruction, while providing some boost in brightness, dynamic range, and black level. However, there are instances where we might want to increase brightness beyond this conservative value, even at the expense of some of the image detail. As demonstrated in [15, 16], there are many situations in which ambient illumination makes much of the detail present in an image at a given luminance imperceptible. This is especially true with projectors, which are often set up in rooms with uncontrolled lighting. In these cases, it may be that aggressively increasing the brightness of key parts of the image past the point at which low-luminance detail is lost, may still improve perceived image quality.
References and links
1. D. Dewald, D. Segler, and S. Penn, “Advances in contrast enhancement for DLP projection displays,” J. Soc. Inf. Disp. 11, 177–181 (2004). [CrossRef]
2. K. E. Peterson, “Silicon as a Mechanical Material,” Proceedings of the IEEE 70(5), 420–457 (1982). [CrossRef]
3. I. Jung, U. Krishnamoorthy, and O. Solgaard, “High fill-factor two-axis gimbaled tip-tilt-piston micromirror array actuated by self-aligned vertical electrostatic combdrives,” J. Microelectromechanical Syst. 15(3),563–571 (2006) [CrossRef]
4. H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, L. Whitehead, M. Trentacoste, A. Ghosh, and A. Vorozcovs. “High dynamic range display systems,” ACM Transactions on Graphics , 23(3), 760–768, (2004). [CrossRef]
5. M. Trentacoste, W. Heidrich, L. Whitehead, H. Seetzen, and G. Ward, “Photometric image processing for high dynamic range displays,” J. Visual Commun. Image Represent. 18(5), 439–451 (2007). [CrossRef]
6. A. Pavlovych and W. Stuerzlinger, “A High-Dynamic Range Projection System,” Progress in biomedical optics and imaging 6(39) (2005).
7. G. Damberg, H. Seetzen, G. Ward, W. Heidrich, and L. Whitehead, “High Dynamic Range Projection Systems,” SID Symposium Digest of Technical Papers 38(1), 4–7 (2007). [CrossRef]
8. P. G. J. Barten, “Physical model for the contrast sensitivity of the human eye,” Proc. of SPIE 1666, 57–72 (1992). [CrossRef]
9. P. Burt and E. Adelson, “A multiresolution spline with application to image mosaics,” ACM Transactions on Graphics 2(4), 217–236 (1983). [CrossRef]
10. E. P. Simoncelli. “matlabPyrTools” http://www.cns.nyu.edu/~lcv/software.html. Accessed 02/22/07.
11. P. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” Proceedings of the 24th annual conference on Computer graphics and interactive techniques, 369–378 (1997).
12. E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics) (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2005).
13. S. J. Daly, “Visible differences predictor: an algorithm for the assessment of image fidelity,” Proc. SPIE 1666, 2–15 (1992). [CrossRef]
14. R. Mantiuk, S. Daly, K. Myszkowski, and H.-P. Seidel, “Predicting Visible Differences in High Dynamic Range Images - Model and its Calibration,” in Human Vision and Electronic Imaging X, IS&T/SPIE’s 17th Annual Symposium on Electronic Imaging (2005), B. E. Rogowitz, T. N. Pappas, and S. J. Daly, eds., vol. 5666, pp. 204–214 (2005).
15. K. Devlin, A. Chalmers, and E. Reinhard, “Visual calibration and correction for ambient illumination,” ACM Transactions on Applied Perception (TAP) 3(4), 429–452 (2006).
16. R. Heckaman, “Effect of DLP projector white channel on perceptual gamut,” Journal of the SID 14(9), 755–761 (2006).
