Open Access
Add to BibSonomy
Abstract
Local dimming techniques have been widely studied to achieve a high contrast ratio and low power consumption for liquid crystal displays. The luminance of a backlight is reduced according to some characteristics of an input image and the pixel data are boosted to compensate for the dimmed backlight. In addition, because a backlight block is affected by adjacent ones, the pixel compensation algorithm requires huge processing power as well as many iterations along with the overall luminance profile information of a backlight. However, a proposed deep-learning-based local dimming algorithm generates the compensated image directly from an input image without any information of backlight’s dimming levels. The proposed compensation network is constructed on the basis of the U-net to maintain the high-resolution features in the up-sampling paths through skip-connections. In addition, it is also ensured that the bi-linear interpolation can be used without visible image quality degradation for the reduction on the number of parameters. The proposed networks are trained and verified on a DIV2K 2K image dataset.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Whereas organic light emitting diode (OLED) displays become the mainstream in small-size displays of smartwatches, smartphones, and tablet computers, liquid crystal displays (LCDs) still are adopted in large-size applications such as laptop computers, monitors, and televisions (TVs), due to their low price and high reliability. OLED displays produce no light at black pixels except for the reflected one, which can lead to the substantial contrast ratio (CR) improvement and power consumption reduction for low gray images [1, 2]. However, LCDs cannot avoid the low CR caused by some light leakage and some constant amount of power consumption owing to the always-on backlight unit (BLU) of the fixed luminance. To address these issues for LCDs, Local dimming techniques have become an essential solution [3–6]. The overall process of the conventional local dimming LCD is illustrated in Fig. 1. Basically, when an input image is given from an external image source, a local dimming system adjusts each pixel data of a panel as well as the light intensity of a BLU separately, while a conventional LCD without dimming functionality uses the constant light intensity BLU. In the local dimming LCD, an input image is divided into small regions that correspond to BLU sub-blocks and the dimming level of each BLU sub-block is decided according to the maximum or average pixel value of each image region. Whereas, the pixel data of the panel should be increased locally to compensate for the reduced intensity of the BLU. Ideally, the combination of locally dimmed BLU and compensated panel image allows the original image to be displayed with lower power consumption. In addition, the local dimming LCD can achieve also higher CR due to the reduced black luminance [7].
Local dimming LCDs contain trade-offs between power saving and some visible artifacts such as halo, clipping, block artifact, color distortion, dimming flicker, and color gamut transformation [8–10]. A weighted roll-off dimming scheme alleviates block artifacts by applying simple bi-linear interpolation to the pixel compensation algorithm without any modification of dimming levels [11]. However, this simple interpolation causes serious color distortion for very low dimmed regions, leading to the limitation on the power reduction.
On the other hand, quantum-dot (QD) films have been recently adopted in a BLU to provide wide color gamut [12, 13]. But, this QD-BLU brings about a difficult problem that color components have different light spreading functions (LSFs) to be addressed separately unlike existing white light emitting diode (WLED) BLUs of only one spreading characteristic. In practice, the pixel compensation algorithm depends on the characteristics of a given BLU, that is, if a different BLU is used, the pixel compensation must be modified accordingly. Therefore, the pixel compensation algorithm dedicated to the QD-BLU should be come up with.
In this paper, we propose a deep-learning-based [14, 15] local dimming algorithm for QD-BLU LCDs. The neural network can address a variety of BLUs including QD-BLUs by changing its parameters without any hardware modification. The proposed network is named as a local dimming neural network (LDNN) and it can be placed at a TV-set side or an LCD-module side as shown in Fig. 2. Since the pixel compensation algorithm of the LDNN can cover the variation of BLU characteristics, the dimming levels are obtained on the basis of one specific BLU (ex. an ideal local dimming BLU without light spreading) regardless of the actual BLU characteristics. While TV-set makers need to have all parameters over their LCD-module vendors for the case of an LDNN integrated TV-set as depicted in Fig. 2(a), they don’t need to care about BLU characteristics for the case of an LDNN integrated LCD-module as shown in Fig. 2(b). Consequently, it would be a good solution for TV-makers to integrate the proposed LDNN in LCD-modules with their own parameters.
Fig. 2 Overall architecture of the proposed local dimming system. The proposed LDNN can be implemented (a) in a TV-set side or (b) in an LCD-module side.
The resultant LDNN gives rise to the compensated image directly from the input one without dimming level information.
2. Proposed deep-learning-based pixel compensation network
A proposed LDNN consists of down-sampling and up-sampling paths that include many convolution layers. The LDNN needs to extract dimming levels of BLU sub-blocks only from an input image as well as to make up the high-resolution compensated image. The extraction of dimming levels requires the low-resolution information to cover the large receptive field due to broad LSFs of BLU sub-blocks. Whereas, the high-resolution image construction should be conducted with the high-resolution information.
To satisfy these contrary conditions, the U-net [16] is adopted as the basic architecture of the proposed LDNN as shown in Fig. 3. The U-net with a shape similar to a sand-glass reduces the spatial resolutions of layers as it goes deeper up to an intermediate layer to enable the large receptive field and, after that, increases them again to give rise to the high-resolution image like the input one. To transfer the high-resolution information of the down-sampling part to the up-sampling part, skip-connection paths are implemented [17, 18]. In Fig. 3, there are two types of skip-connection paths represented in blue and black lines, respectively. While the data through blue skip-connection paths are concatenated as the additional channels of the corresponding up-sampling layers, the data through black paths are added to the deeper layers. These additions via a skip-connection allow forward neural networks to learn about the residual data that contain high sparsity leading to fast and stable convergence.
Fig. 3 The proposed LDNN architecture with a sandglass shape. The upper blue paths denote the skip-connections that concatenate the data of convolution layers to the up-sampling layers. On the other hand, the lower black arrows represent the skip-connections leading to addition. Layers marked in blue are strided convolution layers for down-sampling and layers marked in green are strided transposed convolution layers for up-sampling, respectively. Other convolution layers are implemented with the stride of 1. The numbers above each layer denote the spatial resolution of each convolution layer.
The target resolution of the input image is full-HD (1920 × 1080) and the number of dimming sub-blocks is 16 × 9, which means that each sub-block covers 120 × 120 pixels. Thus, the feature layer of 16 × 9 that is the same as the resolution of the BLU is achieved through down-sampling convolution layers with strides of 5, 4, 3, and 2 that are presented as blue boxes in Fig. 3, instead of pooling and interpolation. This strided convolution layer outperforms pooling and interpolation methods. In addition, red, green, and blue color components are separately processed, contrary to most existing local dimming systems which deal with only intensity components. The resultant receptive field has the size of 1004 × 1004 pixels that is equal to about 8 × 8 BLU sub-blocks. For up-sampling layers that are presented as green boxes in Fig. 3, transposed convolution layers known as deconvolution layers [19] are in use with strides of 2, 3, 4, and 5 that are in the reverse order of convolution layers. The filter sizes of up-sampling layers are 3 × 3, 5 × 5, 7 × 7, and 9 × 9, respectively. The number of channels of all layers except for input, output, and concatenated layers components is set to 64.
Convolution layers between down-sampling layers and up-sampling layers are implemented as a residual network with a skip-connection of addition as marked with red dot-line boxes of Fig. 3 to cope with gradient vanishing and exploding problems of the deep neural network training [20]. As depicted in Fig. 4(a), the skip-connection from the input to the output in a residual block (RB) can alleviate the issues. When the RB has the different number of channels in the input layer from the output layer, a convolution layer with 1 × 1 filter is inserted in a skip-connection path as presented in Fig. 4(b).
Fig. 4 Residual block (a) with a direct skip-connection (b) with a skip-connection through a 1 × 1 convolution layer.
The low-resolution information is needed to extract the dimming levels for an input image and the LSF of a BLU sub-block has a Gaussian shape in general. Therefore, it is expected that the high-resolution information for the overall BLU profile can be built with a bi-linear interpolation instead of an up-sampling layer using a transposed convolution, leading to the substantial reduction on the number of parameters. The spatial resolution of each channel is increased by a 2-dimensional interpolation as illustrated in Fig. 5. This simple LDNN (sLDNN) with bi-linear interpolation layers that are marked in yellow is described in Fig. 6. Unlike the LDNN, the convolution layers between interpolation (up-sampling) layers in the sLDNN are implemented with RBs with 1 × 1 convolution layer to deal with the increased number of channels by the skip-connection of concatenation, where the high-resolution information from the down-sampling path is concatenated to each corresponding interpolation output layer. The comparison of LDNN and sLDNN architectures is summarized in Table 1 showing that the number of parameters in the sLDNN are substantially smaller by 56.2 % than in the LDNN.
Fig. 6 sLDNN architecture. The yellow boxes denote bi-linear interpolation. Because the concatenation increases the number of channels in the up-sampling network and interpolations increase only the spatial resolution of each channel, RBs with 1 × 1 convolution layer (RB(b)) are used, while RBs with direct skip-connections (RB(a)) are adopted in the down-sampling network.
Table 1. The comparison of LDNN and sLDNN.
3. Training
The overall training process is illustrated in Fig. 7. The training process requires two paths for compensated image (ICP) and dimmed BLU image (LD), respectively. The input image (IIN) is also used as the ground truth image that is compared to the output image () estimated by the element-wise product of ICP and LD. The LDNN is updated to minimize the difference between and IIN that is defined as a loss. LD is made up from the LSFs of a BLU sub-block that is modelled as the sum of two Gaussian functions as expressed in Eq. (1). This LSF model is established based on the captured image when only one BLU sub-block is fully turned on. (mx, my) denotes the center pixel position and (x, y) is the pixel position on horizontal and vertical axes. The profile model is adjusted to match the measured one of the QD-BLU. As a result, red and green models are established with a, σ1, and σ2 of 0.8, 110, and 230, respectively, and the blue model is constructed with 0,9, 100, and 230. The resultant LSF models are illustrated in Fig. 8.
Fig. 8 LSF models (a) for red/green LED sub-blocks and (b) for blue LED sub-blocks. The horizontal axis is normalized by the size of a BLU sub-block.
The training and test images are prepared with DIV2K [21] that is a 2K resolution image dataset commonly used for image restoration tasks. The dataset consists of 800 training images, 100 validation images, and 100 test images. Because high-resolution ground truths for test images are not released, validation images are utilized as test images for the LDNN evaluation. All images are resized to the 1920 × 1080 resolution and the number of training images is augmented to 2,400 by horizontal and vertical flips. LD for each input image is calculated from maximum gray levels in the image regions corresponding to 9 × 16 BLU sub-blocks. As the loss function (L), mean squared errors are in use to measure the difference of IIN and as described in Eq. (2). The division by 3 in the last part represents three red, green, and blue color components.
Because the LDNN is a fully convolutional network (FCN), arbitrary-sized images are allowed to be handled when the matched BLU profile can be provided. The parameters of the proposed network are initialized using a Xavier method [22] and an ADAM optimizer [23] is adopted with setting , , and . The training rate is initialized as and reduced to 20 % every 10 epochs. Our networks are evaluated with the TensorFlow framework [24] and NVIDIA Titan X graphics processing unit (GPU). The network training takes about 35 hours and 22 hours for convergence with respect to LDNN and sLDNN, respectively. The smaller number of weights leads to the faster convergence in sLDNN.
4. Test results
Our proposed local dimming system of the LDNN trained with training images is tested over 100 validation images of the DIV2K dataset. Test results of LD, ICP, and for three input images (IIN) are shown in Fig. 9. It is ensured that the LDNN successfully gives rise to compensated images only from input images without any information about BLU dimming levels.
To evaluate the performance of the proposed networks, LDNN and sLDNN, peak signal-to-noise ratio (PSNR), structural similarity (SSIM) index [25], and color difference (CD) are compared with the previous weighted pixel compensation (WPC) dimming method [11]. CD is defined as the Euclidean distance of and components between IIN and in the CIELAB space [26].
Fig. 10 Performance evaluation results of WPC [11], LDNN, and sLDNN. Red box regions are separately shown for the clearer comparison of algorithms. Power is the BLU power consumption ratio for the fully turned-on BLU power and GT stands for ground truth.
As presented in Fig. 10, LDNN and sLDNN exhibit significant improvement results in all the performance metrics of PSNR, SSIM, and CD over the WPC algorithm. Because all three algorithms generate only compensated panel images without any modification of dimmed BLU profiles, their BLU power consumption values are same. While the WPC can cope with block artifacts well for low CR images, it cannot avoid the over-compensation over low gray image regions because the light spreading effects from adjacent BLU sub-blocks are not considered. Consequently, the boundaries between low and high gray regions are observed with too much boosted luminance. However, both proposed LDNN and sLDNN do not show any perceivable artifacts even without any information about dimmed BLU profiles. The average performance results over all test images are summarized in Table 2, showing that the LDNN achieves the best performance and the sLDNN with the smaller number of parameters is good enough. sLDNN on the GPU and TensorFlow platform spends about 100 ms for a full-HD image that is equivalent to the frame rate of 10 Hz, while LDNN takes 124 ms. Although even sLDNN is too slow to apply to display systems, we are very sure that the dedicated hardware implementation enables the higher frame rate of 60 Hz [27–30]. Furthermore, the processing time difference of LDNN and sLDNN will be widened in the hardware implementation since sLDNN requires much smaller memory bandwidth for weights than LDNN.
Table 2. Evaluation summary over test images.
The proposed network has been also evaluated for the different segmentation number of the BLU, especially the larger number of 36 × 48 than 16 × 9 that enables higher CR and lower power consumption [31]. In this setting, a BLU sub-block has a rectangular shape but not a square one. For the DIV2K validation dataset, the average performance values of LDNN and sLDNN are shown in Table 3. Although the evaluation results are slightly degraded compared to the 9 × 16 segmentation BLU, LDNN and sLDNN of 36 × 48 BLU sub-blocks still achieve good pixel compensation over all the metrics. As mentioned above, the larger segmentation leads to more reduction on the power consumption.
Table 3. Evaluation summary of LDNN and sLDNN at 36×48 BLU sub-blocks.
Fig. 11 Output image examples of sLDNN networks with skip-connections and without skip-connections on test images. Both networks are separately trained with same training images.
Table 4. Average performance comparison of sLDNN networks with and without skip-connections on test images.
To verify that skip-connection paths are required to make up the high-resolution compensated image and the main deep-network path is needed to extract the dimmed BLU profile, we run training and test processes again for the sLDNN without skip-connections and compare its performance with the sLDNN with skip-connections. Some output image examples of sLDNN networks with and without skip-connections are shown in Fig. 11 and the comparison results for average performance values are presented in Table 4. As expected, the sLDNN without skip-connections can achieve only the blurred image of the dimming-applied BLU profile, leading to the conclusion that the skip-connections transfer the high-resolution information for the compensated image generation from the down-sampling path to the up-sampling path.
5. Conclusion
In this paper, we demonstrate a local dimming algorithm based on deep neural networks that enables the compensation of pixel data transferred to a panel without any information about dimming levels of BLU sub-blocks. To maintain high-resolution features for high-resolution compensated image generation as well as low-resolution features for LSFs of BLU sub-blocks, an U-net architecture is adopted with skip-connections of concatenation and addition. The proposed LDNN successfully generates compensated images even in a local dimming system using a QD-BLU with different LSFs for red, green, and blue colors. Furthermore, the evaluation results of the sLDNN lead to the conclusion that the up-sampling layers can be replaced with simple bi-linear interpolation in local dimming applications without serious performance degradation.
Funding
Samsung Display Co., Ltd.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
References
1. B. Geffroy, P. le Roy, and C. Prat, “Organic light-emitting diode (OLED) technology: materials, devices and display technologies,” Polym. Int. 55 (6), 572–582 (2006). [CrossRef]
2. K. Müllen and U. Scherf, Organic light emitting devices: synthesis, properties and applications(John Wiley & Sons, 2006).
3. H. Cho and O. Kwon, “A local dimming algorithm for low power LCD TVs using edge-type LED backlight,” IEEE Trans. Consum. Electron. 56 (4), 2054–2060 (2010). [CrossRef]
4. H. Chen, J. Sung, T. Ha, and Y. Park, “Locally pixel-compensated backlight dimming on LED-backlit LCD TV,” J. Soc. Inf. Disp. 15 (12), 981–988 (2007). [CrossRef]
5. D. M. Hoffman, N. N. Stepien, and W. Xiong, “The importance of native panel contrast and local dimming density on perceived image quality of high dynamic range displays,” J. Soc. Inf. Disp. 24 (4), 216–228 (2016). [CrossRef]
6. G. Tan, Y. Huang, M.-C. Li, S.-L. Lee, and S.-T. Wu, “High dynamic range liquid crystal displays with a mini-led backlight,” Opt. Express 26 (13), 16572–16584 (2018). [CrossRef] [PubMed]
7. H. Nam and E.-J. Song, “Low color distortion adaptive dimming scheme for power efficient LCDs,” Opt. & Laser Technol. 48, 52–59 (2013). [CrossRef]
8. H. Chen, T. H. Ha, J. H. Sung, H. R. Kim, and B. H. Han, “Evaluation of LCD local-dimming-backlight system,” J. Soc. Inf. Disp. 18 (1), 57–65 (2010). [CrossRef]
9. H. Cho and O.-K. Kwon, “A backlight dimming algorithm for low power and high image quality LCD applications,” IEEE Trans. Consum. Electron. 55 (2), 839–844 (2009). [CrossRef]
10. W. Huang, J.-M. Li, L.-M. Yang, Z.-L. Jin, Z.-G. Zhong, Y. Liu, Q.-Y. Chou, and F. Li, “Local dimming algorithm and color gamut calibration for RGB LED backlight LCD display,” Opt. & Laser Technol. 43 (1), 214–217 (2011). [CrossRef]
11. S.-K. Kim, S.-J. Song, and H. Nam, “Bilinear weighting and threshold scheme for low-power two-dimensional local dimming liquid crystal displays without block artifacts,” Opt. Eng. 53 (6), 063110 (2014). [CrossRef]
12. E. Jang, S. Jun, H. Jang, J. Lim, B. Kim, and Y. Kim, “White-light-emitting diodes with quantum dot color converters for display backlights,” Adv. Mater. 22 (28), 3076–3080 (2010). [CrossRef] [PubMed]
13. Z. Luo, Y. Chen, and S.-T. Wu, “Wide color gamut LCD with a quantum dot backlight,” Opt. Express 21, 26269–26284 (2013). [CrossRef] [PubMed]
14. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in), Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds. (Curran Associates, Inc., 2012), pp. 1097–1105.
15. I. Goodfellow, Y. Bengio, and A. Courville, Deep learning (MIT, 2016).
16. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.
17. J. Yamanaka, S. Kuwashima, E. D. Kurita, Takio, S. Xie, Y. Li, D. Zhao, and E.-S. M. El-Alfy, “Fast and accurate image super resolution by deep CNN with skip connection and network in network,” in Neural Information Processing, (Springer, 2017), pp. 217–225.
18. A. E. Orhan and X. Pitkow, “Skip connections eliminate singularities,” arXiv 1701.09175 (2017).
19. R. Fergus, M. D. Zeiler, G. W. Taylor, and D. Krishnan, “Deconvolutional networks,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2010), pp. 2528–2535.
20. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2016), pp. 770–778.
21. E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, (IEEE, 2017).
22. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feed forward neural networks,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, (AISTATS,2010), pp. 249–256.
23. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference for Learning Representation, (2015).
24. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation, (USENIX, 2016), pp. 265–283.
25. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13 (4), 600–612 (2004). [CrossRef] [PubMed]
26. W. G. Backhaus, R. Kliegl, and J. S. Werner, Color vision: Perspectives from different disciplines(Walter de Gruyter, 2011).
27. C. Wang, L. Gong, Q. Yu, X. Li, Y. Xie, and X. Zhou, “Dlau: A scalable deep learning accelerator unit on fpga,” IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 36 (3), 513–517 (2017).
28. Y. Kim, J.-S. Choi, and M. Kim, “A real-time convolutional neural network for super-resolution on fpga with applications to 4k uhd 60 fps video services,” IEEE Trans. Circuits Syst. Video Technol. (to be published).
29. J.-W. Chang, K.-W. Kang, and S.-J. Kang, “An energy-efficient fpga-based deconvolutional neural networks accelerator for single image super-resolution,” IEEE Trans. Circuits Syst. Video Technol. (to be published).
30. J. Lee, C. Kim, S. Kang, D. Shin, S. Kim, and H.-J. Yoo, “Unpu: An energy-efficient deep neural network accelerator with fully variable weight bit precision,” IEEE J. Solid-State Circuits 54 (1), 173–185 (2019). [CrossRef]
31. T. Shiga, S. Shimizukawa, and S. Mikoshiba, “Power savings and enhancement of gray-scale capability of lcd tvs with an adaptive dimming technique,” J. Soc. Inf. Disp. 16 (2), 311–316 (2008). [CrossRef]
