Open Access
Add to BibSonomy
Abstract
Visible light communication based on display and camera can simultaneously provide the functionalities of display and invisible information transmission. There are widely potential applications in information broadcasting, stealth advertising, etc. In this paper, a real-time display camera communication system based on LED displays and smartphones is proposed. To solve the rate mismatch problem and the symbol unsynchronization problem between the LED display panel with a relatively high frame rate and the smartphones’ camera with a relatively low frame rate, we propose an Alternate Bit-flipping Repeat Coding scheme for adjustable frame rate matching and symbol synchronization with the advantage of eliminating flickers perceivable to human eyes. Fast image processing algorithms are also designed to decrease the computational complexity of extracting information from the captured video frames in real-time. An experimental system was implemented on a 16×16 points LED display panel with a refresh rate of 150 fps and a commercial Android smartphone with an 8 megapixels camera working at 30 fps and a self-developed Android APP. The experimental results show that the real-time LED display camera communication system can achieve a data transmission rate of 30 bps for one LED display point and a maximum distance of 80 cm, and the computational time on the Android smartphone for data receiving is only 11.827 ms.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
In recent years, LED displays are widely used in advertising media, sports stadiums, stage backgrounds, municipal engineering, etc [1]. Visible light communication (VLC) based on LED has received extensive attention due to its rich spectrum resources, high transmission rate, and strong anti-electromagnetic interference capabilities [2]. Using high-speed signals that exceed the human eye's perceptible frequency to modulate the LED display panel can realize invisible information transmission without affecting the display, which has the outstanding advantage of the dual functionality of display and communication [3]. Smartphone with built-in camera provides an ideal platform to implement LED display camera communication (LED-DCC) system. Nowadays, even low-end smartphones are equipped with high-resolution complementary metal-oxide-semiconductor (CMOS) sensor cameras. As a photoelectric sensor, the CMOS camera can be used to conveniently receive the VLC signal emitted by the LED display panel to construct a LED-DCC system and is easy to integrate with other mobile Internet applications.
However, in practical applications, LED display panels have to maintain a relatively high refresh rate to ensure the quality of the displayed images/videos, while the smartphones’ CMOS camera usually works at a relatively low capture frame rate, typically 30 fps. Therefore, there is a rate mismatch problem between the transmitter and the receiver in the LED-DCC systems. Moreover, there is another problem, symbol unsynchronization between the LED display panel with line-by-line display mode and the CMOS image sensor with a rolling shutter effect. These two problems will result in a series of problems such as signal demodulation, inter-symbol crosstalk, and decoding errors, which may seriously deteriorate the performance of the LED-DCC system.
Several DCC systems have been proposed before [4–10]. In 2004, Hranilovic and Kschischang presented a point-to-point, multiple-input/multiple output (MIMO) optical channel using a 512×512 pixel LCD panel and 154×154 pixels of a CCD camera [4]. In 2013, Hu et al. proposed LightSync [5] which allows smooth communication between the LCD screen and the camera at any combination of the transmit and receive frame rates, as long as the receive rate is at least half the transmit rate. InFrame++ [6] leverages the spatial-temporal flicker-fusion property of human vision system and the fast frame rate of modern display. It multiplexes data onto full-frame video contents through novel complementary frame composition, hierarchical frame structure, and CDMA-like modulation. It is able to achieve 150-240 kbps at 120FPS over a 24’ LCD monitor with one data frame per 12 display frames. SBVLC [7] proposed in 2016 addresses geometric models and physical security enhancement mechanisms for barcode communications. In 2016, DisCo [8] is proposed and based on a rolling shutter camera to execute the decoding by translating the temporal sequence into a spatial pattern. In 2019, Chen et al. proposed a video-based robust and unobtrusive code [9], leveraging several important properties of the human vision system and considering the single-frame and multi-frame formats, are proposed for achieving an unobtrusive display-to-camera communication with a high data rate. In 2020, Jung et al. proposed an optical camera communication system based on complementary color barcode [10]. However, all of them use LCD as signal transmitter and the data transmission rate is relatively low due to the limitation of LCD’s modulation rate. Moreover, the low modulation rate may cause human-eye noticeable flickers and these systems have to use complicated algorithms for data insertion which significantly reduces the practicability of the DCC system.
In this paper, we propose a real-time LED-DCC system based on LED display panels and smartphones. An Alternate Bit-flipping Repeat Coding (ABRC) scheme is proposed to enable adjustable frame rate matching between the LED-DCC display point being modulated to transmit VLC signals and CMOS cameras capturing VLC signals for demodulation. Thus, it supports the capture and extraction of the communication signal of the high display refresh rate LED display panel by the smartphone’s camera with a low capture frame rate. Meanwhile, we propose fast image processing algorithms for data receiving to support real-time LED-DCC.
The demo system is implemented on a 16×16 points LED display panel with a refresh rate of 150 fps and a commercial Android smartphone with a 3840×2160 megapixels camera working at 30 fps. Experimental results show that the real-time LED-DCC system can achieve a transmission rate of 30 bps for one LED display point with a maximum distance of 80 cm, and the computational time on the Android smartphone for data receiving is only about 11.827 ms.
The remainder of this paper is organized as follows. Section 2 introduces the proposed real-time LED-DCC system including the proposed ABRC scheme and the fast image processing algorithms. The experimental results are detailed in Section 3. Section 4 concludes the paper.
2. Proposed LED-DCC system
As shown in Fig. 1, the proposed LED-DCC system mainly includes two sub-systems, the transmitter sub-system and the receiver sub-system.
In the transmitter sub-system, hidden data to be transmitted is first encoded by the proposed ABRC scheme to increase the reliability and adaptability of data transmission. Then, via the data insertion process, as shown in Fig. 2, each data bit (bi) in the encoded data frame is combined with one pixel (pj(x,y)) located at the coordinates (x, y) in each image frame (framej) of the video data to be displayed. Therefore, the data to be transmitted is embedded and hidden in the video content data. Finally, the combined data is fed into the modulator to drive the LED display to show the video content in a high frame rate, says 150 fps. At meanwhile, the hidden data is also broadcasted by the LED display point (LEDj(x,y)) in a baud rate same as the frame rate, which can avoid flicker perceived by human eyes. Note that the capacity of data transmission can be multiplied since a video frame is a 2-D pixel array and each pixel can be used as one independent channel for data transmission.
In the receiver sub-system, the CMOS camera of a smartphone is used as an opto-electronic sensor to capture the brightness signal of the LED display as video frames, i.e., a series of 2-D pixel array values. Therefore, to extract the hidden bit information from the video frames, fast image processing algorithms are proposed to guarantee the real-time performance of data receiving. First, a fast ROI detection algorithm is deployed to precisely locate the area of the LED display in the captured image frames and the image coordinates of the LED display point for data transmission. Then, the pixel values around the LED display point are demodulated as “0/1” symbols using an adaptive binarization technique. Finally, the demodulated symbols are decoded by the proposed ABRC scheme to recover the hidden data.
In this LED-DCC system, two key techniques are proposed, i.e., the proposed ABRC scheme for asynchronous data transmission and the fast image processing algorithms for real-time LED-DCC data receiving. These two techniques are described in detail as follows.
2.1 Proposed ABRC scheme for rate adaption and symbol synchronization
As described in Section 1, a LED-DCC system requires the LED display to maintain a relatively high display frame rate to ensure the stability of video/image display and avoid flicker perceivable to human eyes. However, the CMOS camera of smartphones usually works at a relatively low capture frame rate, typically 30 fps. Therefore, inspired by orthogonal spread-spectrum coding, the ABRC scheme for asynchronous data transmission is proposed to solve the problems of data rate mismatch and symbol unsynchronization. The original information bit is replicated in an alternate bit-flipping mode multiple times. The replication is used for adjustable frame rate matching, while the alternate flipped symbols provide effective calibration functionality for frame synchronization. The detailed encoding/decoding steps are as follows.
- (1) Given the refresh frame rate of a LED display is fpsled = 150fps and the capture frame rate of a camera is fpscam = 30fps, the replication rate of the ABRC scheme is set as 5 for rate adaptation.
- (2) For each original data bit (dk), it is encoded as a spread-spectrum code chip (cpk) according to its value, “01010” for dk = 0, and “10101” for dk = 1, respectively. Taking an original bit string “0110” as an example, the encoded codeword is “01010 10101 10101 01010”.
From the perspective of data transmitters and LED displays, this design has an important advantage. When the codeword is combined with video data and the combined data is used to drive LED display, the length of runs of symbols (i.e., 1 s or 0 s) represents the temporal intensity over a shorter period, which may exceed the persistence of human eye and causes noticeable brightness change, i.e., flicker. Therefore, the run-length of codewords should be carefully limited to mitigate any potential flicker resulting from modulating light sources for communication. The ARBC scheme can break long runs of the same symbols (i.e., 1 s or 0 s) and stable the LED display.
- (3) For frame synchronization, the original data bits are prefixed by a Start Frame Delimiter (SFD), which is a specific bit string “00000010”, and suffixed by an End Frame Delimiter (EFD), which is a specific bit string “00011001”, as shown in Fig. 3. The SFD and the EFD are used to indicate the start and the stop of receiving the data frame. Note that the SFD and the EFD are also encoded using step (2) before it is sent out.
- (4) At the receiver’s side, since the smartphone’s camera has a relatively low receiving rate, it can only periodically sample several symbols instead of each symbol in the transmitted codeword. Therefore, as shown in Table 1, the decoding phase may have 4 possible cases according to the symbols received.
- (a) The receiver has well symbol synchronization with the transmitter. Therefore, the sampled symbols have the same values as the original bits, i.e., “0110”, as shown in the first sub-column in the third column in Table 1. The received data can be directly extracted.
- (b) The receiver has one symbol period delayed with the transmitter. In this case, the sampled symbols still have the same values as the original bits, i.e., “0110”, as shown in the first sub-column in the fourth column in Table 1. The received data can be directly extracted, while the final data extraction has a delay of one camera frame time, i.e., 1/30 second.
- (c) The receiver has one symbol period in advance of the transmitter. In this case, the sampled symbols have the inverse values with the original bits, i.e., “1001”, as shown in the first sub-column in the fifth column in Table 1. Flipping the sampled symbols can correctly recover the original data bits in a simple way.
Note that for case (a), (b), and (c), the receiver may receive two possible bit values, original or inverse value. In our scheme, the SFD is designed not only as a delimiter, but also as an indicator for symbol synchronization and bit flipping. Since the SFD is encoded in the same way as data bits, once the original SFD is detected in the received symbol series, it means case (a) or (b) occurs and no flipping operation is required. Otherwise, if the inverse SFD (“11111101”) is detected, it means case (c) exists and all the received symbols should be flipped to recover the correct data.
- (d) For the cases that the receiver has more than one symbol period delayed or advanced with the transmitter, the handling is similar to that for cases (b) and (c).
Table 1. Encoding and decoding of the proposed ABRC scheme
2.2 Fast image processing algorithms for real-time LED-DCC data receiving
In the LED-DCC system, the hardware device for signal capturing and data receiving is the CMOS camera of smartphones. As the hardware configuration of smartphones’ cameras has been improved significantly in recent years, the captured image in our system usually has a high resolution, e.g. 3840×2160 pixels. Higher resolution can provide more pixels for opto-electronic imaging and thus increase the reliability and distance of data transmission. However, higher computational complexity is also required, which will cause a longer processing time delay. The typical capture frame rate of smartphones’ cameras is 30 fps. It means that one video frame will be captured and available every 1/30 second, and our image processing algorithms should be fast enough to finish a series of processing for an available video frame in 1/30 second to support real-time data receiving. Therefore, efficient algorithms are desired for LED-DCC data receiving.
As shown in Fig. 4, the image processing for LED-DCC data receiving mainly includes two steps, finding out the location of the LED-DCC point (LEDj(x,y)) in captured images and converting the captured pixel values in captured images as “0/1” symbols (bri) to reconstruct a received codeword. In this paper, a fast ROI detection algorithm is proposed to precisely locate the image coordinates of the LED-DCC point for data transmission and an adaptive binarization method to demodulate the pixel values around the LED-DCC point as “0/1” symbols. Considering the fact that the camera used for capturing LED-DCC signals is set to work at an underexposure mode with a fast exposure time and a small aperture, we can use a simplified contour detection and image segmentation algorithm to extract the LED-DCC point in the captured images, instead of using conventional computation-intensive contour-finding algorithms.
Figure 5(a) shows that the pixel values of most regions in the captured image are zero (dark) except for the area of LED display. It is worth noting that the sum of the pixel values in each row (column) is near zero except for those with LED display areas. By integrating the pixel values vertically, we can easily locate the left and right boundaries of a LED display area. Similarly, the range of a LED display area on the vertical axis can also be obtained by integrating the pixel values horizontally. The pixel coordinates of the top left and bottom right corners are therefore obtained, which identify the rectangle area that contains a LED display area, as marked with a red rectangle in Fig. 5(b). Once the LED display area in the captured image is located, the image coordinates of the LED-DCC point can be quickly detected with a relative position offset, as shown in Fig. 5(c). Finally, to extract the symbols from the pixel values of the LED-DCC point, an adaptive binarization method based on Sauvola [11] is utilized to decide the binarization threshold based on the pixel values around the LED-DCC point. Then, the value of the symbol of the LED-DCC point can be determined by binarizing the pixel value on the LED-DCC point.
Fig. 5. Processing of LED-DCC point detection. (a) Captured image. (b) LED display area detection. (c) LED-DCC point area detection.
3. Experimental results
To examine the performance of the proposed LED-DCC system, an experimental system was implemented on a 16×16 points LED display panel and a commercial Android smartphone with a self-developed Android APP for a series of experiments, including data transmission rate, transmission distance, and computational time for real-time data receiving.
3.1 Experimental system
As shown in Fig. 6(a), the experimental LED-DCC system mainly includes two parts, the LED display panel with a microcontroller for video display and invisible data transmission, and the smartphone for receiving the hidden data via its camera and the developed APP. The realization of the experimental system is shown in Fig. 6(b), and the detailed configurations of the experimental system are shown in Table 2.
Fig. 6. The experimental LED-DCC system. (a) The schematic diagram of the experimental system. (b) The realization of the experimental system.
Table 2. The hardware configurations of the experimental system
The Arduino microcontroller is used to implement the proposed ARBC scheme for encoding hidden data and combining the encoded data with video content data. As a demonstration, the LED display panel is a small panel containing 16×16 LED display points with a pattern of turn right sign. And the LED display points are controlled by the integrated 74HC595 8-bit Shift Register. Note that the LED display point at (16, 16), i.e., at the lower right corner, is selected as the LED-DCC point in our experiments.
At the receiver side, the VLC signals receiving, the fast image processing algorithms, and the ABRC decoding were implemented with Java in the self-developed APP. Note that the smartphone’s camera was set to work at an underexposure mode with the ISO value at 50 and the exposure time at 1/125 second.
3.2 Data transmission performance
In this experiment, the data transmission performance of a single point of the proposed LED-DCC system is evaluated, in terms of data transmission rate and frame error rate (FER). Since the size of the LED display panel is only 4×4 cm2, the distance between the LED display panel and the smartphone is fixed as 30 cm. We used the string “Traffic accident ahead” as an example of transmission data. This string has 22 characters in total, and the size after ASCII encoding is 176 bits. Therefore, with inserting an 8-bit SFD and an 8-bit EFD, the total length of the data frame is 192 bits and the final length of the ABRC-encoded frame is 960 symbols.
The encoded frame is repeatedly sent 100 times through the LED display panel, and the experimental smartphone and self-developed APP are used to photograph the LED display panel and extract the hidden information in real-time, as shown in Fig. 7. The experimental results show that when the transmission distance is 30 cm, the self-developed APP can correctly extract these 100 frames’ hidden data without using any error correction code. Therefore, the proposed LED-DCC system can achieve a data transmission rate of 30bps with a FER of 0 for a single LED-DCC point at the distance of 30 cm. Note that only one LED-DCC point is used as the data transmission channel in this experiment. If all the LED display points are used as LED-DCC points, then multi-channel parallel communication can be realized, and the total data transmission rate can reach 16×16×30 bps = 7,680 bps.
3.3 Data transmission performance versus distance
In this experiment, we increase the distance between the LED display panel and the smartphone from 10 cm to 100 cm. At each transmission distance, 100 data frames, each of which has 960 symbols, were transmitted and compared with the received data to evaluate the data transmission performance for different distances. The results are shown in Fig. 8.
It can be seen from Fig. 8 that when the distance is less than 30 cm, the data transmission rate is as high as 30 bps. When the distance is increased to 40 cm, the data transmission rate decreases to about 28.5bps since about 5 of the 100 frames were lost (i.e., FER = 0.05). When the distance exceeds 50 cm, the data transmission rate drops sharply. The main reason is that the light-emitting area of the LED-DCC point is small, i.e., 3.14 mm2. When the transmission distance is too far, it is hard to distinguish the image formed by the LED-DCC point from other LED display points due to the smartphone’s resolution which leads to difficulties in information extraction and decoding errors. When the distance reaches 80 cm, the data transmission rate drops to near 0 Therefore, the maximum transmission distance of the experimental LED-DCC system is 80 cm.
Note that the LED-DCC point we currently use is only 2 mm in diameter while the effective communication distance can reach 80 cm. In practical application scenarios, the size of an outdoor LED display point usually reaches 5 mm or more. According to the basic principles of triangle geometry and optical imaging, the maximum distance of information transmission can reach 2 m. In addition, clustering multiple adjacent LED-DCC points for hidden data transmission can also increase the light-emitting area and the communication distance can be multiplied as well. Since the distance of the experimental LED-DCC system is short, we assumed that the channel characteristics is stable. When the communication distance is extended and more complex application scenarios are considered, the characteristics of the optical wireless channel should be fully analyzed [12].
3.4 Computational time for real-time data receiving
In the above experiments, the processing time of each stage for data receiving were measured in the APP. The average computational time for each processing stage is shown in Table 3. ROI detection is the most time-consuming, but the average computational time for decoding a single video frame is only about 11.827 ms, which is lower than the video frame arrival time interval (33 ms). Therefore, it could be seen that the proposed processing algorithms can support real-time LED-DCC.
Table 3. The average computational time for data receiving
Although increasing the resolution of the camera could help extending the transmission distance. However, since the number of pixels will be increased, the computational time of image processing will also be increased. As shown in Table 3, most of the time is consumed for the LED display area detection. Our image processing algorithms provide a lightweight ROI detection method by summing the pixel values in one column or one row to detect boundaries of display area quickly. Moreover, since we don’t need the precise boundaries, sampling detection can be used to further reduce the computational time for the LED display area detection. Therefore, the influence of increasing the resolution of cameras will be acceptable.
4. Conclusion
This paper proposes a real-time LED-DCC based on LED display panels and smartphones. We have solved the problem of rate mismatch and symbol unsynchronization between the LED display panel and the smartphone’s CMOS image sensor the by proposed ABRC scheme. Meanwhile, this paper also proposes fast image processing algorithms to reduce the computational complexity and processing time as well for real-time LED-DCC data receiving. An experimental system is implemented on a 16×16 points LED display panel and a commercial Android smartphone with a self-developed APP. n 8 megapixels camera working at 30 fps. The experimental results show that the real-time LED-DCC can achieve a transmission rate of 30bps for one LED display point and a maximum distance of 80 cm, and the computational time on the Android smartphone for data receiving is only 11.827 ms. And the transmission rate and the transmission distance can be multiplied when multiple LED-DCC points are used with MIMO technology and point clustering.
Funding
National Key Research and Development Program of China (2019YFE0123600); National Natural Science Foundation of China (No. 61771222, No. 61872109); Fundamental Research Funds for the Central Universities (No. 21620439); Natural Scientific Research Innovation Foundation in Harbin Institute of Technology (HIT.NSRIF.2020078); Guangdong Provincial Postgraduate Education Innovation Project (2019SFKC08); Horizon 2020 Framework Programme (Marie Sklodowska-Curie grant No. 872172); .
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. G. Kasilingam, M. Ramalingam, and C. Sekar, “A survey of light emitting diode (LED) display board,” Indian J. Sci. Technol. 7(2), 185–188 (2013). [CrossRef]
2. A. Jovicic, J. Li, and T. Richardson, “Visible light communication: Opportunities, challenges and the path to market,” IEEE Commun. Mag. 51(12), 26–32 (2013). [CrossRef]
3. Y. Goto, I. Takai, T. Yamazato, H. Okada, T. Fujii, S. Kawahito, S. Arai, T. Yendo, and K. Kamakura, “A New Automotive VLC System Using Optical Communication Image Sensor,” IEEE Photonics J. 8(3), 1–17 (2016). [CrossRef]
4. S. Hranilovic and F. R. Kschischang, “Short-range wireless optical communication using Pixelated transmitters and imaging receivers,” IEEE Int. Conf. Commun.2(c), 891–895 (2004).
5. W. Hu, H. Gu, and Q. Pu, “Lightsync: Unsynchronized visual communication over screen-camera links,” Proc. Annu. Int. Conf. Mob. Comput. Networking, MOBICOM 15–26 (2013).
6. A. Wang and B. Zeng, “InFrame ++ : Achieve Simultaneous Screen-Human Viewing and Hidden Screen-Camera Communication Categories and Subject Descriptors,” MobiSys 181–195 (2015).
7. B. Zhang, K. Ren, G. Xing, X. Fu, and C. Wang, “SBVLC: Secure barcode-based visible light communication for smartphones,” IEEE Trans. Mob. Comput. 15(2), 432–446 (2016). [CrossRef]
8. K. Jo, M. Gupta, and S. K. Nayar, “DisCO: Display-camera communication using rolling shutter sensors,” ACM Trans. Graph. 35(5), 1–13 (2016). [CrossRef]
9. C. Chen, W. Huang, L. Zhang, and W. H. Mow, “Robust and unobtrusive display-to-camera communications via blue channel embedding,” IEEE Trans. Image Process. 28(1), 156–169 (2019). [CrossRef]
10. S. Y. Jung, J. H. Lee, W. Nam, and B. W. Kim, “Complementary Color Barcode-Based Optical Camera Communications,” Wirel. Commun. Mob. Comput. 2020, 1–8 (2020). [CrossRef]
11. ZBar, “ZBar bar code reader,” (2011), http://zbar.sourceforge.net/.
12. A. Al-Kinani, C. X. Wang, L. Zhou, and W. Zhang, “Optical wireless communication channel measurements and models,” IEEE Commun. Surv. Tutor. 20(3), 1939–1962 (2018). [CrossRef]
