Open Access
Add to BibSonomy
Abstract
In the industrial environment, the positioning of mobile terminals plays an important role in production scheduling. Visible light positioning (VLP) based on a CMOS image sensor has been widely considered as a promising indoor positioning technology. However, the existing VLP technology still faces many challenges, such as modulation and decoding schemes, and strict synchronization requirements. In this paper, a visible light area recognition framework based on convolutional neural network (CNN) is proposed, where the training data is the LED images acquired by the image sensor. The mobile terminal positioning can be realized from the perspective of recognition without modulating LED. The experimental results show that the mean accuracy of the optimal CNN model is as high as 100% for the two-class and the four-class area recognitions, and is more than 95% for the eight-class area recognition. These results are obviously superior to other traditional recognition algorithms. More importantly, the model has high robustness and universality, which can be applied to various types of LED lights.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
The Industrial Internet of Things (IIoT) is the application and expansion of the internet of things in the industrial field, which applies intelligent terminals with perception and interaction capabilities, ubiquitous technology and intelligent analysis technology to all aspects of the industrial production process [1,2]. The positioning of mobile terminals plays an indispensable role in the safety control of manufacturing sites (such as large-scale warehouse). Compared with traditional wireless positioning technology, visible light has rich spectrum resources, controllable radiation range, endogenous security, deep coverage, personalized customization, and other advantages [3,4]. Visible light positioning (VLP) makes use of light sources and light receivers, which can joint lighting, communication, and positioning without additional equipment. It has become a potential choice for IIoT.
Visible light positioning (VLP) can be divided into imaging positioning and non-imaging positioning according to different receivers. The non-imaging positioning takes photodetectors (PDs) as receivers. The positioning algorithm mainly includes a fingerprint of received signal strength and trilateral positioning [5,6], Time of Arrival (TOA) [7,8], time difference of arrival (TDOA) [9,10], and Angle of Arrival (AOA) [11,12]. These methods require joint positioning of multiple lights or strict synchronization, and the positioning performance is greatly interfered with by ambient light or reflected light. Unlike the PD-based methods, the imaging positioning takes CMOS image sensors as receivers, which is not directly affected by ambient light. It includes two types of position methods. One is to use photogrammetry to achieve positioning through the geometric relationship between LED and LED images [13]. If positioning is realized under a single LED, other sensors are required to assist positioning, such as motion sensors and inertial navigation [14,15]. The other is to use the rolling shutter images acquired by CMOS camera for positioning [16,17]. The transmitted data is captured on the images in form of light and dark stripes. The position is predicted only when the captured LED image contains a complete data packet and is decoded correctly. However, it can only provide approximate positioning, and the positioning accuracy depends on the LED distribution density. The current VLP researches are still mainly focused on positioning accuracy and decoding scheme, which do not fully exploit the natural and unique advantages of LED in IIoT applications. As a transmitter of VLP, LED has consistency in light intensity and position. CMOS image sensor can convert the received light intensity into image pixels by PD array. The rolling shutter effect can effectively avoid multipath and background interference. Meanwhile, the image sensor is composed of a CMOS array and microlens. When the beam passes through the lens, the focus imaging point on the CMOS imaging plane, the center of the lens, and the center of the light source are collinear. The approximate location of the target can be predicted by the collinear relationship and the consistency. The VLP can be realized from the perspective of image recognition. Compared with the traditional image recognition methods, the convolutional neural network (CNN) exhibits competitive performance [18,19]. Several researches investigate that CNN can be applied to VLP system and have achieved better positioning performance [20–22].
In this paper, we propose and demonstrate an LED area recognition method for VLP. This method can realize target location through a smartphone embedded with CMOS image sensor without modulating LED. Essentially, the VLP is transformed into a visible light recognition problem by the consistency of the visible light intensity and location and the collinear relationship of the imaging principle. To ensure the accuracy of recognition, we also propose an LED area recognition framework based on CNN. By the supervised learning of LED images, an optimal LED area recognition model with high recognition accuracy and robustness is established. To verify the effectiveness of the proposed scheme, we built a prototype and conducted extensive evaluation. The experimental results show that the mean recognition accuracy of CNN model is as high as 100% for two-class and four-class area recognitions, and is more than 95% for eight-class area recognition. Compared with traditional algorithms, the LED area recognition model has higher recognition accuracy. More importantly, this model has high generalization performance and can be applied to different types of LED lights.
2. System principle
The VLP system based on a CMOS camera is shown in Fig. 1. The transmitter uses an unmodulated beacon-free LED. At the receiver, a smartphone embedded with a CMOS image sensor is used to collect LED images in different areas using photogrammetry theory. Meanwhile, the rolling shutter effect mainly senses the light intensity and is relatively insensitive to the color, so the collected rolling shutter image avoids the interference of the background in the environment. Next, VLP is converted into visible light area recognition by the proposed area recognition framework. The framework can classify LED images into different positioning areas.
The image sensor is composed of a CMOS array and microlens. CMOS is an imaging module. When the light beam passes through the lens, the focus imaging point on the CMOS image plane, the center of the lens, and the center of the light source are collinear. This collinear relationship and the image of the light source on the imaging plane are used to identify the location. The image sensor imaging model is shown in Fig. 2. In the figure, the position of the LED is the actual installation position. The camera center $C$ is the camera coordinate system with the optical center of the image sensor as the origin. $I$ is the projection of the LED on the image plane.
In this paper, we divide the LED radiation area into three types, as shown in Fig. 3. The positioning within an LED lighting region is converted into three types of identification problems, namely two-class area $\{A_0,A_1\}$, four-class area $\{B_0,B_1, B_2, B_3\}$ and eight-class area $\{C_0,C_1, C_2, C_3, C_4, C_5, C_6, C_7\}$. The more detailed the area division, the higher the positioning accuracy. In the case of two-class, the recognizable sub-unit-cell is a rectangle; in the case of four-class, the recognizable sub-unit-cell is a square; in the case of eight-class, the recognizable sub-unit-cell is an isosceles triangle. We compare the performance of the three types of area recognition and take the recognition accuracy as the positioning standard.
3. LED area recognition framework
In this section, a simple CNN framework is proposed to handle the area recognition task, as shown in Fig. 4. The framework mainly includes two parts: the offline training phase and the online testing phase. In the offline phase, the input images and the convolution kernel are used for continuous convolution and pooling operations to extract image features. At the same time, supervised learning is used to optimize the parameters of CNN to generate the optimal training model. In the online testing phase, the LED images collected in real time are input into the trained model for testing and the test results are visualized.
The CNN framework consists of two convolution layers, two pooling layers, and a fully connected layer. Convolution layer: a size $3 \times 3$ convolution kernel is used for convolution processing of input images, and the convolution operation has translation invariance. It can support neurons to learn features and has high robustness. The convolution process can be expressed as:
The $\mathbf {A_i}({\mathbf {x}_i})$ through the linear region of the ReLU activation function $\mathbf {f}(\cdot )$[23] as the input of the next layer, which can be expressed as:
The max-pooling operation with size $2 \times 2$ is used to divide the input image into several rectangular regions and output the maximum value for each sub-region. The fully connected layer is used to connect all features and feed the output value to the softmax classifier. The softmax classifier maps the outputs to a list of probabilities $\mathbf {q} = (q_0, q_2,\ldots, q_M)$ over all the classes. The cross-entropy loss function of the CNN network is used to calculate the error rate, and can be expressed as:
where $p_j$ is the ground-truth of the $j$-th class, $q_j$ is the predicted probability of the $j$-th class.This paper adopts supervised learning, and the data needs to be labeled before feeding into the CNN framework. LED images are continuously collected when the target moves along a certain trajectory. The training data are labeled according to the divided area in Fig. 3. In the offline phase, our goal is to train the CNN classification model with the loss function $L(f(x),y)$ to obtain a high performance adaptive recognition model. The error back-propagation updates the set of $\boldsymbol {\theta }$ through the gradient descent learning algorithm. The training process of the network can be summarized as the following optimization problems:
In the online phase, we test the accuracy of the new images and visualized them (category and percentage of accuracy).
where TP is the number of positive samples predicted to be positive; FN is the number of positive samples predicted as negative samples; FP is the number of negative samples predicted to be positive, and TN is the number of negative samples predicted as negative samples.4. Experiment and results
4.1 Experimental setup
To evaluate the performance of the proposed LED area recognition framework, we build a prototype as shown in Fig. 5. At the transmitter, two types of LED are employed, and installed on the ceiling without any processing. At the receiver, there is a mobile auto-guided vehicle (AGV) equipped with a smartphone. Here, we employ ViVo V1916A to capture the LED images. The specific experimental parameters are shown in Table 1. The AGV moves in a straight line on a table with size $80cm \times 80cm$. The physical dimensions of the sub-unit-cell are a rectangle of $80cm \times 40cm$ in two-class, a square of $40cm \times 40cm$ in four-class, and an isosceles triangle of $40cm$ in eight-class. To build the datasets, we collect 32 groups of data at an interval of $2.5cm$, and the amount of data in each area is basically equal, avoiding the error caused by the uneven distribution of data. We collect nearly 3000 images and divided them into three datasets: 2700 images for training and validation, and the rest of 300 images for testing. Besides, we also collect nearly 1000 new images to test the generalization performance of the trained model. The recognition model is trained with TensorFlow on the NVIDIA GeForce RTX2060 GPU.
Fig. 5. Experimental setup for evaluating the proposed LED area recognition framework: (a) experimental platform; (b) the LED transmitter (LED1 and LED2); (c) the receiver (AGV carries a smartphone); (d) LED images acquisition process in positioning area.
Table 1. Experimental parameters.
4.2 Experimental results
4.2.1 LED1 area recognition results
We first test the recognition performance of the LED1 in three types of areas. Figure 6 shows the loss value of train dataset and the accuracy of validation dataset in the training phase. When the iteration reaches 50 epochs, the loss value tends to converge and the accuracy tends to be consistent. On this basis, the optimal model is selected for offline testing, and its generalization performance for newly acquired images is tested.
Fig. 6. The loss value and accuracy of the three types of area recognition model in the training phase.
We compare the accuracy of the three types of area recognition in LED1 from the confusion matrix, as shown in Fig. 7. Figure 7(a) shows the accuracy of LED1’s two-class area recognition, it can be seen that the mean accuracy of $A_0$ and $A_1$ area is 100%. This also verifies that the LED images based on the collinear relationship are quite different in the two areas. Figure 7(b) shows the accuracy of LED1’s four-class area recognition. It has the same results as the two-class area recognition and the mean accuracy reaches 100%. Figure 7(c) shows the accuracy of LED1’s eight-class area recognition and the mean accuracy ups to 99%. Although the mean accuracy is slightly lower than the former two, the overall accuracy is still high. These results show that when the radiation range of a single LED is divided into regions, the sparser the regions are, the higher the recognition accuracy is. Besides, we consider that high accuracy is related to data acquisition mode. Due to a large amount of data collection, the fingerprint database is also large. The classification features learned by the training framework are more detailed. Meanwhile, the CMOS image sensor is composed of a PD array, these results also verify the consistency of target location and light intensity. Finally, we compare the proposed CNN area recognition framework with traditional algorithms, such as KNN and SVM [25], as shown in Table 2. Here, the features of KNN and SVM algorithms are the R, G, B, and grayscale values of LED images. It can be seen that the accuracy of the proposed CNN framework is significantly higher than other two algorithms.
Table 2. Mean accuracy of different algorithms
4.2.2 LED2 area recognition results
The accuracy of the three types of area recognition in LED2 is shown in Fig. 8. Figure 8(a) shows the accuracy of $A_0$ and $A_1$, and the mean accuracy is as high as 100%. The recognition results of the two-class are equal to the results of LED1. Figure 8(b) shows the accuracy of $B_0$, $B_1$, $B_2$, and $B_3$, and the mean accuracy reaches 99.2%. It is slightly lower than that of LED1. Figure 8(c) shows the recognition accuracy of the eight-class area, with a mean accuracy of 98.4%. The recognition results are comparable to that of LED1. Based on the recognition results of LED1, the recognition accuracy in the LED radiation area is independent of the LED design. In addition, the beam angle of LED1 is $78^\circ$ and that of LED2 is $12.5^\circ$. Although the two are very large differences, the recognition performance is almost the same. The experimental results also prove that the performance of the proposed method is independent of the beam angle of LED light. Compared with traditional algorithms, such as KNN and SVM, the mean recognition accuracy is shown in Table 2. It can be seen that the accuracy of the proposed CNN framework is significantly higher than other two algorithms. Finally, we test the recognition results of LED2 with the trained model of LED1 and test the recognition results of LED1 with the trained model of LED2, as shown in Fig. 9. It can be seen that the area recognition results for the two-class are the same, and the mean accuracy ups to 100%. For four-class area recognition, the mean accuracy of the LED2 image tested by the trained LED1 model is 92.5%, and the mean accuracy of the LED1 image tested by the trained LED2 model is 94.4%. Although the former is slightly lower than the latter, both are over 90%. For the region recognition of eight-class, the mean accuracy of LED2 images tested by the trained LED1 model is 83.4%, and the mean accuracy of LED1 images tested by the trained LED2 model is 91.2%. The former is significantly lower than the latter, which may be related to LED2 images. These two types of LED images in some areas are quite different. In general, the trained LED area recognition model has a high universality and can be applied to LED lights of different designs.
4.2.3 Generalization performance of the optimal area recognition model
Generalization performance is the adaptability of a model to new samples. We collect new LED images at different locations and test the generalization performance of the optimal LED area recognition model. Figure 10 shows the recognition performance of the trained LED1 and LED2 area recognition models for new samples. For two-class area recognition, the two types of LED lights have the same accuracy for the new samples, both up to 100%. For four-class area, the recognition accuracy of the two types of LED lights is similar. For eight-class area recognition, the recognition accuracy of LED1 is slightly higher than that of LED2 light, but both are more than 95%. It can be seen that the trained optimal model has high generalization performance. In addition, Fig. 11 shows some visual results of eight-class area recognition of LED1, including the category and accuracy of the prediction. It can be seen that the accuracy of each area is quite high. Figure 12 shows part of the visual results of eight-class area recognition of LED2. Similarly, it has a high recognition accuracy for new samples at any location.
Fig. 11. The visual results of partial samples of LED1, involving the predicted categories and the corresponding percentage of accuracy.
Fig. 12. The visual results of partial samples of LED2, involving the predicted categories and the corresponding percentage of accuracy.
5. Conclusions
In this paper, we propose an LED area recognition framework for visible light positioning based on CMOS image sensors. It converts visible light positioning into visible light area recognition without modulating and demodulating LED. This scheme does not require strict synchronization, and the recognition area can be divided according to the positioning accuracy requirements. The experiment results verify the performance of two types of LED in three cases. The three cases are divided into two-class area recognition, four-class area recognition, and eight-class area recognition. In three cases, the mean accuracy of the two types of LED lights in two-class area recognition reaches 100%, the mean accuracy in four-class area recognition exceeds 98%, and the mean accuracy in eight-class area recognition exceeds 95%, both of which have high accuracy. Compared with traditional algorithms, such as SVM and KNN, the proposed LED area recognition framework based on CNN has obvious advantages. In addition, a mutual test of the two types of LED lights is performed. The results show that the area recognition model trained by one type of LED light can be used for different types of LED lights. Meanwhile, it has good generalization performance for new samples. Therefore, the proposed scheme is feasible. Our current and future work can involve expanding the single light to the multi-light environment and designing a multi-cell recognition framework.
Funding
National Natural Science Foundation of China (62271505); Key Technologies Research and Development Program (2022YFB28022804).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. A. Mahmood, L. Beltramelli, S. Fakhrul Abedin, S. Zeb, N. I. Mowla, S. A. Hassan, E. Sisinni, and M. Gidlund, “Industrial iot in 5g-and-beyond networks: Vision, architecture, and design trends,” IEEE Trans. Ind. Inf. 18(6), 4122–4137 (2022). [CrossRef]
2. D. Schneider, A. Shrotri, H. Flatt, O. Stübbe, A. Wolf, R. Lachmayer, and C.-A. Bunge, “Impact of industrial environments on visible light communication,” Opt. Express 29(11), 16087–16104 (2021). [CrossRef]
3. M. Kavehrad, “Sustainable energy-efficient wireless applications using light,” IEEE Commun. Mag. 48(12), 66–73 (2010). [CrossRef]
4. P. H. Pathak, X. Feng, P. Hu, and P. Mohapatra, “Visible light communication, networking, and sensing: A survey, potential and challenges,” IEEE Commun. Surv. Tutorials 17(4), 2047–2077 (2015). [CrossRef]
5. A. H. A. Bakar, T. Glass, H. Y. Tee, F. Alam, and M. Legg, “Accurate visible light positioning using multiple-photodiode receiver and machine learning,” IEEE Trans. Instrum. Meas. 70, 1–12 (2021). [CrossRef]
6. K. Wang, Y. Liu, and Z. Hong, “Rss-based visible light positioning based on channel state information,” Opt. Express 30(4), 5683–5699 (2022). [CrossRef]
7. X. Sun, J. Duan, Y. Zou, and A. Shi, “Impact of multipath effects on theoretical accuracy of toa-based indoor vlc positioning system,” Photonics Res. 3(6), 296–299 (2015). [CrossRef]
8. M. F. Keskin and S. Gezici, “Comparative theoretical analysis of distance estimation in visible light positioning systems,” J. Lightwave Technol. 34(3), 854–865 (2016). [CrossRef]
9. C. Zhang, D. Li, X. Feng, L. Tan, L. Yang, and X. Yang, “Indoor visible light positioning method based on tdoa and fingerprint,” in International Conference on Electronic Technology, Communication and Information (IEEE, 2021), pp. 397–401.
10. S. M. Sheikh, H. M. Asif, K. Raahemifar, and F. Al-Turjman, “Time difference of arrival based indoor positioning system using visible light communication,” IEEE Access 9, 52113–52124 (2021). [CrossRef]
11. H. Steendam, “A 3-d positioning algorithm for aoa-based vlp with an aperture-based receiver,” IEEE J. Select. Areas Commun. 36(1), 23–33 (2018). [CrossRef]
12. C.-Y. Hong, Y.-C. Wu, Y. Liu, C.-W. Chow, C.-H. Yeh, K.-L. Hsu, D.-C. Lin, X.-L. Liao, K.-H. Lin, and Y.-Y. Chen, “Angle-of-arrival (aoa) visible light positioning (vlp) system using solar cells with third-order regression and ridge regression algorithms,” IEEE Photonics J. 12, 1–5 (2020). [CrossRef]
13. J.-W. Lee, S.-J. Kim, and S.-K. Han, “3d visible light indoor positioning by bokeh based optical intensity measurement in smartphone camera,” IEEE Access 7, 91399–91406 (2019). [CrossRef]
14. H. Cheng, C. Xiao, Y. Ji, J. Ni, and T. Wang, “A single led visible light positioning system based on geometric features and cmos camera,” IEEE Photonics Technol. Lett. 32(17), 1097–1100 (2020). [CrossRef]
15. Y. Wang, B. Hussain, and C. P. Yue, “Arbitrarily tilted receiver camera correction and partially blocked led image compensation for indoor visible light positioning,” IEEE Sens. J. 22(6), 4800–4807 (2022). [CrossRef]
16. K.-L. Hsu, Y.-C. Wu, Y.-C. Chuang, C.-W. Chow, Y. Liu, X.-L. Liao, K.-H. Lin, and Y.-Y. Chen, “Cmos camera based visible light communication (vlc) using grayscale value distribution and machine learning algorithm,” Opt. Express 28(2), 2427–2432 (2020). [CrossRef]
17. Y. Liu, H.-Y. Chen, K. Liang, C.-W. Hsu, C.-W. Chow, and C.-H. Yeh, “Visible light communication using receivers of camera image sensor and solar cell,” IEEE Photonics J. 8(1), 1–7 (2016). [CrossRef]
18. X.-X. Du, Y. Mu, Z.-W. Ye, and Y.-J. Zhu, “A passive target recognition method based on led lighting for industrial internet of things,” IEEE Photonics J. 13(4), 1–8 (2021). [CrossRef]
19. M. Guo, P. Zhang, Y. Sun, W. Zhang, Y. Zhou, and Y. Yang, “Object recognition in optical camera communication enabled by image restoration,” Opt. Express 30(20), 37026–37037 (2022). [CrossRef]
20. D.-J. Li, Z.-H. Wei, and J.-B. Fang, “Deep learning-based robust visible light positioning for high-speed vehicles,” Photonics 9(9), 632 (2022). [CrossRef]
21. L.-S. Hsu, D.-C. Tsai, C.-W. Chow, Y. Liu, Y.-H. Chang, Y.-Z. Lin, C.-H. Yeh, Y.-C. Wang, and Y.-Y. Chen, “Using data pre-processing and convolutional neural network (cnn) to mitigate light deficient regions in visible light positioning (vlp) systems,” J. Lightwave Technol. 40(17), 5894–5900 (2022). [CrossRef]
22. H. M. Chan, C.-W. Chow, L.-S. Hsu, Y. Liu, C.-W. Peng, Y.-H. Jian, and C.-H. Yeh, “Utilizing lighting design software for simulation and planning of machine learning based angle-of-arrival (aoa) visible light positioning (vlp) systems,” IEEE Photonics J. 14(6), 1–7 (2022). [CrossRef]
23. X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” J. Mach. Learning Res. 15, 315–323 (2011).
24. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference for Learning Representations (2015), p. 1–1.
25. H. Zhang, A. Berg, M. Maire, and J. Malik, “Svm-knn: Discriminative nearest neighbor classification for visual category recognition,” in Computer Society Conference on Computer Vision and Pattern Recognition (IEEE, 2006), pp. 2126–2136.
