Open Access
Add to BibSonomy
Abstract
Image-based target tracking methods rely on continuous image acquisition and post-processing, which will result in low tracking efficiency. To realize real-time tracking of fast moving objects, we propose an image-free target tracking scheme based on the discrete cosine transform and single-pixel detection. Our method avoids calculating all the phase values, so the number of samples can be greatly reduced. Furthermore, complementary modulation is applied to reduce the measurement noise, and background subtraction is applied to enhance the contrast. The results of simulations and experiments demonstrate that the proposed scheme can accomplish the tracking task in a complex background with a sampling ratio of less than 0.59% of the Nyquist-Shannon criterion, thereby significantly reducing the measurement time. The tracking speed can reach 208 fps at a spatial resolution of 128 × 128 pixels with a tracking error of no more than one pixel. This technique provides a new idea for real-time tracking of fast-moving targets.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Target tracking is crucial in various fields such as remote sensing [1], traffic navigation [2], and biomedicine [3]. Generally, target tracking methods include image-based and image-free methods, where the image-based method is most popular. Ghost imaging (GI) is a novel imaging method that has advantages of super-resolution and anti-interference, which can be used to track hidden targets under ultralow light conditions [4]. Initially, GI was realized with entangled photon pairs generated by spontaneous parametric down-conversion [5,6]. Later, many other light sources such as pseudo-thermal light [7], true thermal light [8], and X-rays [9] were used to realize GI. In addition, a series of algorithms such as normalized ghost imaging [10], differential ghost imaging [11] and compressed sensing [12] have been proposed to improve the imaging quality.
For real-time target tracking via GI, several studies have been conducted [13–18]. Li et al. reported a method for tracking a moving target with an unknown constant speed in the imaging plane [19]. Gong et al. then developed this scheme to track an axially moving target with an unknown constant speed [20,21]. Hu et al. used deep learning to chart the target’s trajectory and images with a sampling ratio as low as 3.7% [22]. Sun et al. proposed a single-pixel tracking and imaging scheme to acquire images at weak photon fluxes where the average number of detected photons was about 1.88 photons/speckle for each illumination pulse [23]. Later, our group proposed a moving compensation based computational GI for a transversely moving target [24]. Benefiting from the single-pixel detection, GI can perform well even at ultra-low photon fluxes.
However, owing to the character of the intensity correlation used in GI, numerous measurements are required to retrieve a fairly-high image quality, which restricts the imaging speed and further restricts the tracking speed. Shi et al. proposed a modulation scheme based on the Hadamard basis to transform two-dimensional (2D) images into one-dimensional (1D) projection curves, which provide the location information of the moving object. Compressed sensing was then applied to achieve an ultra-low sampling ratio, but the computational complexity was very high [25]. Zhang et al. proposed another image-free real-time detection and tracking method for fast-moving objects using a Fourier basis, which uses a single-pixel detector for 2D tracking or dual-pixel detectors for three-dimensional tracking [26,27]. The discrete Fourier transform (DFT) does work well for image reconstruction but its basis matrices are complex, and it is very resource consuming to obtain full phase information. Our discrete cosine transform (DCT) method, a subset of DFT, can avoid calculating all the phase values, so the number of samples can be greatly reduced. For sparse objects, the sampling rate can be compressed to less than 0.59%, and the information obtained is sufficient for our purpose. This improvement is vital for tracking of fast moving objects.
In our computational GI system, a specified 1D discrete cosine basis is designed as the modulation pattern, and the inverse discrete cosine transform is used to obtain the 1D projection curves of the moving object. In this way, the position of the target on the x-axis and y-axis can be determined. By repeating the process, we can obtain the entire trajectory. Simulations and experiments have been conducted to demonstrate the efficiency of our scheme.
2. Principle of image-free target tracking
Single-pixel detection mainly consists of a single-pixel detector and a series of modulation patterns. The commonly used modulation patterns include random, Hadamard, and Fourier patterns. The measurement at the single-pixel detector, defined here as ${S_i}$, includes information of the target F (x, y) and modulation pattern Pi (x, y), which can be expressed as
where $F({x,y} )$ is the spatial distribution of the original image with $M \times N$ pixels. The discrete cosine transform, being a subset of DFT, is widely used for image compression. The 2D DCT and inverse DCT of an image can be expressed asAccording to Eqs. (2) and (3), the ‘slice’ $D({u,0} )$ and $D({0,v} )$ of the frequency spectrum can be further written as
The actual one-dimensional projection of the target at moment t is recorded as ${f_t}{(x)_{}}\textrm{an}{\textrm{d}_{}}{f_t}(y)$, respectively, obtained by accumulating pixel values along the x-axis and y-axis. The expressions of ${f_t}{(x)_{}}\textrm{an}{\textrm{d}_{}}{f_t}(y)$ are ${f_t}(x) = \sum\limits_y {F(x,y)} {,_{}}{f_t}(y) = \sum\limits_x {F(x,y)} $. Substituting ${f_t}(x )$ and ${f_t}(y )$ into (4) and (5), respectively, we obtain
In order to eliminate the background light interference, we adopt the complementary modulation method. First, a series of grayscale patterns which include the discrete cosine information is generated using the following equation:
where ${P_{x1}}({u,0} )$ and ${P_{x2}}({u,0} )$ are complementary grayscale patterns along the x-axis with spatial frequency (u,0), a is the average light intensity of the light field, and b is the contrast. If the modulation patterns need to be loaded to a binary spatial light modulator, their values must be positive, so a and b are set to 0.5. For the complementary modulation patterns ${P_{y1}}\; ({0,v} ),$ and ${P_{y2}}\; ({0,v} )$ in the y direction we only need to replace u and x with v and y. Then, substituting Eqs. (8) and (9) into Eq. (6) we obtain: where $\varepsilon \; $denotes the background noise. Similarly, substituting ${P_{y1}}\; ({0,v} ),$ and ${P_{y2}}\; ({0,v} )$ into Eq. (7), we can obtain D1(0,v) and D2(0,v). As we can see, Eqs. (10) and (11) contain the same background noise, so the latter can be eliminated using a difference method:As the target moves, the complex stationary background will cause blurring of the projection of the target, so we need to obtain the background values ${D_b}(u,0)$ and ${D_b}(0,v)$ in advance then use subtraction to eliminate the background interference. Thus, ${f_t}(x)$ and ${f_t}(y)$ can be recovered as follows:
The central coordinates of the target Lt (x) and Lt (y) at moment t are thus:
The principle of the above scheme is illustrated in Fig. 1. Figure 1(a) is the target, Fig. 1(b) the complementary pattern, Fig. 1(c) the normalized target projection recovered from the obtained discrete cosine spectrum, and Fig. 1(d) is the positioning result. The recovered information is converted from a 2D image to a 1D projection, which greatly reduces the amount of information that has to be processed.
Fig. 1. Principle of image-free target tracking scheme. (a) Target. (b) Complementary modulation patterns of four sets of 1D discrete cosine basis. (c) 1D projection curves of the target on the x-axis (lower figure) and y-axis (upper figure). (d) Reconstructed location, xt1, xt2, yt1 and yt2 are the respective boundary positions.
3. Results and discussion
3.1 Simulation results
In the frequency domain, the target information is mainly concentrated in the low frequency part, and the light field modulation based on the DCT can use the low frequency patterns to collect the corresponding information of the target, thereby achieving under-sampling. The compressed sampling rate $\eta \; $ is defined as:
where k is the number of samples.To demonstrate the effectiveness of the proposed scheme, the relationship between the sampling rate and tracking accuracy is analyzed. We conducted many simulations on tracking an aeroplane, and we first took a scene where a small object is in a large background, as shown in Fig. 1(a). The results of the recovered projection are shown in Fig. 2.
Fig. 2. Simulation of the relationship between the sampling rate and tracking accuracy. (a) and (e) Real projections of the target on the y- and x-axes, respectively. The second, third and fourth column figures depict recovered 1D projection curves with sampling numbers of 12, 24 and 48, respectively.
In the figure, the upper and lower rows show the 1D projections on the y-axis and x-axis, respectively. In the first column, Figs. 2(a) and (e) show the projection curves of the original target, while the second, third and fourth columns show the recovered curves with sampling numbers of 12, 24, and 48, respectively. As can be seen, with the increase of the sampling rate, the recovered projections are closer to the original projections, so higher tracking accuracy will be obtained.
To further characterize the relationship between the sampling rate and tracking accuracy, the root mean square error (RMSE) was used to evaluate the recovered results.
Fig. 3. Comparison of RMSE obtained by the DCT (solid lines) and cake-cutting ordered Hadamard basis (dotted lines). (a) Small target on a large background, (d) large target on a smaller background, (g) aircraft on a nonzero gray background. (b), (e) and (h), the RMSE obtained, respectively, by the gray-scale DCT (DCT-X and DCT-Y) and the Hadamard basis. (c), (f) and (i), the RMSE obtained, respectively, by the binary DCT (BDCT- X and BDCT-Y) and the Hadamard basis.
Figures 3(b), 3(e), and 3(h) show, respectively, the gray-scale discrete cosine and Hadamard basis for each scene. These three figures show that the RMSE of the recovered projection using gray-scale DCT is always lower than that of the Hadamard basis. In other words, the proposed method requires fewer samples than the latter for the same RMSE. However, since a DMD is commonly used as the modulator in GI, the discrete cosine basis needs to be binarized. Thus, we further compare the RMSE obtained from the binarized DCT with that of the Hadamard basis, and the results are shown in Figs. 3(c), 3(f), and 3(i). We can see that the RMSE of the recovered projections using the binarized DCT is much lower when the sampling rate is low. These comparisons also show that our proposed method requires fewer samples. Figures 3(h) and 3(i) also show that the Hadamard basis method is not suitable for a non-zero background. Moreover, our DCT method is suitable for different sparsity targets. Figures 3(c), 3(f), and 3(i) also demonstrate that a small target needs more samples when the RMSE is set to be less than 0.1. In Fig. 3(c), when the number of samples on both the x- and y-axes exceeds 24, the RMSE is less than 0.1. When complementary modulation is used, the total number of samples is 96, and the corresponding sampling rate is 0.15%.
From these simulations we can see that our image-free target tracking scheme can effectively obtain the position information of a moving target at different moments in time, whatever the relative size of the target to the background.
3.2 Experimental results
Figure 4 shows a schematic of the experiments performed to demonstrate the validity of our method. The setup consisted of a near-infrared laser (YSL Photonics, SC-5), a DMD (ViALUX, V7001), a step motor, a bucket detector (Thorlabs, PDA20CS2), and a data acquisition board (EM9118). First, the laser beam was expanded to illuminate the DMD, which was preloaded with a discrete cosine basis to modulate the light field, as shown in Fig. 4(b). According to the simulation results, 24 low-frequency discrete cosine bases for each axis were used to restore the projection curves, so a total of 96 bases were required. A Floyd-Steinberg dithering algorithm was used to binarize the discrete cosine basis [29]; Fig. 4(c) shows a magnified portion of the basis. Lens ${\textrm{L}_1}$ projected the structured light on the target, while the field of view was 34 mm × 34 mm. To simulate the movement of the target, we used a step motor to control the speed of the moving target. The single-pixel detector and lens ${\textrm{L}_2}\; $collected the total light intensity transmitted by the target.
Fig. 4. (a) Experimental setup of image-free target tracking scheme. (b) Illustration of four sets of the binary 1D discrete cosine basis, and (c) magnification of part of Fig. (b).
In the experiment, the moving target was a round hole of diameter 5 mm (∼19 pixels) cut out of a strip of black cardboard, while the stationary background consisted of a cloud and tree, also cut out of two black cardboard triangles assembled with a gap in between. In this way, as the target strip was moved along the diagonal gap, light could be transmitted through both target and the cloud and tree background. The size of the screen was 128×128 pixels. In Fig. 5, the white dotted line represents the trajectory of the moving target. The step motor advanced the target from point “a” to point “e” with a constant speed of 2 mm/s, and the total distance travelled was 10 mm, as shown in Fig. 5. On the trajectory, a, b, c, and d are reference points which were later calibrated and used to evaluate the accuracy of our scheme. Simulation results proved that 96 speckle patterns can meet the tracking demand, thus we preloaded a set of 1000 × 96 discrete cosine bases into the DMD to obtain 1000 sets of coordinates. In this case, from Eq. (15), our sampling rate was ∼0.59%.
Fig. 5. Experimental setup. The target moves along the trajectory through point a to e across the background.
To visualize the influence of the refresh rate of the DMD on the recovered trajectories of ${f_t}(x )$ and ${f_t}(y )$, we chose refresh rates of 5, 10, 15, and 20 kHz, as shown in Fig. 6, where the horizontal axes represent the number of frames and the vertical axes are the position coordinates; the color bar represents the values of ${f_t}(x )$ and ${f_t}(y )$. The time required for each frame can be calculated from the term 96/RDMD, where RDMD is the refresh rate.
Fig. 6. Experimental results with different refresh rates. Trajectories of ${f_t}(x )$ and ${f_t}(y )$ with a refresh rate of: (a) and (b), 5 kHz, (c) and (d), 10 kHz, (e) and (f), 15 kHz, (g) and (h), 20 kHz.
The trajectory of the moving target was drawn using the central coordinates of the target. Figures 7(a–d) show the recovered central coordinates with refresh rates of 5, 10, 15 and 20 kHz. In Figs. 7 (a–d), the blue and red lines represent the central coordinates on the x-axis and y-axis, respectively. Figures 7(e–h) show the change with frame number of the recovered central coordinates (x, y), for corresponding refresh rates of 5, 10, 15 and 20 kHz. It should be noted that Figs. 6 and 7 are the results of the same set of experiments. Statistically, it only takes 0.14 ms to obtain one coordinate (x, y) for the moving target. The results demonstrate that our GI method is time efficient, and allows real-time target tracking.
Fig. 7. (a–d) Recovered central coordinates of the target with refresh rates of 5, 10, 15, and 20 kHz. The blue/red lines represent the central coordinates on the x- and y-axes. (e–h) Change with number of frames of the coordinates (x, y), corresponding to refresh rates of 5, 10, 15, and 20 kHz.
To verify the accuracy of our method we need to compare our measured trajectory with the true reference coordinates. To calibrate the reference coordinates, we controlled the target to stop at points a, b, c, and d, and used a differential GI algorithm to reconstruct images of the target, from which we obtained the actual coordinates of the reference points. Comparing these with the values obtained by our method, we can deduce the accuracy of the reconstructed coordinates. In the detection of moving objects, the distance d that the target has moved through during one measurement cycle is given by
where n is the number of samples in one cycle, v the velocity of the target, and f the DMD refresh rate. In our experiment, the size of a single pixel is $0.27\textrm{mm} \times 0.27\; \textrm{mm}$, v is $2\; \textrm{mm}/\textrm{s}$, n is 96, and f is 20 kHz, thus d is $0.0013\; \textrm{mm}$. Since the pixel size $0.27\; \textrm{mm} > > 0.0013\; \textrm{mm}$, the resolution accuracy is limited by the size of the imaging pixel; thus, the tracking accuracy is also one pixel. The experimental results are presented in Table 1, which shows the coordinates of the reference points reconstructed by differential GI and by DCT. At points a and b, the DCT recovered coordinates are consistent with the true coordinates. At points c and d, the recovered coordinates “y” deviate from the actual coordinates by one pixel, which is within the limits of experimental error for our target of diameter 5 mm. Our results showed that when the movement speed is 2 mm/s, for refresh rates faster than 5 kHz, the refresh rate has almost no influence on the measurement accuracy, and the fastest tracking speed can reach 208 frames per second.
Table 1. Coordinates of the reference points obtained by differential GI and by DCT at different refresh rates
4. Conclusion
In this study we have proposed a fast image-free target-tracking scheme based on DCT. The method is cost-efficient, requiring only one DMD and a single-pixel detector, and uses only 0.14 ms to calculate one coordinate (x, y), so real-time target tracking is possible. It is noteworthy that the number of modulated patterns can be changed according to application requirements, so that the tracking speed can be further optimized. Since the scheme uses complementary modulation patterns and background subtraction, it is robust to interference from ambient light and a stationary background. Because of the extremely low sampling rate of DCT it is possible to achieve a very high tracking accuracy at 5 kHz even with a low refresh rate spatial light modulator. With further improvements, it should also be possible to achieve fast image-free tracking of multiple targets.
Funding
National Natural Science Foundation of China (61973018, 61975229); Civil Space Project (D040301); National Key Research and Development Program of China (2018YFB0504302); Beijing Institute of Technology Research Fund Program for Young Scholars (2020CX04104).
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. X. Q. Lu, X. T. Zheng, and Y. Yuan, “Remote sensing scene classification by unsupervised representation learning,” IEEE Trans. Geosci. Remote Sensing 55(9), 5148–5157 (2017). [CrossRef]
2. D. Cireşan, U. Meier, J. Masci, and J. Schmidhuber, “Multi-column deep neural network for traffic sign classification,” Neural Netw. 32, 333–338 (2012). [CrossRef]
3. S. T. Acton and N. Ray, “Biomedical image analysis: tracking,” Synthesis Lectures on Image Video, and Multimedia Processing 2(1), 1–152 (2006). [CrossRef]
4. M. P. Edgar, G. M. Gibson, R. W. Bowman, B. Sun, N. Radwell, K. J. Mitchell, S. S. Welsh, and M. J. Padgett, “Simultaneous real-time visible and infrared video with single-pixel detectors,” Sci. Rep. 5(1), 10669 (2015). [CrossRef]
5. T. B. Pittman, Y. H. Shih, D. V. Strekalov, and A. V. Sergienko, “Optical imaging by means of two-photon quantum entanglement,” Phys. Rev. A 52(5), R3429–R3432 (1995). [CrossRef]
6. D. V. Strekalov, A. V. Sergienko, D. N. Klyshko, and Y. H. Shih, “Observation of two-photon “ghost” interference and diffraction,” Phys. Rev. Lett. 74(18), 3600–3603 (1995). [CrossRef]
7. A. Gatti, E. Brambilla, M. Bache, and L. A. Lugiato, “Ghost imaging with thermal light: comparing entanglement and classical correlation,” Phys. Rev. Lett. 93(9), 093602 (2004). [CrossRef]
8. D. Zhang, Y. H. Zhai, L. A. Wu, and X. H. Chen, “Correlated two-photon imaging with true thermal light,” Opt. Lett. 30(18), 2354–2356 (2005). [CrossRef]
9. A. X. Zhang, Y. H. He, L. A. Wu, L. M. Chen, and B. B. Wang, “Tabletop x-ray ghost imaging with ultra-low radiation,” Optica 5(4), 374–377 (2018). [CrossRef]
10. B. Q. Sun, S. S. Welsh, M. P. Edgar, J. H. Shapiro, and M. J. Padgett, “Normalized ghost imaging,” Opt. Express 20(15), 16892–16901 (2012). [CrossRef]
11. F. Ferri, D. Magatti, L. A. Lugiato, and A. Gatti, “Differential ghost imaging,” Phys. Rev. Lett. 104(25), 253603 (2010). [CrossRef]
12. O. Katz, Y. Bromberg, and Y. Silberberg, “Compressive ghost imaging,” Appl. Phys. Lett. 95(13), 131110 (2009). [CrossRef]
13. W. Jiang, X. Li, X. Peng, and B. Sun, “Imaging high-speed moving targets with a single-pixel detector,” Opt. Express 28(6), 7889–7897 (2020). [CrossRef]
14. S. M. Jiao, M. J. Sun, Y. Gao, T. Lei, Z. W. Xie, and X. C. Yuan, “Motion estimation and quality enhancement for a single image in dynamic single-pixel imaging,” Opt. Express 27(9), 12841–12854 (2019). [CrossRef]
15. S. Sun, J. H. Gu, H. Z. Lin, L. Jiang, and W. T. Liu, “Gradual ghost imaging of moving objects by tracking based on cross correlation,” Opt. Lett. 44(22), 5594–5597 (2019). [CrossRef]
16. S. Sun, H. Z. Lin, Y. K. Xu, J. H. Gu, and W. T. Liu, “Tracking and imaging of moving objects with temporal intensity difference correlation,” Opt. Express 27(20), 27851–27861 (2019). [CrossRef]
17. Y. Y. Huang, C. Ou Yang, K. Fang, Y. F. Dong, J. Zhang, L. M. Chen, and L. A. Wu, “High speed ghost imaging based on a heuristic algorithm and deep learning,” Chin. Phys. B 30(6), 064202 (2021). [CrossRef]
18. C. L. Luo, W. X. Wan, and W. L. Gong, “Single-pixel imaging for dynamic targets via a non-diffracting beam,” Opt. Lasers Eng. 139, 106450 (2021). [CrossRef]
19. E. R. Li, Z. W. Bo, M. L. Chen, W. L. Gong, and S. S. Han, “Ghost imaging of a moving target with an unknown constant speed,” Appl. Phys. Lett. 104(25), 251120 (2014). [CrossRef]
20. X. H. Li, C. J. Deng, M. L. Chen, W. L. Gong, and S. S. Han, “Ghost imaging for an axially moving target with an unknown constant speed,” Photonics Res. 3(4), 153–157 (2015). [CrossRef]
21. W. L. Gong and S. S. Han, “The influence of axial correlation depth of light field on lensless ghost imaging,” J. Opt. Soc. Am. B 27(4), 675–678 (2010). [CrossRef]
22. H. K. Hu, S. Sun, H. Z. Lin, L. Jiang, and W. T. Liu, “Denoising ghost imaging under a small sampling rate via deep learning for tracking and imaging moving objects,” Opt. Express 28(25), 37284–37293 (2020). [CrossRef]
23. S. Sun, H. K. Hu, Y. K. Xu, H. Z. Lin, E. F. Zhang, L. Jiang, and W. T. Liu, “Single-Pixel Tracking and Imaging under Weak Illumination,” Signal Image and Video Process (2020).
24. Z. H. Yang, W. Li, Z. Y. Song, W. K. Yu, and L. A. Wu, “Tracking compensation in computational ghost imaging of moving objects,” IEEE Sens. J. 21(1), 1 (2020). [CrossRef]
25. D. F. Shi, K. X. Yin, J. Huang, K. Yuan, W. Y. Zhu, C. B. Xie, D. Liu, and Y. J. Wang, “Fast tracking of moving objects using single-pixel imaging,” Opt. Commun. 440, 155–162 (2019). [CrossRef]
26. Q. W. Deng, Z. B. Zhang, and J. G. Zhong, “Image-free real-time 3-D tracking of a fast-moving object using dual-pixel detection,” Opt. Lett. 45(17), 4734–4737 (2020). [CrossRef]
27. Z. B. Zhang, J. Q. Ye, Q. W. Deng, and J. G. Zhong, “Image-free real-time detection and tracking of fast-moving object using a single-pixel detector,” Opt. Express 27(24), 35394–35401 (2019). [CrossRef]
28. W. K. Yu, “Super Sub-Nyquist single-pixel imaging by means of cake-cutting hadamard basis sort,” Sensors 19(19), 4122 (2019). [CrossRef]
29. Z. B. Zhang, X. Y. Wang, G. A. Zheng, and J. G. Zhong, “Fast Fourier single-pixel imaging via binary illumination,” Sci. Rep. 7(1), 12029 (2017). [CrossRef]
