Open Access
Add to BibSonomyAbstract
A novel method for registering imagery with Light Detection And Ranging (LiDAR) data is proposed. It is based on the phenomenon that the back-projection of LiDAR point cloud of an object should be located within the object boundary in the image. Using this inherent geometrical constraint, the registration parameters computation of both data sets only requires LiDAR point clouds of several objects and their corresponding boundaries in the image. The proposed registration method comprises of four steps: point clouds extraction, boundary extraction, back-projection computation and registration parameters computation. There are not any limitations on the geometrical and spectral properties of the object. So it is suitable not only for structured scenes with man-made objects but also for natural scenes. Moreover, the proposed method based on the inherent geometrical constraint can register two data sets derived from different parts of an object. It can be used to co-register TLS (Terrestrial Laser Scanning) LiDAR point cloud and UAV (Unmanned aerial vehicle) image, which are obtaining more attention in the forest survey application. Using initial registration parameters comparable to POS (position and orientation system) accuracy, the performed experiments validated the feasibility of the proposed registration method.
© 2015 Optical Society of America
1. Introduction
By fusion of data from optical imagery and LiDAR point cloud, a scene can be completely represented with spectral information and 3D surface information [1]. This is very important for a number of remote sensing applications such as feature extraction, object classification, change detection, 3D city modeling, 3D biophysical parameter estimation and ecological modeling [2], etc. The prerequisite of using both data sets together is accurate registration of both data sets in a common reference frame [3].
Therefore, numerous registration methods have been developed in the last decade [4]. If a stereo pair of optical images is available, the 3D surface can be reconstructed by photogrammetry and then registered with LiDAR data by matching point clouds; Iterative closest point (ICP) algorithm can effectively resolve this 3D registration task. If only a single image is available, the optical imagery and LiDAR data registration methods often consist of three steps: feature extraction (corners, lines or patches are extracted from both data sets), feature matching (the extracted features from both data sets are matched) and transformation model (registration parameters are computed using the matched features from both data sets). The LiDAR data provides high density surface information in homogenous areas. However, sudden elevation changes along the surface are not clearly visible in the LiDAR data due to insufficient points. On the other hand, an optical image provides high quality details along object boundaries with height variations. Furthermore, there are substantial differences in characteristics of optical imagery and LiDAR data. Therefore, it is difficult to define and match the common features in both data sets. Usually, the control points are adopted in image-image registration. In the case of image-LiDAR registration, the main problem is to determine correct control points from both data sets and then their accurate matching. So the use of linear features has been relatively popular [5]. Sometimes, roof centroids are also adopted as common features [6]. The limitation of feature based registration methods is that they can only be used in structured scenes with man-made objects. Salient points based methods can be used in both structured scenes and natural scenes [7]. However, the problem of relief displacement which apparently exists in optical imagery has not been considered. Moreover, it is complex and thus difficult to implement. Recently, a new mutual information-based approach was reported [8] which can also be used in both structured scenes and natural scenes. But it needs not only LiDAR point clouds but also LiDAR intensity data.
In this paper, we present a novel method to register aerial imagery with LiDAR data. An inherent geometrical constraint is discovered and utilized. The registration of both data sets only needs LiDAR point clouds of several objects and their corresponding boundaries in the image. There are not any limitations on the geometrical and spectral properties of the object. The principle of the proposed method is straightforward. This method can overcome some of the aforesaid shortcomings, and it is suitable for both structured scenes and natural scenes.
2. Inherent geometrical constraint based image-LiDAR registration method
LiDAR takes samples on the object’s surface to form the point cloud, so these sample points must be within the object. This situation is illustrated in Fig. 1 and Fig. 2. Figure 1 is the point clouds of a scene. The crown of a tree and the roof of a building are picked up, and point clouds of them are labeled by white dots. These two sets of point clouds are back projected into the optical image and labeled by white cross marks on Fig. 2. It can be seen that most of these white cross marks are within the boundaries of the crown and the roof. This is an inherent geometrical constraint for data acquired by LiDAR and optical camera, and it is the foundation of the proposed registration method.
Fig. 1 Point clouds of a scene. White dots represent the point clouds of a tree crown and a building roof.
Fig. 2 Optical image of the same scene. The back-projections of the white dots in LiDAR point clouds are labeled by white cross marks.
The principle of the proposed registration method is simple and straightforward. When an optical image is correctly registered with LiDAR point cloud, in other words, when the registration parameters are correctly determined, the back-projections of LiDAR point cloud of an object should be within the object boundary in the image. There are four steps for the proposed registration method: point clouds extraction, boundary extraction, back-projection computation and registration parameters computation.
Step 1: point clouds extraction.
In the proposed method, objects are selected as the registration primitives instead of corners (control points), so these objects are called “control objects”. In this step, point clouds of the control objects are extracted from LiDAR data. Just as the requirement of the control points, these control objects should be evenly distributed in the whole scene.
Step 2: boundaries extraction.
The control objects are found in the optical image, and the boundaries of the control objects are extracted. The boundary can be represented by polygon with a number of vertices. Under the precondition of ensuring the control object is within the boundary, the area inside the boundary should be as small as possible.
Step 3: back-projections computation.
In this step, the 3D point cloud coordinates of LiDAR data are transformed into the 2D pixel coordinates of the optical image. Being a physical sensor model whose parameters are physically meaningful and rigorous, the collinearity equation of photogrammetry is usually used as the transformation model. The collinearity equation is expressed by Eq. (1). Given the camera’s interior and exterior parameters, it can compute the back-projections of the control object’s point cloud.
where
x,y: image coordinates of a back-projection on the image,
X,Y, Z: 3D coordinates of a LiDAR point,
XS,YS, ZS: the 3D coordinates of the camera’s perspective center,
x0, y0, f: image coordinates of the camera’s principal point and principal distance and
a1~a3, b1~b3, c1~c3: the parameters of the camera’s rotation matrix, which can be calculated by three rotation angles (ω, φ, κ) using Eq. (2).
Step 4: registration parameters computation.
When the collinearity equation is adopted as the transformation model, the registration parameters stand for the exterior orientation parameters of the optical image. The inherent geometrical constraint that the back-projections of LiDAR point cloud of the control object should be within the boundary [9] of this object in the image is utilized to compute the registration parameters. A ratio is defined. This ratio means the number of the back-projections inside the boundary to the number of all back-projections of a control object. Then an objective function is defined by Eq. (3).
where
p: registration parameters (exterior orientation parameters: XS,YS, ZS, ω, φ, κ),
n: number of the control objects in the whole scene.
Through Eq. (1) and Eq. (2), the registration parameters p will influence the pixel location of the back-projection. This will lead to the change of the number of the back-projections in side of the boundary, and then the change of the ratio R. So the value of the objective function of Eq. (3) will change with the registration parameters p. An optimization algorithm such as differential evolution or genetic algorithm can be used to solve the problem, and then the optimized registration parameters p is obtained.
3. Experimental results and discussion
Two data sets were used to assess the performance of the proposed registration method. The first data set comprising airborne LiDAR point cloud and aerial image is a subset of ISPRS Test Project on Urban Classification and 3D Building Reconstruction. Everyone can download it from ISPRS website, and then can process it and compare the registration result with ours. It is noteworthy that the second data set comprises TLS (Terrestrial Laser Scanning) LiDAR point cloud and UAV (Unmanned aerial vehicle) image. Because TLS and UAV observed the object from different view angles and saw different parts of the object, it is a challenge to co-register TLS LiDAR point cloud and UAV image. So, the second example showed the unique advantage of the proposed registration method based on the inherent geometrical constraint.
3.1 Registration of airborne LiDAR point cloud and aerial image
A subset of the Vaihingen test data set was used to test the proposed registration method. It is the data employed to the test of digital aerial cameras performed by the German Association of Photogrammetry and Remote Sensing (DGPF). The ground resolution of the digital aerial images is 8 cm. The Vaihingen test data set provided by DGPF also contains Airborne Laserscanner (ALS) data. The entire DGPF data set consists of 10 ALS strips. Inside an individual strip, the average point density is 4 points/m2. The camera’s interior and exterior orientation parameters were also given [10]. According to this document, the provided exterior orientation parameters should result in a back-projection error better than one pixel (RMS). So the provided parameters can be considered as the ground truth, and the computed exterior orientation parameters will be compared with them to evaluate the proposed registration method.
Four ALS strips (strip 3, 5, 7 and 9) were selected to form the point cloud of the test area. Two images (The file names are 10040083.tif and 10050105.tif, which are indicated by Image A and Image B) were also selected; both of them covered the test area. Eight test objects were chosen for Image A; they are man-made objects with relatively regular shapes, such as building roofs and cars; this experiment is to show that the proposed method is suitable for structured scenes with man-made objects. Another eight test objects were chosen for Image B; they are tree crowns; this experiment is to show that the proposed method is also suitable for natural scenes with objects of irregular shapes. Some of the selected control objects and their boundaries are shown in Fig. 3 and Fig. 4.
Differential evolution (DE) was adopted for the computation of the registration parameters. DE is a type of evolutionary algorithm. The advantages of DE are its simple structure, ease of use, speed and robustness. The detailed usage of DE can be found in [11]. In our experiments, DE/rand/1/bin was adopted as the strategy of DE, and the input arguments of DE were set as follows:
Number of population members: 100 Maximum number of iterations: 150
Step size from interval: 0.1 Crossover probability constant from interval: 0.8
Lower and upper bounds are critical to DE. The bounds should cover the region where the global minimum is expected. In our experiments, the registration parameters are the exterior orientation parameters of the camera. The initial values of the exterior orientation parameters can be obtained by POS (position and orientation system) or resection algorithm. Considering the accuracy of the initial values, the lower and upper bounds of the position and orientation parameters were set as Table 1:
Table 1. Bounds of the position and orientation parameters
As an evolutionary algorithm, DE relies in part on random sampling. This makes it a nondeterministic method, which may yield somewhat different solutions on different runs. So the program was run 20 times, and the solution with the best objective function value (0.053 for Image A and 0.068 for Image B) was chosen as the final registration parameters, which are shown in Table 2.
Table 2. Comparison of the registration parameters. GT stands for ground truth; RP stands for registration parameters; DF is the difference between GT and RP.
The registration quality is also assessed visually via overlays of the optical imagery with the LiDAR points back-projected to image space using the derived registration parameters. A small cut out portion of the overlaid, back-projected LiDAR point cloud and optical image is shown in Fig. 5. Both natural and artificial features of LiDAR and imagery data are matched. For example, there are two street lights in the left image of Fig. 5, and yellow dots can be seen on the top of street lights in the right image of Fig. 5.
Fig. 5 Optical image (left) and overlay of back-projected LiDAR point cloud and optical image (right).
The experimental results validate the feasibility of the proposed method. Moreover, a better result should be obtained if the point clouds and boundaries of the control objects can be extracted more accurately, or more number of population members and iterations can be set for DE.
3.2 Registration of TLS LiDAR point cloud and aerial image
This data set covered a 150m x 150m forest area beside Yongding River in Beijing, China. Riegl VZ-1000 TLS LiDAR was used to scan this area, and each scan has more than 20 million points (as shown in Fig. 6). Aerial images were also acquired at 150 m flying height with a Sony NEX-5R camera mounted on a small six-rotor UAV (as shown in Fig. 7).
Several trees evenly distributed across the scene were selected as control objects. The initial values of the registration parameters were set to (0,0,150) for translation (unit is meter) and (0,0,0) for rotation (unit is degree). And then both TLS LiDAR point cloud and UAV image were processed as described in section 3.1. In Fig. 7, the point clouds of the control objects were back projected into the UAV image using the initial values of the registration parameters (green points) and the optimized values of the registration parameters (red points). It can be seen that most of the red back-projections are within the boundaries of the tree crowns (blue closed curves). That means the UAV image was registered to the TLS LiDAR point cloud by the proposed inherent geometrical constraint based registration method
3.3 Discussion
Due to the availability of each registration method’s experimental data and source code or executable program, it is not easy to compare results of different registration methods. This problem will be solved if some standard benchmark data can be published and widely used by different registration methods, just like the well-known middlebury data set for comparison of two-frame stereo correspondence algorithms in computer vision. In this paper, we tested our registration method by a subset of ISPRS Test Project on Urban Classification and 3D Building Reconstruction. Everyone can download it from ISPRS website, and then can process it and compare the registration result with ours. This ISPRS Test Project can be used to quantitatively evaluate the registration methods for LiDAR point cloud and optical image, if some check points can be considered in the future.
The experiment of section 3.2 demonstrates one important advantage of the proposed registration method. It can co-register LiDAR point cloud with optical image even though they represent different parts of an object. For example, Fig. 8 is the point clouds of the tree in the middle of the test area (as shown in Fig. 6 and Fig. 7). The red point cloud is derived by image matching of UAV images and the blue point cloud is TLS LiDAR point cloud. Obviously, UAV image cannot see the lower part of the tree and the TLS LiDAR cannot acquire the upper portion of the tree crown because of the limitations of position and scan angle range. As mentioned previously, salient points based method [7] and mutual information-based method [8] can be used in natural scenes. Salient points based method uses descriptors similarity. Mutual information-based method uses statistical similarity. So two data sets to be registered must have common parts. This condition cannot be satisfied in the case of section 3.2.
Although TLS LiDAR only obtained the point cloud of the lower part of the tree, the back-projections of this point cloud still have to be within the boundary of the tree crown extracted from UAV image. This is the inherent geometrical constraint between point cloud and optical image. The proposed registration method utilized this inherent geometrical constraint and successfully registered two data sets in section 3.2.
4. Summary
Fusion of optical imagery and LiDAR point cloud is important for a number of remote sensing applications. The prerequisite of using combine data sets together is accurate registration of both data sets in a common reference frame. In this paper, an inherent geometrical constraint was discovered that the back-projections of LiDAR point cloud of an object should be within the boundary of this object in the image. This geometrical constraint was utilized to register the optical imagery with LiDAR data. The introduced method only needs LiDAR point clouds of several objects and their corresponding boundaries in the image. There are no limitations on the geometrical and spectral properties of the object. It is suitable not only for structured scenes with man-made objects but also for natural scenes. Moreover, the proposed method based on the inherent geometrical constraint can register two data sets derived from different parts of an object. It can be used to co-register TLS LiDAR point cloud and UAV image, which are obtaining more attention in the forest survey application.
The emphasis of this paper is put on validating the feasibility of the discovered geometrical constraint for the registration purpose. In order to make the proposed registration method more practical for real world applications, further research should be carried out on automatic selection of control objects, extraction and matching of point cloud and boundary. The techniques of segmentation and feature detection for point cloud and image will improve the automation of the registration procedures.
Acknowledgments
This work is supported by the National Basic Research Program of China (973 Program) Grant No. 2013CB733402 and free exploration project of state key laboratory of remote sensing science Grant No. 14ZY-04. This work is also supported by National Natural Science Foundation of China Grant No. 41171265, 41331171 and 40801131.
The author would like to acknowledge the provision of the Vaihingen data set by the German Society for Photogrammetry, Remote Sensing and Geoinformation (DGPF). [Cramer, 2010]: http://www.ifp.uni-stuttgart.de/dgpf/DKE-PAllg.html (in German).
References and links
1. J. Zhang, “Multi-source remote sensing data fusion: status and trends,” International Journal of Image and Data Fusion 1(1), 5–24 (2010). [CrossRef]
2. H. Buddenbaum, S. Seeling, and J. Hill, “Fusion of full-waveform LiDAR and imaging spectroscopy remote sensing data for the characterization of forest stands,” Int. J. Remote Sens. 34(13), 4511–4524 (2013). [CrossRef]
3. A. V. Kanaev, B. J. Daniel, J. G. Neumann, A. M. Kim, and K. R. Lee, “Object level HSI-LIDAR data fusion for automated detection of difficult targets,” Opt. Express 19(21), 20916–20929 (2011). [CrossRef] [PubMed]
4. R. Mishra and Y. Zhang, “A review of optical imagery and airborne LiDAR data registration methods,” Open Remote Sensing Journal 5(1), 54–63 (2012). [CrossRef]
5. P. R. O. Nnholm, M. Karjalainen, H. Kaartinen, K. Nurminen, and J. Hyypp A, “Relative orientation between a single frame image and LiDAR point cloud using linear features,” Photogrammetric Journal of Finland 23(2), 1–16 (2013).
6. E. Mitishita, A. Habib, and A. Machado, “Photogrammetric model orientation using LiDAR dataset,” in Proceedings of the 21st ISPRS Congress (Beijing, 2008), pp. 2488–2492.
7. R. M. Palenichka and M. B. Zaremba, “Automatic extraction of control points for the registration of optical satellite and LiDAR images,” IEEE Trans. Geosci. Rem. Sens. 48(7), 2864–2879 (2010). [CrossRef]
8. E. G. Parmehr, C. S. Fraser, C. Zhang, and J. Leach, “Automatic registration of optical imagery with 3D LiDAR data using statistical similarity,” ISPRS J. Photogramm. Remote Sens. 88, 28–40 (2014). [CrossRef]
9. K. Hormann and A. Agathos, “The point in polygon problem for arbitrary polygons,” Comput. Geom. 20(3), 131–144 (2001). [CrossRef]
10. F. Rottensteiner, G. Sohn, M. Gerke, and J. D. Wegner, “ISPRS test project on urban classification and 3D building reconstruction,” (2013). http://www2.isprs.org/tl_files/isprs/wg34/docs/ComplexScenes_revision_v4.pdf
11. J. Adeyemo and F. Otieno, “Optimizing planting areas using differential evolution (DE) and linear programming (LP),” Int. J. Phys. Sci. 4(4), 212–220 (2009).
