LiDAR-camera-system-based 3D object detection with proposal selection and grid attention pooling

Pei An; Pei An; Junxiong Liang; Jie Ma

doi:10.1364/AO.443856

Applied Optics
Vol. 61,
Issue 11,
pp. 2998-3007
(2022)
•https://doi.org/10.1364/AO.443856

LiDAR-camera-system-based 3D object detection with proposal selection and grid attention pooling

Pei An, Junxiong Liang, and Jie Ma

Not Accessible

Your library or personal account may give you access

Get PDF
Email
Share
Get Citation
Copy Citation Text
Pei An, Junxiong Liang, and Jie Ma, "LiDAR-camera-system-based 3D object detection with proposal selection and grid attention pooling," Appl. Opt. 61, 2998-3007 (2022)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Citation alert
Save article

Check for updates

Related Topics
Optics & Photonics Topics
?

The topics in this list come from the Optics and Photonics Topics applied to this article.

About this Article
History
- Original Manuscript: September 20, 2021
- Revised Manuscript: January 27, 2022
- Manuscript Accepted: March 14, 2022
- Published: April 4, 2022
Virtual Issues
Applied Optics Artificial Intelligence and Machine Learning in Optical Information Processing (2022)

Abstract

3D object detection is an important module for autonomous driving. A LiDAR camera optical system is suitable for accurate object detection, for it provides both 3D structure and 2D texture features. However, as LiDAR and a camera have different sensor properties, it is challenging to generate effective fusion features. Motivated by this, we propose, to the best of our knowledge, a novel LiDAR–camera based 3D object detection method. First, proposal selection is presented to utilize accurate 2D proposals predicted from RGB images to improve the quality of 3D proposals. It contains a (i) proposal addition and (ii) proposal filter. To increase the recall rate, the proposal addition generates extra 3D proposals via back-projecting 2D proposals on LiDAR depth. The proposal filter removes unrelated 3D proposals by matching 2D proposals with intersection-over-union thresholds. Then, considering the LiDAR mechanism, grid attention pooling is employed to estimate weights of grid points from LiDAR and image features to generate salient pooling features. Comparisons and ablation studies demonstrate that the proposed method achieves better performance and benefits the advanced application of a LiDAR camera system.

Full Article | PDF Article

More Like This

LiDAR-camera-system-based unsupervised and weakly supervised 3D object detection

Haosen Wang, Tiankai Chen, Xiaohang Ji, Feng Qian, Yue Ma, and Shifeng Wang
J. Opt. Soc. Am. A 40(10) 1849-1860 (2023)

LiDAR-camera system extrinsic calibration by establishing virtual point correspondences from pseudo calibration objects

Pei An, Yingshuo Gao, Tao Ma, Kun Yu, Bin Fang, Jun Zhang, and Jie Ma
Opt. Express 28(12) 18261-18282 (2020)

Fusion of an RGB camera and LiDAR sensor through a Graph CNN for 3D object detection

Jinsol Choi, Minwoo Shin, and Joonki Paik
Opt. Continuum 2(5) 1166-1179 (2023)

Previous Article Next Article

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Figures (10)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Tables (7)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Equations (7)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

c	Label	$w_{c}$	$l_{c}$	$h_{c}$	$ϕ [c]$	$τ [c]$
0	Car	1.6 m	3.9 m	1.56 m	0.8	0.7
1	Pedestrian	0.6 m	0.8 m	1.73 m	0.9	0.5
2	Cyclist	0.6 m	1.76 m	1.73 m	0.8	0.5

Methods	Modality	R2D/P2D, Easy	R2D/P2D, Mod	R2D/P2D, Hard	R3D/P3D, Easy	R3D/P3D, Mod	R3D/P3D, Hard	AP3D, Mod/Hard
M3D-RPN [4]	Camera	97.5%/64.8%	92.5%/75.5%	75.0%/78.4%	35.0%/15.2%	22.5%/14.2%	17.5%/15.3%	9.71%/7.42%
Pseudo-LiDAR $+ +$ [25]	Camera	97.5%/50.2%	87.5%/52.2%	80.0%/58.7%	77.5%/33.0%	55.0%/27.9%	47.5%/31.9%	42.43%/36.99%
PointRCNN [2]	LiDAR	97.5%/48.4%	95.0%/45.8%	90.0%/53.8%	92.5%/48.9%	82.5%/37.6%	77.5%/44.7%	75.64%/70.70%
PV-RCNN [3]	LiDAR	100.0%/58.7%	97.5%/60.2%	95.0%/67.9%	95.0%/51.5%	87.5%/49.0%	82.5%/56.2%	81.46%/76.82%
Part-A2 [26]	LiDAR	97.5%/25.4%	95.0%/23.6%	92.5%/30.2%	92.5%/43.5%	85.0%/20.0%	80.0%/25.4%	78.49%/73.51%
SA-SSD [27]	LiDAR	100.0%/57.6%	97.5%/64.8%	92.5%/71.9%	92.5%/47.9%	85.0%/51.4%	80.0%/58.8%	79.79%/74.16%
TANet [6]	LiDAR	95.0%/72.1%	92.5%/76.1%	87.5%/81.0%	90.0%/56.1%	82.5%/60.5%	75.0%/66.8%	75.94%/68.82%
PointPillars [28]	LiDAR	97.5%/9.0%	97.5%/9.5%	95.0%/12.8%	90.0%/8.1%	85.0%/8.2%	80.0%/10.8%	74.31%/68.99%
MV3D [29]	$L i D A R + C a m e r a$	-/-	-/-	-/-	87.5%/57.5%	75.0%/61.7%	62.5%/66.4%	63.63%/54.00%
ContFuse [30]	$L i D A R + C a m e r a$	-/-	-/-	-/-	92.5%/56.0%	77.5%/61.6%	70.0%/67.1%	68.78%/61.67%
AVOD $+$ FPN [8]	$L i D A R + C a m e r a$	97.5%/41.2%	92.5%/53.5%	87.5%/61.3%	90.0%/42.2%	80.0%/41.0%	72.5%/47.9%	71.76%/65.73%
PointFusion [9]	$L i D A R + C a m e r a$	-/-	-/-	-/-	-/-	-/-	-/-	63.00%/53.27%
F-PointNet [18]	$L i D A R + C a m e r a$	97.5%/52.4%	97.5%/66.9%	87.5%/72.3%	95.0%/45.8%	90.0%/57.2%	80.0%/62.8%	69.79%/60.59%
F-ConvNet [19]	$L i D A R + C a m e r a$	97.5%/65.4%	95.0%/37.0%	82.5%/42.8%	92.5%/54.2%	82.5%/51.5%	72.5%/39.6%	76.39%/66.69%
MMF [11]	$L i D A R + C a m e r a$	100.0%/11.9%	97.5%/18.3%	95.0%/23.1%	95.0%/16.1%	85.0%/14.9%	77.5%/18.5%	77.43%/70.22%
EP-Net [12]	$L i D A R + C a m e r a$	97.5%/62.1%	97.5%/59.1%	92.5%/66.1%	95.0%/56.7%	85.0%/47.3%	80.0%/54.9%	79.28%/74.59%
3D-CVF [14]	$L i D A R + C a m e r a$	97.5%/84.1%	95.0%/86.1%	87.5%/89.3%	92.5%/67.0%	85.0%/70.7%	77.5%/76.3%	80.05%/73.11%
Pointpainting [10]	$L i D A R + C a m e r a$	100.0%/70.6%	95.0%/64.9%	92.5%/71.8%	90.0%/49.1%	80.0%/49.3%	75.0%/55.2%	71.70%/67.08%
PS-GAP (our method)	$L i D A R + C a m e r a$	97.5%/67.0%	97.5%/67.2%	92.5%/74.2%	92.5%/52.8%	85.0%/54.1%	82.5%/61.7%	79.27%/76.43%
PS-GAP v2 (our method)	$L i D A R + C a m e r a$	97.5%/54.2%	97.5%/56.6%	95.0%/64.7%	92.5%/43.6%	87.5%/45.4%	82.5%/53.1%	80.18%/76.21%
Our method ranks in $F_{1}$ score (last column is Mod AP rank):
$F_{1}$ score rank in camera based methods		1	1	1	1	1	1/( $+ 32.43 %$ )	1/( $+ 38.38 %$ )
$F_{1}$ score rank in LiDAR based methods		2	2	2	3	2	1/( $+ 2.81 %$ )	2/( $- 0.65 %$ )
$F_{1}$ score rank in LiDAR–camera based methods		3	3	2	3	3	1/( $+ 0.23 %$ )	1/( $+ 0.13 %$ )
Our method ranks in $F_{2}$ score (last column is hard AP rank):
$F_{2}$ score rank in camera based methods		1	1/( $+ 0.92 %$ )	1/( $+ 12.50 %$ )	1	1	1/( $+ 34.02 %$ )	1/( $+ 39.44 %$ )
$F_{2}$ score rank in LiDAR based methods		1	1/( $+ 0.76 %$ )	1/( $+ 0.17 %$ )	2	1	1/( $+ 2.66 %$ )	2/( $- 0.39 %$ )
$F_{2}$ score rank in LiDAR–camera based methods		3	2/( $- 3.64 %$ )	1/( $+ 0.29 %$ )	3	3	1/( $+ 1.44 %$ )	1/( $+ 1.84 %$ )

Methods	Modality	Car, Easy	Car, Moderate	Car, Hard	Ped, Easy	Ped, Moderate	Ped, Hard	Cyc, Easy	Cyc, Moderate	Cyc, Hard	mAP
PointRCNN [2]	LiDAR	88.26%	77.73%	76.67%	65.62%	58.57%	51.48%	82.76%	62.83%	59.62%	69.28%
TANet [6]	LiDAR	88.21%	77.85%	75.62%	70.80%	63.45%	58.22%	85.98%	64.95%	60.40%	71.72%
PV-RCNN [3]	LiDAR	89.37%	83.69%	78.70%	63.12%	54.84%	51.78%	86.06%	69.47%	64.50%	71.28%
EP-Net [12]	$L i D A R + C a m e r a$	92.28%	82.59%	80.14%	–	–	–	–	–	–	–
3D-CVF [14]	$L i D A R + C a m e r a$	89.97%	79.88%	78.47%	–	–	–	–	–	–	–
PS-GAP (our method)	$L i D A R + C a m e r a$	89.78%	86.60%	85.35%	65.15%	60.53%	58.34%	88.24%	72.05%	67.48%	74.83%
Gain		−2.50%	$+ 2.91 %$	$+ 5.21 %$	−5.65%	−2.92%	$+ 0.12 %$	$+ 2.18 %$	$+ 2.58 %$	$+ 2.98 %$	$+ 3.55 %$

Methods	Proposal Addition	Refinement with GP	Refinement with GAP	Proposal Filter	$f_{l}$	Modality	Car, Easy	Car, Moderate	Car, Hard	Average 3D Recall/Precision
Baseline-raw	$\times$	$\sqrt$	$\times$	$\times$	–	LiDAR	92.10%	84.36%	82.48%	90.0%/46.9%
Baseline-64	$\times$	$\times$	$\sqrt$	$\times$	64	LiDAR	92.04%	84.83%	83.72%	90.0%/51.6%
Baseline-96	$\times$	$\times$	$\sqrt$	$\times$	96	LiDAR	93.13%	85.57%	84.05%	90.0%/53.8%
Baseline-128	$\times$	$\times$	$\sqrt$	$\times$	128	LiDAR	93.10%	85.66%	84.15%	90.0%/55.2%
Baseline-PF	$\sqrt$	$\times$	$\sqrt$	$\times$	96	$L i D A R + C a m e r a$	94.17%	86.83%	84.48%	91.7%/59.6%
PS-GAP	$\sqrt$	$\times$	$\sqrt$	$\sqrt$	96	$L i D A R + C a m e r a$	95.00%	87.96%	85.87%	91.7%/62.4%
Gain							$+$ 2.90%	$+$ 3.60%	$+$ 3.39%	$+$ 1.7%/15.5%

$ϕ [0]$	mAP of Car	$ϕ [1]$	mAP of Ped.	$ϕ [2]$	mAP of Cyc.
0.5	88.53%	0.5	62.48%	0.5	76.95%
0.6	88.75%	0.6	62.52%	0.6	77.15%
0.7	89.12%	0.7	62.59%	0.7	77.63%
0.8	89.57%	0.8	62.67%	0.8	78.12%
0.9	89.02%	0.9	62.77%	0.9	77.33%
1.0	88.36%	1.0	62.35%	1.0	76.08%
None	88.49%	None	62.41%	None	76.34%
Gain	$+ 1.08 %$	Gain	$+ 0.36 %$	Gain	$+ 1.78 %$

Abstract

Data availability

Cited By

Figures (10)

Tables (7)

Equations (7)

Applied Optics