MAFFNet: real-time multi-level attention feature fusion network with RGB-D semantic segmentation for autonomous driving

Tongfei Lv; Yu Zhang; Lin Luo; Xiaorong Gao

doi:10.1364/AO.449589

Applied Optics
Vol. 61,
Issue 9,
pp. 2219-2229
(2022)
•https://doi.org/10.1364/AO.449589

MAFFNet: real-time multi-level attention feature fusion network with RGB-D semantic segmentation for autonomous driving

Tongfei Lv, Yu Zhang, Lin Luo, and Xiaorong Gao

Not Accessible

Your library or personal account may give you access

Get PDF
Email
Share
Get Citation
Copy Citation Text
Tongfei Lv, Yu Zhang, Lin Luo, and Xiaorong Gao, "MAFFNet: real-time multi-level attention feature fusion network with RGB-D semantic segmentation for autonomous driving," Appl. Opt. 61, 2219-2229 (2022)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Citation alert
Save article

Check for updates

Abstract

Compared with RGB semantic segmentation, RGB-D semantic segmentation can combine the geometric depth information to effectively improve the segmentation accuracy. Considering the application of RGB-D semantic segmentation in autonomous driving, we design a real-time semantic segmentation network, that is, MAFFNet, which can effectively extract depth features and combine the complementary information in RGB and depth. We also design a multi-level attention feature fusion module that can excavate the available context information of RGB and depth features. At the same time, its inference speed can also meet the demands of autonomous driving. Experiments show that our network achieves excellent performance of 74.4% mIoU and an inference speed of 15.9 Hz at a full resolution of ${{2048}} \times {{1024}}$ on the cityscapes dataset. Using multi-source learning, we mixed the cityscapes and lost and found as the multi-dataset. Our network is also superior to previous algorithms in using the multi-dataset to detect small obstacles outside the road.

Full Article | PDF Article

More Like This

ASA-BiSeNet: improved real-time approach for road lane semantic segmentation of low-light autonomous driving road scenes

Yang Liu, Fulong Yi, Yuhua Ma, and Yongfu Wang
Appl. Opt. 62(19) 5224-5235 (2023)

Polarization-driven semantic segmentation via efficient attention-bridged fusion

Kaite Xiang, Kailun Yang, and Kaiwei Wang
Opt. Express 29(4) 4802-4820 (2021)

Robustifying semantic cognition of traversability across wearable RGB-depth cameras

Kailun Yang, Luis M. Bergasa, Eduardo Romera, and Kaiwei Wang
Appl. Opt. 58(12) 3141-3155 (2019)

Previous Article Next Article

Data availability

Data underlying the results presented in this article can be found in cityscapes [43] and lost and found [44].

43. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 3213–3223.

44. P. Pinggera, S. Ramos, S. Gehrig, U. Franke, C. Rother, and R. Mester, “Lost and found: detecting small road hazards for self-driving vehicles,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2016), pp. 1099–1106.

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Figures (9)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Tables (6)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Equations (12)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Method	mIoU(%)
Single RGB branch	54.6
RGB-D fusion without MAFF	59.3
RGB-D fusion in RGB branch	67.9
MAFFNet	74.4

Method	Stage1	Stage2	Stage3	Stage4	mIoU(%)
ResNet-18	$✓$				67.3
ResNet-18		$✓$			69.9
ResNet-18			$✓$		72.5
ResNet-18				$✓$	73.6
ResNet-18	$✓$	$✓$			69.7
ResNet-18	$✓$	$✓$	$✓$		72.8
ResNet-18	$✓$	$✓$	$✓$	$✓$	74.4

Method	Roa	Sid	Bui	Wal	Fen	Pol	TLi	TSi	Veg	Ter	Sky	Ped	Rid	Car	Tru	Bus	Tra	Mot	Bic	mIoU
ERFNet	97.4	80.6	90.3	55.8	50.1	57.5	58.6	68.2	90.9	61.2	93.1	73.0	53.2	91.8	59.1	70.1	66.7	44.9	67.1	70.0
DSNet	96.7	76.9	90.3	55.0	58.6	57.4	61.2	71.7	90.4	55.2	92.4	73.3	58.1	90.9	64.4	77.7	72.8	52.5	69.4	71.8
RFNet	96.3	78.3	90.1	46.7	55.9	58.1	63.9	71.3	90.9	60.1	92.3	77.2	58.4	93.1	69.3	81.1	69.4	53.6	72.4	72.5
MAFFNet	97.4	82.5	91.0	45.8	59.0	60.8	64.3	73.5	91.2	62.1	92.5	78.6	60.2	93.7	70.1	84.6	77.1	56.6	72.8	74.4

Method	Multi-Model	Resolution	mIoU(%)	FPS	Param/M
FCNS [2]	$\times$	$512 \times 1, 024$	65.3	2.0	134.5
SegNet [20]	$\times$	$1, 024 \times 2, 048$	56.1	11.9	29.5
DeepLabv2 [1]	$\times$	$512 \times 1, 024$	69.8	1.5	44
HRFR [23]	$\times$	$512 \times 1, 024$	73.6	1.3	N/A
ENet [24]	$\times$	$1, 024 \times 2, 048$	58.3	52.6	0.36
DSNet [31]	$\times$	$2048 \times 1024$	71.8	36.5	0.9
ERFNet [26]	$\times$	$1, 024 \times 2, 048$	70.0	13.2	2.1
ICNet [27]	$\times$	$720 \times 1, 280$	69.5	31.5	26.5
CGNet [29]	$\times$	$1, 024 \times 2, 048$	64.8	17.8	0.5
LDFNet [15]	$✓$	$1, 024 \times 2, 048$	68.5	16.9	2.3
RFBNet [16]	$✓$	$1, 024 \times 2, 048$	72.0	N/A	N/A
RFNet [40]	$✓$	$1, 024 \times 2, 048$	72.5	20.4	23.7
VGG-D [51]	$✓$	$1, 024 \times 2, 048$	64.4	N/A	N/A
SA-Gate [22]	$✓$	$1, 024 \times 2, 048$	82.8	N/A	63.4
SSMA [52]	$✓$	$1, 024 \times 2, 048$	82.2	N/A	56.4
AsymmetricFusion [53]	$✓$	$1, 024 \times 2, 048$	82.1	N/A	43.5
MAFFNet	$✓$	$1, 024 \times 2, 048$	74.4	15.9	26.5

Dilation Rates	mIoU(%)
(1, 2, 4, 8)	71.8
(1, 3, 6, 9)	70.2
(1, 6, 12, 18)	72.6
(1, 9, 18, 27)	71.4

Method	Roa	Sid	Bui	Wal	Fen	Pol	TLi	TSi	Veg	Ter	Sky	Ped	Rid	Car	Tru	Bus	Tra	Mot	Bic	SOb	mIoU
ERF-PSPNet	89.3	65.1	82.1	43.4	39.8	46.9	48.2	46.1	86.4	45.2	88.4	68.5	57.2	90.1	55.6	62.1	65.6	56.9	64.8	60.3	63.1
SwiftNet	95.7	60.6	89.1	50.9	53.6	56.9	61.1	71.4	90.7	55.0	92.2	75.2	58.5	92.7	65.3	81.3	70.0	56.2	72.1	62.8	70.6
RFNet	96.0	60.6	90.8	50.2	59.9	60.0	62.6	72.8	91.1	57.3	92.5	76.1	57.9	93.3	73.8	82.3	73.2	54.0	72.7	67.9	72.2
MAFFNet	97.0	67.5	91.0	49.2	57.3	60.9	63.6	72.6	91.1	59.6	93.2	75.3	59.1	93.2	70.4	84.9	75.0	53.6	72.6	65.7	72.6

Abstract

Data availability

Cited By

Figures (9)

Tables (6)

Equations (12)

Applied Optics