Method of depth simulation imaging and depth image super-resolution reconstruction for a 2D/3D compatible CMOS image sensor

Shijie Guo; Shijie Guo; Quanmin Chen; Quanmin Chen; Zhe Zhao; Zhe Zhao; Jiangtao Xu; Jiangtao Xu; Kaiming Nie; Kaiming Nie

doi:10.1364/AO.493280

Applied Optics
Vol. 62,
Issue 17,
pp. 4439-4454
(2023)
•https://doi.org/10.1364/AO.493280

Method of depth simulation imaging and depth image super-resolution reconstruction for a 2D/3D compatible CMOS image sensor

Shijie Guo, Quanmin Chen, Zhe Zhao, Jiangtao Xu, and Kaiming Nie

Not Accessible

Your library or personal account may give you access

Get PDF
Email
Share
Get Citation
Copy Citation Text
Shijie Guo, Quanmin Chen, Zhe Zhao, Jiangtao Xu, and Kaiming Nie, "Method of depth simulation imaging and depth image super-resolution reconstruction for a 2D/3D compatible CMOS image sensor," Appl. Opt. 62, 4439-4454 (2023)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Citation alert
Save article

Check for updates

Abstract

This paper presents a depth simulation imaging and depth image super-resolution (SR) method for two-dimensional/three-dimensional compatible CMOS image sensors. A depth perception model is established to analyze the effects of depth imaging parameters and evaluate the real imaging effects. We verify its validity by analyzing the depth error, imaging simulation, and auxiliary physical verification. By means of the depth simulation images, we then propose a depth SR reconstruction algorithm to recover the low-resolution depth maps to the high-resolution depth maps in two types of datasets. With the best situation in depth accuracy kept, the root mean square error (RMSE) of Middlebury dataset images are 0.0156, 0.0179, and 0.0183 m. The RMSE of RGB-D dataset images are 0.0223 and 0.0229 m. Compared with other listed conventional algorithms, our algorithm reduces the RMSE by more than 16.35%, 17.19%, and 23.90% in the Middlebury dataset images. Besides, our algorithm reduces the RMSE by more than 9.71% and 8.76% in the RGB-D dataset images. The recovery effects achieve optimized results.

Full Article | PDF Article

More Like This

Method for power reduction of demodulation driver circuit in indirect time-of-flight CMOS image sensor

Kaiming Nie, Guan Tian, Quanmin Chen, Zeqing Wang, Jiangtao Xu, and Zhiyuan Gao
Appl. Opt. 60(34) 10649-10659 (2021)

3D reconstruction of light-field images based on spatiotemporal correlation super-resolution

Wei Feng, Junhui Gao, Jichen Sun, and Henghui Wang
Appl. Opt. 62(12) 3016-3027 (2023)

Snapshot super-resolution indirect time-of-flight camera using a grating-based subpixel encoder and depth-regularizing compressive reconstruction

Hodaka Kawachi, Tomoya Nakamura, Kazuya Iwata, Yasushi Makihara, and Yasushi Yagi
Opt. Continuum 2(6) 1368-1383 (2023)

Previous Article Next Article

Data availability

The public Middlebury dataset and NYU RGB-D dataset used in this paper are available in Refs. [24] and [25]. The related data of the proposed depth simulation model and SR algorithm presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

24. H. Hirschmuller and D. Scharstein, “Evaluation of cost functions for stereo matching,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2007), pp. 21–28.

25. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor Segmentation and Support Inference from RGBD Images,” in European Conference on Computer Vision (ECCV) (2012), pp. 746–760.

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Figures (25)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Tables (8)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Equations (37)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Parameter	Value	Unit
Modulated light wavelength $λ$	870	nm
Background light wavelength $λ_{B}$	550	nm
F-number $F_{#}$	2.8	\
Focal length $f_{l e n s}$	8	mm
Total integration time $T_{i n t}$	6	ms
Quantum efficiency $Q_{E}$	0.25	\
Pixel pitch $P_{p i x e l}$	6	µm
Modulated light transmittance rate $r_{t}$	0.86	\
Background light attenuation rate $r_{b}$	0.001	\

Parameter	Value	Unit
Modulated light wavelength $λ$	850	nm
Modulation frequency $f_{m}$	40	MHz
Radiant intensity $A$	300	mW/sr
Aperture factors $F$	1.6	\
Integration $T_{i n t}$	8	ms
Pixel pitch $P_{p i x e l}$	15	µm
Pixel array	$320 \times 240$	\

	Aloe
	Scale Factor
	RMSE(m)				PSNR(dB)
Algorithm	X2	X4	X8	X16	X2	X4	X8	X16
BI	0.0266	0.0359	0.0466	0.0608	127.8318	125.2276	122.9617	120.6514
JBU	0.0329	0.0480	0.0691	0.0929	125.9855	122.7046	119.5399	116.9692
GF	0.0273	0.0353	0.0625	0.0665	127.6062	125.3740	120.4119	119.8730
NAFDU	0.0266	0.0362	0.0508	0.0704	127.8318	125.1553	122.2122	119.3780
JTF	0.0214	0.0268	0.0372	0.0536	129.7212	127.7668	124.9186	121.7462
Li et al. [19]	0.0239	0.0294	0.0404	0.0572	128.7615	126.9625	124.2018	121.1815
Song et al. [20]	0.0349	0.0401	0.0520	0.0693	125.4730	124.2666	122.0094	119.5148
Proposed	0.0179	0.0199	0.0205	0.0225	131.2724	130.3524	130.0943	129.2858

	Flowerpots
	Scale Factor
	RMSE(m)				PSNR(dB)
Algorithm	X2	X4	X8	X16	X2	X4	X8	X16
BI	0.0347	0.0490	0.0656	0.0876	125.5229	122.5255	119.9914	117.4794
JBU	0.0411	0.0582	0.0724	0.0926	124.0526	121.0310	119.1347	116.9972
GF	0.0355	0.0459	0.0806	0.0875	125.3249	123.0932	118.2028	117.4893
NAFDU	0.0264	0.0334	0.0455	0.0611	127.8974	125.8545	123.1692	120.6086
JTF	0.0221	0.0271	0.0347	0.0482	129.4416	127.6701	125.5229	122.6685
Li et al. [19]	0.0257	0.0266	0.0345	0.0473	128.1308	127.8318	125.5731	122.8322
Song et al. [20]	0.0333	0.0375	0.0455	0.0590	125.8806	124.8488	123.1692	120.9124
Proposed	0.0183	0.0198	0.0213	0.0229	131.0804	130.3961	129.7618	129.1327

	Lampshade
	Scale Factor
	RMSE(m)				PSNR(dB)
Algorithm	X2	X4	X8	X16	X2	X4	X8	X16
BI	0.0240	0.0328	0.0409	0.0495	128.7252	126.0120	124.0950	122.4374
JBU	0.0365	0.0445	0.0551	0.0731	125.0836	123.3623	121.5064	119.0511
GF	0.0239	0.0298	0.0446	0.0492	128.7615	126.8451	123.3428	122.4902
NAFDU	0.0215	0.0242	0.0326	0.0462	129.6807	128.6532	126.0651	123.0366
JTF	0.0205	0.0218	0.0255	0.0332	130.0944	129.5603	128.1987	125.9067
Li et al. [19]	0.0210	0.0213	0.0261	0.0369	129.8851	129.7619	127.9967	124.9889
Song et al. [20]	0.0227	0.0265	0.0342	0.0467	129.2089	127.8645	125.6489	122.9431
Proposed	0.0156	0.0167	0.0178	0.0209	132.4669	131.8751	131.3210	129.9265

	Kitchen
	Scale Factor
	RMSE(m)				PSNR(dB)
Algorithm	X2	X4	X8	X16	X2	X4	X8	X16
BI	0.0365	0.0504	0.0645	0.0816	125.0836	122.2808	120.1382	118.0956
JBU	0.0403	0.0512	0.0704	0.0938	124.2233	122.1440	119.3780	116.8854
GF	0.0371	0.0446	0.0541	0.0672	124.9419	123.3427	121.6655	119.7820
NAFDU	0.0322	0.0427	0.0533	0.0667	126.1723	123.7209	121.7949	119.8469
JTF	0.0247	0.0279	0.0384	0.0539	128.4755	127.4173	124.6428	121.6976
Li et al. [19]	0.0256	0.0305	0.0422	0.0564	128.1646	126.6434	123.8232	121.3038
Song et al. [20]	0.0354	0.0408	0.0517	0.0660	125.3494	124.1162	122.0596	119.9385
Proposed	0.0223	0.0239	0.0258	0.0279	129.3633	128.7615	128.0970	127.4173

	Bedroom
	Scale Factor
	RMSE(m)				PSNR(dB)
Algorithm	X2	X4	X8	X16	X2	X4	X8	X16
BI	0.0438	0.0503	0.0659	0.0881	123.4999	122.2981	119.9517	117.4299
JBU	0.0509	0.0595	0.0731	0.0935	122.1951	120.8391	119.0511	116.9132
GF	0.0372	0.0479	0.0662	0.0876	124.9186	122.7227	119.9123	117.4793
NAFDU	0.0287	0.0346	0.0467	0.0632	127.1718	125.5479	122.9431	120.3151
JTF	0.0251	0.0294	0.0388	0.0542	128.3359	126.9625	124.5528	121.6494
Li et al. [19]	0.0276	0.0302	0.0382	0.0515	127.5112	126.7293	124.6881	122.0933
Song et al. [20]	0.0353	0.0398	0.0495	0.0655	125.3739	124.3318	122.4373	120.0046
Proposed	0.0229	0.0241	0.0266	0.0290	129.1327	128.6891	127.8318	127.0815

	Experiment Images
Algorithm	Aloe	Flowerpots	Lampshade
BI	0.21	0.22	0.19
JBU	8.52	8.54	8.44
GF	0.43	0.44	0.39
NAFDU	10.68	10.74	10.59
JTF	8.64	8.69	8.53
Li et al. [19]	17.56	17.62	17.49
Song et al. [20]	14.84	14.88	14.75
Proposed	8.83	8.87	8.75

	Kitchen
	Scale Factor
	RMSE(m)				PSNR(dB)
Algorithm	X2	X4	X8	X16	X2	X4	X8	X16
BI	0.0365	0.0504	0.0645	0.0816	125.0836	122.2808	120.1382	118.0956
JBU	0.0403	0.0512	0.0704	0.0938	124.2233	122.1440	119.3780	116.8854
GF	0.0371	0.0446	0.0541	0.0672	124.9419	123.3427	121.6655	119.7820
NAFDU	0.0322	0.0427	0.0533	0.0667	126.1723	123.7209	121.7949	119.8469
JTF	0.0247	0.0279	0.0384	0.0539	128.4755	127.4173	124.6428	121.6976
Li et al. [19]	0.0256	0.0305	0.0422	0.0564	128.1646	126.6434	123.8232	121.3038
Song et al. [20]	0.0354	0.0408	0.0517	0.0660	125.3494	124.1162	122.0596	119.9385
Proposed	0.0223	0.0239	0.0258	0.0279	129.3633	128.7615	128.0970	127.4173

Abstract

Data availability

Cited By

Figures (25)

Tables (8)

Equations (37)

Applied Optics