On the space-time statistics of motion pictures

Dae Yeol Lee; Hyunsuk Ko; Jongho Kim; Alan C. Bovik

doi:10.1364/JOSAA.413772

Journal of the Optical Society of America A
Vol. 38,
Issue 7,
pp. 908-923
(2021)
•https://doi.org/10.1364/JOSAA.413772

On the space-time statistics of motion pictures

Dae Yeol Lee, Hyunsuk Ko, Jongho Kim, and Alan C. Bovik

Not Accessible

Your library or personal account may give you access

Get PDF
Email
Share
Get Citation
Copy Citation Text
Dae Yeol Lee, Hyunsuk Ko, Jongho Kim, and Alan C. Bovik, "On the space-time statistics of motion pictures," J. Opt. Soc. Am. A 38, 908-923 (2021)

Export Citation
- BibTex
- Endnote (RIS)
- HTML
- Plain Text
Citation alert
Save article

Check for updates

Abstract

It is well known that natural images possess statistical regularities that can be captured by bandpass decomposition and divisive normalization processes that approximate early neural processing in the human visual system. We expand on these studies and present new findings on the properties of space-time natural statistics that are inherent in motion pictures. Our model relies on the concept of temporal bandpass (e.g., lag) filtering in lateral geniculate nucleus (LGN) and area V1, which is similar to smoothed frame differencing of video frames. Specifically, we model the statistics of the differences between adjacent or neighboring video frames that have been slightly spatially displaced relative to one another. We find that when these space-time differences are further subjected to locally pooled divisive normalization, statistical regularities (or lack thereof) arise that depend on the local motion trajectory. We find that bandpass and divisively normalized frame differences that are displaced along the motion direction exhibit stronger statistical regularities than for other displacements. Conversely, the direction-dependent regularities of displaced frame differences can be used to estimate the image motion (optical flow) by finding the space-time displacement paths that best preserve statistical regularity.

Full Article | PDF Article

More Like This

What do we perceive from motion pictures? A computational account

Loong-Fah Cheong and Xu Xiang
J. Opt. Soc. Am. A 24(6) 1485-1500 (2007)

Video quality assessment using a statistical model of human visual speed perception

Zhou Wang and Qiang Li
J. Opt. Soc. Am. A 24(12) B61-B69 (2007)

Motion picture of magnified light pulse propagation with extending recordable time of digital light-in-flight holography

Tomoyoshi Inoue, Takamasa Aoyama, Yu Sawashima, Kenzo Nishio, Toshihiro Kubota, and Yasuhiro Awatsuji
Appl. Opt. 61(5) B206-B214 (2022)

Data Availability

Data underlying the results presented in this paper are available in [45].

45. D. Lee, “Space-time-video-regularity,” GitHub, 2021, https://github.com/daelee711/Space-Time-Video-Regularity.

Cited By

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Figures (13)

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Tables (3)

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

Equations (16)

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

Contact your librarian or system administrator
or
Login to access Optica Member Subscription

	Parameters		Contents
		Displacement	Grove2		Grove3		Hydrangea		Rubberwhale		Urban2		Urban3
Methods	Patch Size	Range	AE^a	Time [s]	AE	Time [s]	AE	Time [s]	AE	Time [s]	AE	Time [s]	AE	Time [s]
Proposed (SDN)	$51 \times 51$	$[- 8, 8]$	36.66	7.12	27.35	7.49	27.78	5.94	43.09	5.67	72.33	8.78	33.55	7.32
	$61 \times 61$	$[- 10, 10]$	30.21	7.53	30.35	7.61	27.87	6.43	40.35	5.97	71.23	7.61	29.66	7.50
	$71 \times 71$	$[- 10, 10]$	25.62	6.36	30.91	6.17	17.41	4.60	37.60	4.64	65.88	6.05	31.11	6.53
	$81 \times 81$	$[- 12, 12]$	23.71	5.90	25.44	5.79	22.10	4.77	33.73	4.84	67.44	6.08	27.52	5.77
	$91 \times 91$	$[- 14, 14]$	17.63	7.96	27.26	7.97	23.04	5.56	43.01	5.46	59.07	8.19	21.48	8.12
	$101 \times 101$	$[- 16, 16]$	18.84	7.76	32.84	7.21	22.58	4.80	36.59	4.54	70.34	7.24	26.98	7.72
Horn–Schunck	$i t e r = 6$		42.01	8.60	37.52	8.59	19.45	6.34	11.25	6.31	54.19	8.59	43.02	8.54
	$i t e r = 12$		33.62	17.14	28.21	17.20	8.64	12.57	10.95	12.57	50.14	17.09	33.77	16.93
	$i t e r = 18$		31.10	25.61	23.52	25.77	8.11	18.88	10.85	18.80	47.93	25.77	29.29	25.31
Black–Anandan	$p y r = 2$ , $i t e r = 2$		23.90	19.20	19.56	19.22	8.74	13.90	10.32	13.88	48.56	19.26	27.63	19.17
	$p y r = 2$ , $i t e r = 3$		21.25	28.61	16.71	29.60	8.32	21.13	9.97	20.39	47.15	28.50	23.41	28.87
	$p y r = 2$ , $i t e r = 4$		18.51	38.35	15.11	38.15	8.07	27.45	9.73	27.36	45.71	37.56	20.99	37.49

	Parameters		Contents
		Displacement	Grove2		Grove3		Hydrangea		Rubberwhale		Urban2		Urban3
Methods	Patch Size	Range	EE^a	Time [s]	EE	Time [s]	EE	Time [s]	EE	Time [s]	EE	Time [s]	EE	Time [s]
Proposed (SDN)	$51 \times 51$	$[- 8, 8]$	1.96	7.12	2.33	7.49	2.21	5.94	1.43	5.67	8.98	8.78	4.47	7.32
	$61 \times 61$	$[- 10, 10]$	1.80	7.53	2.61	7.61	2.12	6.43	1.36	5.97	8.91	7.61	3.65	7.50
	$71 \times 71$	$[- 10, 10]$	1.60	6.36	2.87	6.17	1.64	4.60	1.16	4.64	8.49	6.05	3.77	6.53
	$81 \times 81$	$[- 12, 12]$	1.57	5.90	2.82	5.79	2.02	4.77	1.23	4.84	7.72	6.08	3.49	5.77
	$91 \times 91$	$[- 14, 14]$	1.40	7.96	2.95	7.97	2.06	5.56	1.58	5.46	7.71	8.19	2.87	8.12
	$101 \times 101$	$[- 16, 16]$	1.42	7.76	3.08	7.21	2.22	4.80	1.40	4.54	8.05	7.24	3.27	7.72
Horn–Schunck	$i t e r = 6$		2.00	8.60	2.95	8.59	1.84	6.34	0.38	6.31	7.97	8.59	5.91	8.54
	$i t e r = 12$		1.54	17.14	2.49	17.20	0.71	12.57	0.36	12.57	7.81	17.09	5.22	16.93
	$i t e r = 18$		1.42	25.61	2.23	25.77	0.68	18.88	0.36	18.80	7.67	25.77	4.77	25.31
Black–Anandan	$p y r = 2$ , $i t e r = 2$		1.13	19.20	2.01	19.22	0.70	13.90	0.34	13.88	7.82	19.26	4.84	19.17
	$p y r = 2$ , $i t e r = 3$		1.02	28.61	1.73	29.60	0.68	21.13	0.33	20.39	7.75	28.50	4.27	28.87
	$p y r = 2$ , $i t e r = 4$		0.91	38.35	1.56	38.15	0.68	27.45	0.32	27.36	7.61	37.56	3.84	37.49

	Contents
	000000		000011		000016		000026		000030		Overall
Methods	AE	EE	AE	EE	AE	EE	AE	EE	AE	EE^a	AE^b	EE
Horn–Schunck	12.3	9.3	11.9	11.7	14.2	12.6	14.8	11.7	24.2	13.9	14.5	11.7
Black–Anandan	12.5	9.3	11.9	10.6	14.3	11.2	14.9	11.2	23.1	13.2	14.4	10.9
Proposed (TDN)	25.4	13.1	37.7	15.4	32.9	19.2	34.1	19.2	43.3	15.8	34.5	16.8
Proposed (STDN)	28.5	14.2	36.3	15.3	32.6	21.1	33.6	18.8	38.5	15.8	33.9	17.0
Non-displaced	67.4	36.4	69.7	40.4	70.1	31.6	66.9	26.1	58.7	26.1	67.3	36.7

	Parameters		Contents
		Displacement	Grove2		Grove3		Hydrangea		Rubberwhale		Urban2		Urban3
Methods	Patch Size	Range	AE^a	Time [s]	AE	Time [s]	AE	Time [s]	AE	Time [s]	AE	Time [s]	AE	Time [s]
Proposed (SDN)	$51 \times 51$	$[- 8, 8]$	36.66	7.12	27.35	7.49	27.78	5.94	43.09	5.67	72.33	8.78	33.55	7.32
	$61 \times 61$	$[- 10, 10]$	30.21	7.53	30.35	7.61	27.87	6.43	40.35	5.97	71.23	7.61	29.66	7.50
	$71 \times 71$	$[- 10, 10]$	25.62	6.36	30.91	6.17	17.41	4.60	37.60	4.64	65.88	6.05	31.11	6.53
	$81 \times 81$	$[- 12, 12]$	23.71	5.90	25.44	5.79	22.10	4.77	33.73	4.84	67.44	6.08	27.52	5.77
	$91 \times 91$	$[- 14, 14]$	17.63	7.96	27.26	7.97	23.04	5.56	43.01	5.46	59.07	8.19	21.48	8.12
	$101 \times 101$	$[- 16, 16]$	18.84	7.76	32.84	7.21	22.58	4.80	36.59	4.54	70.34	7.24	26.98	7.72
Horn–Schunck	$i t e r = 6$		42.01	8.60	37.52	8.59	19.45	6.34	11.25	6.31	54.19	8.59	43.02	8.54
	$i t e r = 12$		33.62	17.14	28.21	17.20	8.64	12.57	10.95	12.57	50.14	17.09	33.77	16.93
	$i t e r = 18$		31.10	25.61	23.52	25.77	8.11	18.88	10.85	18.80	47.93	25.77	29.29	25.31
Black–Anandan	$p y r = 2$ , $i t e r = 2$		23.90	19.20	19.56	19.22	8.74	13.90	10.32	13.88	48.56	19.26	27.63	19.17
	$p y r = 2$ , $i t e r = 3$		21.25	28.61	16.71	29.60	8.32	21.13	9.97	20.39	47.15	28.50	23.41	28.87
	$p y r = 2$ , $i t e r = 4$		18.51	38.35	15.11	38.15	8.07	27.45	9.73	27.36	45.71	37.56	20.99	37.49

	Parameters		Contents
		Displacement	Grove2		Grove3		Hydrangea		Rubberwhale		Urban2		Urban3
Methods	Patch Size	Range	EE^a	Time [s]	EE	Time [s]	EE	Time [s]	EE	Time [s]	EE	Time [s]	EE	Time [s]
Proposed (SDN)	$51 \times 51$	$[- 8, 8]$	1.96	7.12	2.33	7.49	2.21	5.94	1.43	5.67	8.98	8.78	4.47	7.32
	$61 \times 61$	$[- 10, 10]$	1.80	7.53	2.61	7.61	2.12	6.43	1.36	5.97	8.91	7.61	3.65	7.50
	$71 \times 71$	$[- 10, 10]$	1.60	6.36	2.87	6.17	1.64	4.60	1.16	4.64	8.49	6.05	3.77	6.53
	$81 \times 81$	$[- 12, 12]$	1.57	5.90	2.82	5.79	2.02	4.77	1.23	4.84	7.72	6.08	3.49	5.77
	$91 \times 91$	$[- 14, 14]$	1.40	7.96	2.95	7.97	2.06	5.56	1.58	5.46	7.71	8.19	2.87	8.12
	$101 \times 101$	$[- 16, 16]$	1.42	7.76	3.08	7.21	2.22	4.80	1.40	4.54	8.05	7.24	3.27	7.72
Horn–Schunck	$i t e r = 6$		2.00	8.60	2.95	8.59	1.84	6.34	0.38	6.31	7.97	8.59	5.91	8.54
	$i t e r = 12$		1.54	17.14	2.49	17.20	0.71	12.57	0.36	12.57	7.81	17.09	5.22	16.93
	$i t e r = 18$		1.42	25.61	2.23	25.77	0.68	18.88	0.36	18.80	7.67	25.77	4.77	25.31
Black–Anandan	$p y r = 2$ , $i t e r = 2$		1.13	19.20	2.01	19.22	0.70	13.90	0.34	13.88	7.82	19.26	4.84	19.17
	$p y r = 2$ , $i t e r = 3$		1.02	28.61	1.73	29.60	0.68	21.13	0.33	20.39	7.75	28.50	4.27	28.87
	$p y r = 2$ , $i t e r = 4$		0.91	38.35	1.56	38.15	0.68	27.45	0.32	27.36	7.61	37.56	3.84	37.49

Abstract

Data Availability

Cited By

Figures (13)

Tables (3)

Equations (16)

Journal of the Optical Society of America A