Abstract

In recent years, convolutional neural networks (CNNs) have enabled ubiquitous image processing applications. As such, CNNs require fast forward propagation runtime to process high-resolution visual streams in real time. This is still a challenging task even with state-of-the-art graphics and tensor processing units. The bottleneck in computational efficiency primarily occurs in the convolutional layers. Performing convolutions in the Fourier domain is a promising way to accelerate forward propagation since it transforms convolutions into elementwise multiplications, which are considerably faster to compute for large kernels. Furthermore, such computation could be implemented using an optical $4f$ system with orders of magnitude faster operation. However, a major challenge in using this spectral approach, as well as in an optical implementation of CNNs, is the inclusion of a nonlinearity between each convolutional layer, without which CNN performance drops dramatically. Here, we propose a spectral CNN linear counterpart (SCLC) network architecture and its optical implementation. We propose a hybrid platform with an optical front end to perform a large number of linear operations, followed by an electronic back end. The key contribution is to develop a knowledge distillation (KD) approach to circumvent the need for nonlinear layers between the convolutional layers and successfully train such networks. While the KD approach is known in machine learning as an effective process for network pruning, we adapt the approach to transfer the knowledge from a nonlinear network (teacher) to a linear counterpart (student), where we can exploit the inherent parallelism of light. We show that the KD approach can achieve performance that easily surpasses the standard linear version of a CNN and could approach the performance of the nonlinear network. Our simulations show that the possibility of increasing the resolution of the input image allows our proposed $4f$ optical linear network to perform more efficiently than a nonlinear network with the same accuracy on two fundamental image processing tasks: (i) object classification and (ii) semantic segmentation.

© 2022 Optical Society of America

Optical frontend for a convolutional neural network

Shane Colburn, Yi Chu, Eli Shilzerman, and Arka Majumdar
Appl. Opt. 58(12) 3179-3186 (2019)

Optical convolutional neural network with atomic nonlinearity

Mingwei Yang, Elizabeth Robertson, Luisa Esguerra, Kurt Busch, and Janik Wolters
Opt. Express 31(10) 16451-16459 (2023)

Training optronic convolutional neural networks on an optical system through backpropagation algorithms

Ziyu Gu, Zicheng Huang, Yesheng Gao, and Xingzhao Liu
Opt. Express 30(11) 19416-19440 (2022)

Classification accuracy improvement of the optical diffractive deep neural network by employing a knowledge distillation and stochastic gradient descent β-Lasso joint training framework

Tao Fang, Jingwei Li, Xiang Zhang, and Xiaowen Dong
Opt. Express 29(26) 44264-44274 (2021)

Optical performance monitoring using lifelong learning with confrontational knowledge distillation in 7-core fiber for elastic optical networks

Xu Zhu, Bo Liu, Jianxin Ren, Xiaorong Zhu, Yaya Mao, Xiangyu Wu, Ying Li, Yongfeng Wu, Lilong Zhao, Tingting Sun, Rahat Ullah, and Yunyun Chen
Opt. Express 30(15) 27109-27122 (2022)