1. Introduction

Optical fiber has been a media commonly used in communication, among which multi-mode fiber (MMF) is especially concerned in the field of image transmission for its ability to transmit multiple spatial modes in parallel [1,2]. However, due to modal dispersion and mode coupling, spatially scattered images will be generated through MMF transmission, which raises an urgent demand of image reconstruction methods to restore the effective information. A variety of methods used in information transmission through MMF has been developed, either for light focusing or for image projection, including using phase conjugation [3–7] via the knowledge of phase recorded at the distal end and calibrating transmission matrix (TM) [8–11] of MMF channel to restore the image in reverse. Iterative optimization algorithms can also generate desired images by optimizing the phase of the input field in the MMFs [12–15]. Apart from the ever-present scattering characteristic, researches have shown that MMF is highly variable and random channel subject to environmental changes such as temperature fluctuation and geometric deformation, meantime displaying a time-varying characteristic. These disturbances in MMF system raise difficulties for methods reported above to be applied in unstable conditions. For example, in commonly-used TM methods, recalibrations are necessary before each acquisition after a long-time interval [16].

The last decade has witnessed rapid advancements of deep learning in dealing with ill-posed image reconstruction problems [17]. Recently, researchers have developed neural networks to restore images from scattering MMF media [18–22]. It has been reported that neural network has the ability to generalize on different channel states when the 35 cm-MMF was under constantly changing characteristic variations [23]. Long-term transmission for over one week through 1 m MMF was demonstrated with two different neural networks [24]. However, most MMFs applied to the image transmission systems in the above reports are under 10 m, and increase of the fiber length can aggravate the dispersion and time-varying characteristics of MMF channel, which brings difficulties for image reconstruction. To address the problem, Fan Pengfei et al. [25] proposed a semi-supervised learning approach in MMFs up to 1 km, achieving nearly errorless image transmission over a 200-second period. However, the approach is based on the premise that the model is valid within a short time interval, so it needs continuous data acquisition to accommodate gradual system drift. In addition, the corresponding models need to be trained from scratch at each time, which is time-consuming and restricts the possibility of online transmission.

In digital signal transmission through single-mode fibers (SMFs), due to phase shift of each subcarrier over time, it is necessary to estimate the channel frequency-offset with periodic phase reference. The most common method is to calibrate the transmission characteristics of the system through a large number of data pairs (i.e., preamble) at the initial stage, and then insert reference signals (i.e., pilots) of known labels at intervals during the actual transmission process to assist dynamic channel correction. The pilot-aided approach has been widely used in coherent passive optical network (CPON) to improve the transmission stability of digital signal [26–30].

Here we introduce the concept of spatial pilot to address the problem of spatial information transmission under dynamic long MMF channels. Distributed spatial pilot frames are inserted into the transmitting data frame sequence, serving as the reference for regular transmission channel recalibration. To facilitate online transmission, we propose a fast-adapt learning strategy for the pilot-aided framework in MMF, which freezes 98% of the total parameters and quickly updates the network model using a small amount of pilot frames. Additionally, two different ways for pilot frame insertion is designed to apply for different transmission conditions. To demonstrate the framework, we pre-train a neural network on a large dataset collected through a 100 m MMF with 15 hours time span, and achieve >92% accuracy spatial transmission with the portion of pilot frames less than 1.8%. The fast-adapt strategy produced 70% convergency speed-up compared with traditional fine-tuning approach to support online network update. Transmissions under various scenarios, including long-term continuous transmission, burst transmission and transmission after fiber re-plugging, are demonstrated with two pilot-insertion strategies, implying its good practical applicability.

2. Experimental setup

The experimental setup of MMF spatial transmission is shown in Fig. 1. The continuous-wave laser beam at wavelength 561 nm is expanded by a pair of achromatic lenses L1 and L2 (${f_1}$=10 mm, ${f_2}$=100 mm), and projected onto a Digital Micromirror Device (DMD, ViALUX V-7001). The DMD is able to modulate the amplitude distribution of the light beam by switching the micromirrors “ON” or “OFF”, corresponding to input binary values “1” or “0” at each pixel. We select the MNIST dataset for the image transmission experiment. After resizing the shape to 32 ${\times} $ 32-pixel, the images are binarized and loaded to DMD. Each transmission channel (corresponding to a pixel in patterns) occupies 12 ${\times} $ 12 DMD micromirror pixels, resulting in an effective DMD area of 384 ${\times} $ 384-pixel. The incident laser beam carrying spatially encoded information is consequently coupled into a MMF (SI, 0.22 NA, ∅︀=200µm) using an objective lens (Obj1, Nikon, 20X, 0.25 NA).

 

Fig. 1. The experimental system of spatial transmission through unstable MMF.

Download Full Size | PDF

The incident light field propagates through the scattering MMF channel and suffers strong spatial-polarization mode distortion and coupling. The output speckle images at the distal end of the fiber are imaged by another objective lens (Obj2, Nikon, 20X, 0.25 NA) onto a monochromatic CMOS camera (MER2-230-167U3M). The changing speckles illustrated on the top right corner of Fig. 1 are the recorded output images generated by using the same pattern as the input at different timepoints, showing the time-varying characteristic of the MMF transmission system.

To quantify the time-varying characteristic of the MMF system in further experiments, we conducted a stability test. The stability over time is measured by the SSIM to an initial speckle pattern recorded with the same input modulated by the DMD, as shown in the lower right corner of Fig. 1. After about 100s, the SSIM has dropped to 0.6, showing a non-negligible degree of system drift in 100 m long fibers.

3. Principle

3.1 Spatial pilot-aided fast-adapt framework

As shown in Fig. 2, inspired by the concept of pilot in SMF data transmission, we propose a spatial pilot-aided fast-adapt framework by introducing spatial pilots in MMF spatial information transmission. The advanced neural network is leveraged to model the inverse process of scattering spatial transmission. Here, we denote the pre-train neural network model as ${M_0}$, which is initialized by training from the preamble data. The preamble is composed of a large number of known data pairs, aiming to cover as many of the states of the MMF channel as possible, as to calibrate the general MMF channel.

 

Fig. 2. Schematic illustration of image reconstruction with the spatial pilot-aided fast-adapt framework.

Download Full Size | PDF

In follow-up data transmission, the image data stream will be divided into consecutive sequences of data frames of fixed length, at the beginning of which paired pilot images ${P_i}$ are inserted. To characterize the high-dimensional transmission model in long MMF channel, spatial pilot are a sequence of known images of the same category and same pixel size with the transmission data. Paired pilot images in each data sequence will attach useful knowledge of the current channel states to the pre-train model ${M_0}$, thus generating a series of tuned model ${M_1},\; {M_2},\; {M_3},\; \ldots ,\; {M_N}$ to track the transient transmission characteristics.

During the process of generating tuned models, a fast-adapt strategy is applied that only a few parameters of ${M_0}$ are trainable while the parameters left are frozen to keep their original values. In this case, $\; {M_t},\; t \ge 1$ can be written as ${M_t}({{u_t},{v_0}} )$ in which ${v_0}$ represents the frozen parameters and ${u_t}$ represents the remaining parameters varying according to channel states. All the tuned models are derived from ${M_0}$ and do not depend on the transmission model of the previous status, which enables a fast and direct tracking of channel model without constant observation of state changes. Benefiting from the fast-adapt strategy, tuning time of M_0 will be greatly reduced to the order of tens of seconds, which makes it possible for M_0 to online tuning while transmitting data.

3.2 Network architecture and implementation details

We illustrate the architecture of the neural network and specific structural parameters in Fig. 3. The input layer feeds speckle with a size of 150 × 150-pixel into the neural network, following by three convolutional blocks, which consist of one convolutional layer, batch normalization (BN) and the rectified linear unit (ReLU) nonlinear activation. Three convolutional layers in these blocks have a same kernel size of 3 × 3, which help to abstract the relevance implied in the input data by extracting features. A max pooling layer is applied after the first convolutional block to decrease computation and balance the parameter distribution. The two dense layers combine all the features of images learned from the previous layer to make a general prediction of the object.

 

Fig. 3. Architecture diagram of the CNN model for image reconstruction. The specific values are: h = w = 150, c = 32, f1 = 90 × 90, f2 = 32 × 32.

Download Full Size | PDF

All of the parameters join the pre-training process. In the fine-tuning process, we froze the first fully-connected layer of the network, which has 98% of the total number of parameters, thus decreasing the training time to facilitate online transmission. The Binary Cross Entropy cost was selected for the loss function. In addition, AdamW optimizers with the learning rates of $5 \times \textrm{1}{\textrm{0}^{\textrm{ - 4}}}$ and $6 \times \textrm{1}{\textrm{0}^{\textrm{ - 4}}}$ is selected for pre-training and fine-tuning with paired pilot frames, respectively. The decaying weight is set to $\textrm{1}{\textrm{0}^{\textrm{ - 4}}}$. Based on the convergence of the validation losses, models were designed to stop at 50 epochs. All models were realized in TensorFlow on a workstation equipped with a NVIDIA RTX 3080 graphics processing unit (GPU).

4. Results

4.1 Performance test of the pre-train model

To begin with, we train the pre-train model ${M_0}$ by collecting long-term measured data and verify its reconstruction performance. To obtain a general channel model covering as many channel states as possible, we perform rotation enhancement on the MNIST test set to expand the data size from 10000 to 50000, and project the enhanced dataset onto the DMD 3 frames per second for 15 hours. Before pre-training, 500 data pairs are randomly singled out from the dataset for further performance test. After the training process of pre-train model, we measure the reconstruction accuracy of the 500 test images. We choose the pixel-wise prediction accuracy as the image quality evaluation metric, which is defined as the percentage of correctly predicted pixels within one 32 ${\times} $ 32-pixel input image. Accuracies of the 500 reconstructed images are plot in Fig. 4(a) with a mean value of 93.6%. Examples of reconstructed images are presented in Fig. 4(b), which indicates that the pre-train model performs well on the unseen data acquired within the period that we collected the training data.

 

Fig. 4. Evaluation on the pre-train model. (a) Accuracies of 500 reconstructed images obtained from pre-train model test. (b) Example reconstructed images in (a). (c) Evaluation on the pre-train model without model fine-tuning. (d) Reconstructed images at five timepoints showing deteriorating performance.

Download Full Size | PDF

We further applied the pre-train model for recovering newly collected data 30 seconds after the pre-training data collection. As shown in Fig. 4(c), the reconstruction accuracy using the fixed pre-train model ${M_0}$ will slowly decline within 30 min after the pre-training data collection. After about 15 minutes, reconstruction accuracy has dropped to 60%, where the reconstructed images in Fig. 4(d) are already indistinguishable for handwritten digits. This suggests that a single fixed model doesn’t work well on long optical fibers and regular model update is necessary for long-term transmission.

4.2 Stable transmission with spatial pilot-aided strategy

Our proposed spatial-pilot framework provides reference for supervised network tuning, so as to allow robust and accurate image transmission through varying MMFs over long time. Here, we conduct a comparison experiment of transmission with and without spatial pilot, to validate the feasibility of pilot-aided strategy in spatial information transmission. We have chosen the dataset acquired immediately after collection of pre-train dataset in Fig. 4(c) for stable transmission test, which is divided into several 400s-duration test datasets consisting of 40000 patterns from the transmission set are loaded onto DMD at a frame rate of 100 Hz. In each 400s dataset, 2% of the 40000 image frames at the beginning are selected as labeled pilot frames for fine-tuning the pre-train model. Figure 5(a) illustrates the transmission accuracy using the pre-train model ${M_0}$ and the pilot-aided fine-tuned model ${M_1}$ to ${M_5}$ during the whole time period. The curve is smoothed with a window size of 50. With spatial pilot, the transmission accuracy curve eliminates original downward trend and maintains a mean accuracy of 92.3%. Reconstructed images of ${M_0}$ and tuned models have quite obvious quality gap, as shown in Fig. 5(b), where images transmitted with pilots have only few error pixels and images transmitted without pilots are distinguishable over time. Although ${M_0}$ has generalized on a variety of channel states, it cannot work well for transmissions long time after pre-train dataset collection. With the help of spatial pilots, transmission model is able to adapt to the varying transmission characteristics, thus maintaining high accuracy during the transmission period.

 

Fig. 5. (a) Comparison in transmission accuracy of using pre-train and pilot-aided fine-tuned model. (b) Example reconstructed images in (a) after 800s.

Download Full Size | PDF

4.3 Evaluation on accuracy and time cost of the fast-adapt strategy

When regular update of the model is unavoidable, the fast-adapt strategy shows its advantages, mainly in the accuracy of reconstruction and greatly shortened training time. To quantify the advantages of our proposed fast-adapt approach, we measure the reconstruction accuracy and time cost for network training with different network update approaches using the same pilot dataset. Two baseline training approaches are compared: the first one retrains the network model with the pilot data from scratch, the other one fine-tunes all the network parameters.

We experiment on three training datasets of various pilot ratios ${\textrm{N}_\textrm{t}}\textrm{/N}$, where N is the total number of testing images and ${\textrm{N}_\textrm{t}}$ is the number of data pairs inserted as pilots. After 50 epochs, we evaluate their performance on the test dataset. Figure. 6(a) demonstrates the reconstruction accuracy of the three models with different ${\textrm{N}_\textrm{t}}\textrm{/N}$ ratios over time. The accuracy results of total 400s are divided to five groups corresponding to five time periods of 80s. For all ${\textrm{N}_\textrm{t}}\textrm{/N}$, our fast-adapt strategy has the highest transmission accuracy, which is slightly higher than model trained from scratch, and transmission accuracy of model fine-tuned without frozen layer is 0.5%-1.7% lower than the two approaches above. For example, in the 400s test with ${\textrm{N}_\textrm{t}}{/\textrm{N}\; = \; 2\%}$, the mean reconstruction accuracies of the three models are 92.8%, 92.7%, 92.2% from high to low, respectively. Figure 6(b)-(c) displays some predicted images at 150s and 300s predicted by the three different models when ${\textrm{N}_\textrm{t}}{/\textrm{N}\; = \; 2\%}$, from which we can see that our fast-adapt strategy better preserve the pixel-level details (e.g., the digit ‘5’ reconstructed at T = 300s). When the proportion of training data increases, the reconstruction accuracy of each model has some improvement with similar degrees.

 

Fig. 6. (a) Reconstruction accuracy columns of different tuning strategies in a single cycle of 400s with ${\textrm{N}_\textrm{t}}\textrm{/N}$ = 1.8%, 2.0%, 2.5%. (b) The reconstructed images using the three update models at T1 = 150s. (c) The reconstructed images using three update models at T2 = 300s.

Download Full Size | PDF

Our strategy uses only ∼8.5×$\textrm{1}{\textrm{0}^\textrm{6}}$ trainable parameters, about 2% of the total parameters of the neural network, bringing another advantage of reducing model adaptation time. As presented in Fig. 7, the required training time for one time cycle with 1.8%∼2.5% spatial pilots are recorded and compared. Our frozen strategy achieves more than 70% computation time reduction under the same configuration environment, compared to training from scratch and without frozen layer approaches. The fast adaption property guarantees online network tuning to support long-term continuous transmission.

 

Fig. 7. Comparison of training time using different tuning strategies at three pilot ratios.

Download Full Size | PDF

4.4 Analysis on different pilot-insertion strategies

In the above experiment, pilot frames are only inserted at the beginning of the transmission data sequence. For practical long-term transmission, multiple groups of spatial pilots will be inserted at intervals. Next, we design and analyze on two different strategies in terms of pilot distribution, and their implementation details are demonstrated in Fig. 8(a). The naive one inserts all spatial pilots within a time cycle to the beginning of the data frame, which is termed as the head-first pilot insertion strategy (P_HF). Another strategy inserts discrete single pilot frame equally spaced and distributed within the time cycle, which is termed as interleaved pilot insertion (P_IL). Both of the two strategies only fine-tune once within a single cycle and they share a same tuning time with the same type and amount of data. Because P_HF inserts pilots in the beginning of the cycle, the tuning process begins immediately after paired data acquisition. The difference is that the tuning process of P_IL needs to wait until all data throughout the whole cycle is acquired before it can start.

 

Fig. 8. (a) Schematic diagram of the headfirst and interleaved pilot insertion methods. (b) Reconstruction accuracy of two insertion methods in a single cycle of 400s.

Download Full Size | PDF

We use the same dataset applied in Section 4.2 for experiment and 2% image pairs are selected as pilots to fine-tune the pre-train model, making the remaining data to serve as the test portion. Figure. 8(b) illustrates image reconstruction accuracy of P_HF and P_IL in 400s. Their mean accuracies and mean absolute derivations (MADs) are also plotted on the graphs respectively. The P_IL and P_HF have a similar average accuracy, meanwhile displaying different curve trends. Because P_HF obtains pilot frames for fine-tuning at the beginning of the cycle, it tends to perform better at the beginning of the test set. However, its subsequent test shows fluctuant results, which could be seen clearly from the trend of curve and its MAD. However, the training data of P_IL covers the whole period of the test, which makes the reconstruction results more stable during the whole transmission period. Based on the experimental results, P_HF is applicable to scenarios where updates are more frequent, while P_IL is suitable for scenarios with high requirement for transmission stability.

4.5 Long-term continuous transmission

To further prove that our pilot-aided fast-adapt framework is applicable in a variety of situations, we conducted a series of long-term transmission tests under various scenarios with ${\textrm{N}_\textrm{t}}\textrm{/N}$=2%. Firstly, we demonstrated the long-term continuous transmission lasting over half an hour in the unstable 100 m MMF. 40000 digit patterns in the test set are loaded to DMD with the frame rate of 100 Hz and repeated for 5 times. The two insertion methods of P_HF and P_IL are both tested. Figure 9(a) demonstrates the mean transmission accuracies and MADs of 5 update cycles over 2000s. Since the experiment is conducted several days after the pre-training data collection, the pre-train model results in a poor reconstruction accuracy over the period. As a comparison, with the spatial-pilot aided network adaptation, the long-term test of continuous transmission demonstrates equally good performance compared with single update cycle, with a mean accuracy up to 92.6% during long time periods. Figure 9(b) shows the reconstructed images of P_HF and P_IL at five time points, where all the hand-written digitals have high reconstructed quality visually.

 

Fig. 9. Experiment of continuous image transmission over 30 min. (a) Illustration of pilot inserted continuous transmission and the reconstruction accuracy. (b) Reconstructed images at five time points.

Download Full Size | PDF

4.6 Long-term burst mode transmission

Burst transmission is a common scenario in practical applications. Burst transmission is a common scenario in practical applications, such as time-division multiple access (TDMA) systems in the uplink of the coherent access network. Next, we validate the feasibility of our spatial pilot approach in burst transmission mode. Different from the continuous test above, we deliberately set a blank interval of 400s without transmitting data between every two transmission time cycles, as shown in the top inset of Fig. 10(a), forming a transmission dataset of over 1 h. The mean transmission accuracies and MADs of 5 transmission cycles are also illustrated in Fig. 10(a), which shows a similar transmission performance to that of the continuous condition. A high accuracy of 92.7% in average is preserved throughout the transmission. Figure 10(b) displays the reconstructed images of P_HF and P_IL at five certain time points with most pixel-level details restored.

 

Fig. 10. Experiment of burst image transmission over 1 h. (a) Illustration of pilot inserted burst transmission and the reconstruction accuracy. (b) Reconstructed images at five time points.

Download Full Size | PDF

4.7 Transmission after dramatic configuration change

Another common scenario of optical transmission is that MMF channels are dramatically changed under different configurations. Examples of that are that the fiber is removed and re-plugged, which are often involved in system construction and testing, and it’s necessary for optical fibers to be periodically removed from the system for cleaning and other maintenance in actual communication scenarios. To figure out the effectiveness of our spatial pilot-aided fast-adapt strategy in such scenarios, we delivered a test of transmission after fiber re-plugging. Figure 11(a) illustrates the dataset composition over 30 min, where the removing and re-plugging operation takes about 5s between every two update cycles, while the spatial pilots aided network tuning are produced after each round of fiber re-plugging. The mean accuracies and MADs of the 5 update cycles are plot in Fig. 11(b) and Fig. 11(c) displays the reconstructed images of P_HF and P_IL at five certain time points with most pixel-level details restored.

 

Fig. 11. Experiment of robust image transmission with fiber re-plugging over 30 min. (a) Illustration of pilot inserted resumed transmission. (b) Reconstruction accuracy over time. (c) Reconstructed images at five time points.

Download Full Size | PDF

5. Conclusion

This work is meant to provide a deep learning-based general framework for stable transmission of spatial information over long MMFs. The proof-of-concept experiment has demonstrated the no obvious performance degradation of the proposed framework on overcoming the high instability of a specific MMF channel (ø = 200um, 100 m) over 1 h. Theoretically, our approach has no upper limit of application time. Moreover, when a longer MMF is applied, a higher pilot ratio for tuning or slightly more frequent updates may be required, whereas the overall spatial pilot aided fast-adapt framework is still valid.

In summary, we propose a pilot-aided learning framework for robust MMF image transmission. A fast-adapt strategy of freezing 98% of the total parameters is developed to reduce the network-tuning time by 70%, hence facilitating online transmission. We experimentally validate the fast-adapt strategy of hand-written digits transmission through 100 m MMF, showing that only ∼1.8% of the pilot frames are adequate for model updating to maintain stable transmission over hours. Later, we make an exploration of two pilot insertion methods, head-first insertion and interleaved insertion, and discuss their applicable transmission conditions. Finally, a series of long-term transmission test under different scenarios including continuous transmission, burst transmission and transmission after configuration change are delivered to validate the universality and applicability of our strategy.