Imaging through scattering media using speckle pattern classification based support vector regression

Hui Chen; Yesheng Gao; Xingzhao Liu; Zhixin Zhou

doi:10.1364/OE.26.026663

1. Introduction

Imaging or focusing through scattering media is a common practice in the fields of biomedicine and security [1–7]. Scattering media would diffuse the object image into unrecognizable speckle pattern. Various methods for retrieving the object image from scattered speckle pattern have been proposed. Wavefront shaping technique has emerged as a powerful tool for inverse scattering by controlling the wavefront of incident light [8–11]. However, these techniques are complex and lengthy, since a detector or an optical/acoustical probe is always necessary in the plane of interests to provide feed-back instructions for wavefront modulation. Phase-retrieval based method utilizes the principle of optical memory effects to translate the inverse scattering problem into phase retrieval problem [12–15]. But phase retrieval algorithms are always dependent on the initial points (so needs to restart several times to obtain a satisfied reconstruction) and noise sensitive (since noise would pollute the measured speckle pattern and introduce bias to Fourier amplitude of object image). Ghost imaging can retrieve the information of an unknown object without a spatial-resolving detector toward it, while a reference beam is necessary and sometimes a calibration is needed [16–19]. Machine-learning based sensing approach was introduced to realize imaging through scattering media without time-consuming feedback modulation and reference arm, and is insensitive to noise enough due to its data-driven character. However, the learned inverse scattering function (ISF) was only effective and specified for reconstruction of images located at the same category as training object-image-and-speckle-pattern pairs (OS pairs in short) [20–22]. Specifically, the ISF learned from OS pairs of class 1 was only able to reconstruct objects of class 1 from their speckle patterns well, while failed to reconstruct objects of other classes, and the reconstructions were always ambiguous profiles of objects of class 1.

Exactly the limited imaging capability of the learned ISF, inspired us to try to make classification to scattered speckle patterns [22]. In this paper, we propose to apply image classification to speckle patterns and then use speckle pattern classification based support vector regression (referenced as SPC-SVR) method for single-shot imaging through scattering media. Object classification and recognition has been a hot topic in optical remote sensing, biomedical imaging, and etc [23–26]. In these fields, the targets for classification or recognition are always the objects themselves. However, in a scattering system, objects would scratch into unrecognizable speckle patterns and become unavailable temporarily. The key for classification is to learn more discriminative features of images and determine the optimal label for the given image data. We will prove mathematically that the autocorrelation matrix of the scattered speckle pattern has the same singular values or matrix as that of the object image, and thus speckle patterns could be utilized for classification when object images are unavailable. To realize imaging through scattering media with our proposed SPC-SVR approach, first of all, a database containing adequate well-labeled known OS pairs should be established. The label of each speckle pattern in the known database is defined to be consistent with that of the corresponding object image (referenced as class 1, class 2 and etc.). Then, classification algorithms are used to learn classifiers with speckle patterns in the known OS Database. The considered classification algorithms are, two-dimensional principal component analysis and support vector machine (referenced as PCA+SVM), sparse representation based classification (SRC), K-singular value decomposition (K-SVD) and Kernel K-singular value decomposition (Kernel KSVD), while the considered regression model is support vector regression here [27–33]. Given an unknown speckle pattern, the learned classifiers are firstly adopted to classify the label of the speckle pattern, next, a certain number of OS pairs of the same label are randomly selected to learn an ISF, the learned ISF (of the same label) is then utilized to reconstruct the original object image. Our approach is attractive for several reasons: (1), we realize speckle pattern based classification without knowing the exact corresponding object images. (2), the classifiers, once learned, could be adopted to any unknown speckle patterns efficiently and there is no need to learn them each time. More examples per category would lead to higher classification accuracy. (3), the ISF, once learned through dataset with specific label, should reconstruct every possible object image with the same label efficiently. The more ISFs (of different labels) learned, the more objects (of different labels) would benefit from them and could be reconstructed effectively. Neither time-consuming calculations of scattering effect, nor separately iteration processes are needed for each new object under reconstruction. (4), our approach improves the aforementioned limited imaging capability, extending the practicability and feasibility of scattering imaging with machine-learning-based approach. The approach is expected to be applicable to image reconstruction in X-ray computed tomography, source reconstruction in acoustics, Earth density calculation, decryption in optics, and etc [34–37].

The remains of the paper are organized as follows : the methodology and the mathematical derivation of speckle pattern classification are given in Section 2. Then, we discuss the object image regression methodology based on support vector regression model in Section 3. Section 4 introduces the employed reconstruction fidelity evaluators. Section 5 demonstrates experimental results and quantitative analyses. Conclusions are drawn in Section 6.

2. Speckle pattern classification methodology

Our previous work showed that without classification first, the learned ISF with OS pairs of a certain class was only specified to reconstruct objects from speckles of that class [22]. To realize imaging through scattering media better, speckle pattern classification is proposed to conduct before object image reconstruction or regression. But in scattering imaging systems (see Fig. 1 for example), the object images are always scattered into unrecognizable speckle patterns. In this section, we present the deviation result that the autocorrelation matrix of the scattered speckle pattern has the same singular values or matrix as that of the object image, and thus could be utilized for classification. Since the key for classification is to learn more discriminative features of images so as to determine the optimal label for the given image data, the derivation is mainly based on singular value decomposition (SVD).

Fig. 1 Experimental setup of a scattering system. O, objective; P, pinhole; L, lens; SLM, spatial light modulator.

Download Full Size | PDF

The simplified experimental arrangement of a single-layer scattering system is demonstrated in Fig. 1. Light from a He-Ne laser source is collimated and expanded, then modulated by the amplitude type spatial light modulator (SLM, HES6001, Holoeye, used to modulate object images, and the two polarizers surrounding the SLM are omitted), then the modulated light traveled through a diffuser (DG10-220-MD, Thorlabs, served as a scattering media), and the scattered light was captured by a CMOS camera (C13440-20CU, Hamamatsu). The distance between modulated object image and the diffuser is set to be 25cm, and the image distance from the diffuser to camera is 15cm. The relationship between input object image and corresponding output speckle pattern captured by the image sensor in the scattering system (see Fig. 1) can be described as:

E^{o u t} \circ e^{j ϕ} = K \cdot E^{i n},

where

E^{o u t} \in R^{M_{o u t}}

and

ϕ \in C^{M_{o u t}}

are the vectorized amplitude field and phase field of output speckle pattern, respectively. ◦ denotes the Hadamard product operation. K represents the transmission matrix (TM) of scattering media (i.e., the diffuser here), with size M_out × M_in.

E^{i n} \in R^{M_{i n}}

represents the vectorized input object image. M_in and M_out are the pixel numbers in each input object image and output speckle pattern, respectively. Note that, due to the limitation of image sensor, only the amplitude field of output speckle field can be accessed in experiment.

Multiply both sides of Eq. (1) by its conjugate transpose, then we obtain:

(E^{o u t} \circ e^{j ϕ}) \cdot {(E^{o u t} \circ e^{j ϕ})}^{*} = (K \cdot E^{i n}) \cdot {(K \cdot E^{i n})}^{*} = K \cdot E^{i n} \cdot {(E^{i n})}^{*} \cdot K^{*},

where (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^* is a real symmetric matrix of size M_out × M_out. Similarly, Eⁱⁿ · (Eⁱⁿ)^∗ is a real symmetric matrix of size M_in × M_in. (·)^∗ means the conjugate transpose operation.

According to the transmission matrix theory, the TM K is a unitary matrix [3]. Then through performing SVD to Eⁱⁿ · (Eⁱⁿ)^*, (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^* and E^out · (E^out)* separately, we draw the conclusion that the three matrices share the same singular values or singular value matrix (see appendix A for detailed derivations). If we arrange the singular values in an order from large to small by default, then the singular matrix (owned by the three aforementioned matrices) is deduced to be the autocorrelation of the singular matrix of Eⁱⁿ.

The conclusion that, Eⁱⁿ ·(Eⁱⁿ)^* and E^out · (E^out)^* own the same singular values, is meaningful and can be utilized for speckle classification when object is unavailable. Besides, the conclusion also indicates that speckle pattern classification should perform the same as object classification theoretically.

3. Object image regression methodology

Given an unknown speckle pattern, speckle pattern classification is firstly performed to determine an optimum label. Then based on a dataset of OS pairs with the same label, support vector regression (SVR) is utilized to learn a corresponding ISF. The methodology of the utilized SVR model is introduced as follows.

The relationship between output speckle pattern E^out and input object image Eⁱⁿ can be rewritten as:

\begin{array}{l} E^{o u t} = f (E^{i n}), \\ E^{i n} = f^{- 1} (E^{o u t}), \end{array}

where f (·) and f⁻¹(·) denotes the forward scattering function (SF) and ISF, respectively.

Inverse scattering is to retrieve the object image from its speckle pattern. The flow chart of our approach is showed in Fig. 2. Given an unknown speckle pattern, the first thing to do is speckle pattern classification based on pre-established database (of relationships between known speckle patterns and their labels, SL in short). The flow chart of speckle pattern classification is showed in Fig. 3. Assume that the unknown speckle pattern is classified and labeled with P, then the corresponding ISF would be learned with that class of OS pairs, to conduct regression to reconstruct the object image in a further step.

Fig. 2 Flow chart of the inverse scattering method.

Download Full Size | PDF

Fig. 3 Flow chart of speckle pattern classification.

Download Full Size | PDF

The ISF learning with certain known OS pairs of a certain class is modelled by solving the following support vector regression problem:

\min_{w, b} \frac{1}{2} w^{T} w + C \sum_{n = 1}^{N} \max (0, | w^{T} E_{n}^{o u t} + b - E_{n}^{i n} | - ε),

where w is the inverse sensing matrix and follows f⁻¹(E^out) = w^TE^out + b (b is the intercept vector of ISF), (·)^T means the transpose operation, C means a constant parameter trading between regularization and violation,

E_{n}^{o u t}

means the nth output speckle,

E_{n}^{i n}

represents the nth input object, ε represents a parameter indicating the acceptable error, N is the number of training pairs. For each OS pair, training is conducted pixel by pixel.

As for any L2-regularized linear model, their optimal solution can be represented as the linear combination of independent variables, that is, $w = \sum_{n = 1}^{N} β_{n} E_{n}^{o u t}$ , where β_n is the coefficient corresponding to nth speckle pattern. Substituting it to Eq. (4) and applying the kernel trick (where the radial basis function is served as kernel), we obtain:

\begin{matrix} \min_{β, b} \frac{1}{2} \sum_{n = 1}^{N} β_{n} β_{m} \exp (- γ | | E_{n}^{o u t} - E_{m}^{o u t} | |^{2}) \\ + C \sum_{n = 1}^{N} \max (0, | \sum_{m = 1}^{N} β_{m} \exp (- γ | | E_{n}^{o u t} - E_{m}^{o u t} | |^{2}) - E_{n}^{i n} | - ε), \end{matrix}

where

\exp (- γ | | E_{n}^{o u t} - E_{m}^{o u t} | |^{2})

denotes the applied radial basis kernel function (RBF) in regression process. Solving the problem described in Eq. (5), the ISF can be learned and then the object can be reconstructed finally.

With the methodologies described above, unknown speckle patterns can be classified and the object situated behind scattering media can be reconstructed effectively. The next section introduces the employed indicators for image reconstruction fidelity evaluation.

4. Image reconstruction fidelity evaluation

To evaluate the quality of the scattering image reconstruction results objectively, the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) are used in this paper [38, 39]. Both are commonly and widely used objective evaluation standard of image quality. The PSNR between image x and image y of same size p_x × p_y is defined as:

\begin{array}{l} P S N R (d B) = 10 \cdot \log_{10} (\frac{M A X_{I}^{2}}{M S E}), \\ M S E = \frac{1}{p_{x} p_{y}} \sum_{i = 0}^{p_{x} - 1} \sum_{j = 0}^{p_{y} - 1} {[x (i, j) - y (i, j)]}^{2}, \end{array}

where MAX_I denotes the maximum possible pixel value of images and MAX_I = 1 since a normalization process is conducted before classification and regression. p_x and p_y represent the number of pixels of images in x-axis and y-axis, respectively.

The SSIM between image x and image y is defined as

S S I M (x, y) = \frac{(2 μ_{x} μ_{y} + c_{1}) (2 σ_{x y} + c_{2})}{(μ_{x}^{2} + μ_{y}^{2} + c_{1}) (σ_{x}^{2} + σ_{y}^{2} + c_{2})},

where µ_x and µ_y are the mean value of image x and y respectively, σ_x and σ_y are the variance of image x and y respectively, σ_xy is the covariance of image x and y. c₁ and c₂ are small positive constants used to avoid a null denominator. SSIM is a decimal value between −1 and 1. When the reconstructed image is identical to our input object image (which means that perfect reconstruction is realized), the SSIM reaches value 1.

Substituting image x and image y with our input object image and the reconstructed one respectively, then the PSNR value, as well as the SSIM value between them, can be calculated.

Next, experimental results are demonstrated to verify the methodologies illustrated above.

5. Experimental results

5.1. Database establishment

Before inverse scattering, a database containing well-labeled known OS pairs should be established. To reflect the importance of classification before regression well, in the paper, the MNIST handwritten digit dataset (referenced as MNIST) and the Fashion MNIST dataset (a collection of Zalando’s article images, referenced as Fashion MNIST) are utilized to provide object images [38, 40]. The considered two datasets are all extensively used in image classification, optical character recognition and machine learning research, sharing the same image size of 28 28 and structure of training and testing splits. The MNIST provides 60, 000 training examples×and 10, 000 testing examples, and the distributions of digits over different classes are detailed in Table. 1. The Fashion MNIST is also composed of a training set of 60, 000 examples and a testing set of 10, 000 examples, associating with a label from 10 classes. For each class, there is 6, 000 training examples and 1, 000 testing examples in the Fashion MNIST. The MNIST consisting of a collection of digit images (from digit “0” to “9”, handwritten in different styles), while images in Fashion MNIST are ranging from “T-shirt” to “Ankle boot”. For the convenient of learning the speckle classifiers, as well as the ISFs for speckle of certain label, images are all enlarged to a same size to serve as the input objects of the scattering system (see Fig. 1) one by one, and the corresponding output speckles are also collected one by one. As stated before, each output speckle is labeled according to the label of its input object image. Besides, normalization preprocessing was considered for all object images. In that case, our MNIST OS Dataset and Fashion MNIST OS Dataset can be established, respectively. Figure. 4 and Fig. 5 gives the illustration of some examples (of OS pairs) from the MNIST OS Dataset and the Fashion MNIST OS Dataset.

Table 1. The example numbers in the MNIST Database.

View Table | View all tables in this article

Fig. 4 Illustrations of different OS pairs from the MNIST OS Dataset. From left to right, each column shows normalized OS pair of a certain class from “0” to “4”. (a) object image of “0”, (b) speckle pattern of (a); (c) object image of “1”, (d) speckle pattern of (c); (e) object image of “2”, (f) speckle pattern of (e); (g) object image of “3”, (h) speckle pattern of (g); (i) object image of “4”, (j) speckle pattern of (i).

Download Full Size | PDF

Fig. 5 Illustrations of different OS pairs from the Fashion MNIST OS Dataset. From left to right, each column shows normalized OS pair of a certain class from “T-shirt” to “Sneaker”. (a) object image of “T-shirt”, (b) speckle pattern of (a); (c) object image of “Trouser”, (d) speckle pattern of (c); (e) object image of “Coat”, (f) speckle pattern of (e); (g) object image of “Sandals”, (h) speckle pattern of (g); (i) object image of “Sneaker”, (j) speckle pattern of (i).

Download Full Size | PDF

5.2. limited imaging capability without classification first

Our previous work showed that, without classification before regression or reconstruction, the capability of imaging through scattering media of machine-learning-based method is limited [22]. The limited imaging capability has also been discussed before, where the ISF learned from human faces was only specified for reconstructing human face from their speckle patterns [21]. In a word, with the ISF trained from OS pairs of class 1, the machine-learning-based regression method could only be able to reconstruct objects (of the same class) from their unknown speckles. The limitation is also held up for the ISF trained from OS pairs of any other classes.

Experiments were conducted to demonstrate the limited imaging capability reflected on the MNIST OS Dataset and Fashion MNIST OS Dataset, respectively. Both object images and their speckle patterns are all sampled to 28 × 28, i.e., M_in = 784. To ensure adequately learning, the number of training OS pairs is set as 784 (i.e., M_out = 784). The OS pairs used for learning an ISF of a certain label, are randomly chosen from the exact subset of the OS Database without overlap. In Fig. 6 and Fig. 7, each row is an example, where the listed images in the first four columns are the normalized input object image, measured speckle pattern, reconstruction with ISF1, reconstruction with ISF2 successively from left to right. As for each reconstruction, the calculated fidelity evaluation indicators (i.e., the PSNR and SSIM values) were summarized and inserted at the bottom of the corresponding subfigure. Combining the reconstructions and the indicators, the SSIM indicator seems to be relatively more objective. In Fig. 6, the utilized ISF in the third column is learned from digit “0” OS pairs, while that in the forth column is learned from digit “4” OS pairs. In Fig. 7, the utilized ISF in the third column is learned from “Trouser” OS pairs, while that in forth column is learned from “Sneaker” OS pairs.

Fig. 6 Reconstructions without classification first before learning the ISF with the MNIST OS Dataset. From left to right, the listed images in each column are, (a) and (b) input object images, (c) and (d) speckle patterns, (e) and (f) reconstructions with ISF learned from digit “O” OS pairs, (g) and (h) reconstructions with ISF learned from digit “4” OS pairs, (i) and (j) reconstructions with ISF learned from OS pairs of all the 10 classes.

Download Full Size | PDF

Fig. 7 Reconstructions without classification first before learning the ISF with the Fashion MNIST OS Dataset. From left to right, the listed images in each column are, (a) and (b) input object images, (c) and (d) speckle patterns, (e) and (f) reconstructions with ISF learned from “Trouser” OS pairs, (g) and (h) reconstructions with ISF learned from “Sneaker” OS pairs, (i) and (j) reconstructions with ISF learned from OS pairs of all the 10 classes.

Download Full Size | PDF

Experiments conducted with the MNIST OS Dataset and the Fashion MNIST OS Dataset showed that, the ISF learned with OS pairs of a class can regress or reconstruct objects of the class well, but the imaging capability of the ISF was restricted to objects of the class, and the ISF failed to reconstruct objects of other classes. The restriction was much severer in Fig. 7. This is due to the bigger difference among classes in the Fashion MNIST, which can also be reflected from the relatively lower classification accuracy in next subsection [40]. The phenomenon of limited imaging capability is consistent with that in our previous work [22]. The ISF learned from OS pairs of a certain class always has its limitation to reconstruct images of the other classes from their speckle patterns.

Including more classes of OS pairs to learn an ISF may help. Experiments were conducted to verify. In the experiments, the training OS pairs were randomly but about averagely chosen from all the 10 classes. The other experimental conditions were set the same as former. For comparison, results are listed in the fifth column of Fig. 6 and Fig. 7, respectively. Results showed that, with training OS pairs of more classes, even the learned ISF could regress speckles of the other class to some extent, but the image reconstruction fidelities are relatively lower than those whose speckles were regressed with ISF of the same class. The more ambiguous background should be responsible for this.

To improve the situation, speckle pattern classification is proposed. Next, we’ll show that, with speckle pattern classification based support vector regression method, capability and fidelity of imaging through scattering media can be improved at high probability.

5.3. imaging with speckle pattern classification based support vector regression

Experiments were conducted to validate speckle pattern classification performance. For each experiment, a certain number of OS pairs were chosen from training set of a dataset (i.e., the MNIST OS Dataset or the Fashion MNIST OS Dataset here) for learning classifiers, while speckle patterns (from testing set of the Datasets) were served as test or unknown ones to examine the performances of the classifiers learned through some related classification methods, then recorded the corresponding accuracies. Speckle pattern classification accuracies with several algorithms, i.e., PCA+SVM, SRC, K-SVD and Kernel KSVD, were considered here. Parameters of these algorithms were all empirically determined. The used kernel for SVM was gaussian radial basis function whose parameter γ was set as 0.25, and the number of feature vectors were extracted to 20. In SRC, the sparsity level was set as 20, all training samples were used to generate an overcomplete dictionary. In K-SVD and Kernel KSVD, dictionaries were learned with 30 atoms, the sparsity level was set as 20, and the maximum number of training iterations was set as 60. For kernel KSVD, a polynomial kernel of degree 4 was used. Classification accuracies under different number of training samples were also considered. For comparison, object classification was also conducted. The training samples were objects in object classification and turned to speckle patterns in speckle pattern classification. Table 2 and Table 3 shows the classification accuracies on the MNIST OS Database and the Fashion MNIST OS Database, respectively. P_o is the probability of correct object classification and P_s is the probability of correct speckle classification.

Table 2. Speckle pattern classification accuracies of some related methods on the MNIST OS Database.

View Table | View all tables in this article

Table 3. Speckle pattern classification accuracies of some related methods on the Fashion MNIST OS Database.

View Table | View all tables in this article

From the speckle pattern classification results listed in Table 2 and Table 3, several points could be concluded. First, the seemly unrecognizable speckle patterns could actually be used for image classification indeed and the satisfying speckle pattern classification accuracies were closed to the object classification accuracies, which validated the methodology described before. Second, with more training samples included for classifier learning, both the object and speckle pattern classification accuracies got some improvements for all considered algorithms. Besides, the kernel KSVD always performs best among all the considered algorithms.

Considering our proposed SPC-SVR method, higher classification accuracy means better reconstruction fidelity at higher probability. Once we get the label of an unknown speckle pattern, the exact ISF (learned from OS pairs of the same label) can be applied for object reconstruction. In the experiment, all OS pairs in training set were used for learning classifiers, while 784 (= 28 × 28) OS pairs were randomly chosen from training set for learning ISF of a certain label. Figure. 8 and Fig. 9 demonstrated the object reconstruction results based on the MNIST OS Dataset and the Fashion MNIST OS Dataset, respectively. The corresponding reconstruction fidelity evaluators were summarized and inserted at the bottom of reconstruction results. As shown in the figures, the proposed approach realized high fidelity imaging through scattering media at high probability based on the MNIST dataset and Fashion MNIST dataset. The resulting averaged PSNR value was about 15.90dB and the averaged SSIM value was about 0.42 based on the MNIST dataset, while the averaged PSNR value was about 20.67dB and the averaged SSIM value was about 0.65 based on the Fashion MNIST dataset. The difference was mainly due to the different sparsity levels of samples in the two datasets. The results were consistent with those in Fig. 6 and Fig. 7. Besides, the reconstruction or regression for an object took about one minute where the time for ISF learning was not included.

Fig. 8 Reconstructions using speckle pattern classification based support vector regression with the MNIST OS Database. (a) ∼ (j) show object and reconstruction examples of all the 10 classes from digit “0” to “9”.

Download Full Size | PDF

Fig. 9 Reconstructions using speckle pattern classification based support vector regression with the Fashion MNIST OS Database. (a) ∼ (j) show object and reconstruction examples of all the 10 classes from “T-shirt” to “Ankle boots”.

Download Full Size | PDF

5.4. Comparisons

Comparisons were conducted to validate effectiveness of our proposed method. Reconstruction results of wavefront shaping technique, ghost imaging, and phase-retrieval based method are also demonstrated. For our method, after speckle pattern classification finished, 784 known OS pairs were utilized to learn an ISF for reconstruction. For an arbitrary given unknown speckle pattern, the averaged time for classification was about 5 seconds using Kernel KSVD and that for regression was about 144 seconds. For wavefront shaping technique, genetic algorithm was used to seek for a global optimized modulation on phase type SLM and the imaging time was about 1 hour for 1000 generations. For ghost imaging technique, 10,000 acquisitions of scattered light field and reference light field were used for reconstruction. For phase-retrieval based method, 1000 iterations are adequate for convergency. The reconstructions of compared methods tested on the MNIST Database and the Fashion MNIST Database were showed in Fig. 10 and Fig. 11, respectively.

Fig. 10 Reconstructions based on MNIST Database with different methods. (a) tested original object image; (b) reconstruction with the proposed method; (c) reconstruction with wavefront shaping technique; (d) reconstruction with ghost imaging technique; (e) reconstruction with phase-retrieval based method.

Download Full Size | PDF

Fig. 11 Reconstructions based on Fashion MNIST Database with different methods. (a) tested original object image; (b) reconstruction with the proposed method; (c) reconstruction with wavefront shaping technique; (d) reconstruction with ghost imaging technique; (e) reconstruction with phase-retrieval based method.

Download Full Size | PDF

The phase-retrieval based method seemed to be the most convenient one, since reconstruction can be realized with just a single-shot measurement. Our proposed method required a well-labeled dataset to establish a Database containing adequate OS pairs. The wavefront shaping technique realized reconstruction through time-consuming feedback modulations. And the ghost imaging technique recorded the corresponding scattered light field and reference light field when random phase pattern was modulated on SLM one by one, to calculate image of object. From reconstructions in Fig. 10 and Fig. 11, where ghost imaging technique suffered, one could conclude that reconstruction fidelity of the proposed method was the highest.

6. Conclusions

In this paper, we deduced that in a scattering system, the autocorrelation of an input object image and the autocorrelation of the output speckle pattern have the same singular values, and propose that the seemly unrecognizable speckle patterns can be utilized for image classification when objects are unavailable. Speckle pattern classification based support vector regression method is proposed and utilized for single-shot imaging through scattering media. Speckle pattern classification is used to obtain the label of a given unknown speckle pattern, while support vector regression is designed for learning inverse scattering function of the scattering system, as well as original object reconstruction or regression. Experiments show that classifier with higher speckle pattern classification accuracy means higher object reconstruction or regression fidelity. Comparisons with some existing methods also validate the effectiveness of the proposed method. The proposed speckle pattern classification based support vector regression method, not only makes up the limitation of our previous research, but also provides a generalized and convenient approach for imaging through scattering media. Besides, our method could actually be seen as a generalized and universal solution for inverse problems such as phase retrieval and etc.

Our approach performs well in our setup, but a sufficiently large database of known object images and the corresponding speckle patterns should be established before learning the classifier and ISF for each category. More examples of speckle patterns mean a more stable and knowledgeable classifier, as well as higher reconstruction fidelity or more capable imaging through scattering media. Future issues to be addressed include, developing a efficient and efficiency image classification method (such as the convolutional neural nets) to ensure high speckle pattern classification accuracy and satisfying image reconstruction fidelity, multidimensional object reconstruction with the speckle pattern classification based support vector regression.

Appendix A - detailed derivations about the principle of speckle classification

In this appendix, we give the detailed derivations of the conclusion that Eⁱⁿ · (Eⁱⁿ)^∗, (E ^out ◦ e^jϕ) (E ^out ◦ e^jϕ)^∗ and E^out · (E^out)^* own the same singular values.

According to the transmission matrix theory, the TM K is a unitary matrix [3]. That is, K · K^∗ = I (I is identity matrix). The singular value decomposition (SVD) of K can be defined as:

K = S_{1} V_{1} D_{1}^{*},

where S₁ and D₁ are unitary matrices, following

S_{1} \cdot S_{1}^{*} = S_{1}^{*} \cdot S_{1} = I

and

D_{1} \cdot D_{1}^{*} = D_{1}^{*} \cdot D_{1} = I

respectively, V₁ (whose diagonal elements are arranged in an order from large to small) is the singular value matrix of K. For convenience, the diagonal elements of singular value matrices mentioned in the paper are all arranged in an order from large to small.

Substituting Eq. (8) and K · K^∗ = I, and we obtain:

\begin{matrix} K \cdot K^{*} = (S_{1} V_{1} D_{1}^{*}) \cdot {(S_{1} V_{1} D_{1}^{*})}^{*} \\ = S_{1} V_{1} D_{1}^{*} D_{1} V_{1}^{*} S_{1}^{*} \\ = S_{1} V_{1} V_{1}^{*} S_{1}^{*} \\ = I . \end{matrix}

Meanwhile, define the SVD of Eⁱⁿ as:

E^{i n} = S_{2} V_{2} D_{2}^{*},

where S₂ and D₂ are unitary matrices, V₂ is the singular value matrix of Eⁱⁿ.

Substituting Eq. (8) to Eq. (10) into (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^* = K · Eⁱⁿ · (Eⁱⁿ)^* · K^* (i.e., the Eq. (2) in the main body), we get:

\begin{matrix} (E^{o u t} \circ e^{j ϕ}) \cdot {(E^{o u t} \circ e^{j ϕ})}^{*} = K \cdot E^{i n} \cdot {(E^{i n})}^{*} \cdot K^{*} \\ = (S_{1} V_{1} D_{1}^{*}) \cdot (S_{2} V_{2} D_{2}^{*}) \cdot {(S_{2} V_{2} D_{2}^{*})}^{*} \cdot {(S_{1} V_{1} D_{1}^{*})}^{*} \\ = S_{1} V_{1} D_{1}^{*} S_{2} V_{2} D_{2}^{*} D_{2} V_{2}^{*} S_{2}^{*} D_{1} V_{1}^{*} S_{1}^{*} \\ = S_{1} V_{1} D_{1}^{*} S_{2} V_{2} V_{2}^{*} S_{2}^{*} D_{1} V_{1}^{*} S_{1}^{*} \\ = (S_{1} V_{1} D_{1}^{*} S_{2}) \cdot (V_{2} V_{2}^{*}) \cdot {(S_{1} V_{1} D_{1}^{*} S_{2})}^{*} . \end{matrix}

Obviously, $V_{2} V_{2}^{*}$ is still a diagonal matrix where each element equals to the square that in V₂. The necessary The necessary and sufficient condition for that $V_{2} V_{2}^{*}$ is the singular value matrix of (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^* is that $S_{1} V_{1} D_{1}^{*} S_{2}$ is a unitary matrix.

Then the problem is converted to prove $S_{1} V_{1} D_{1}^{*} S_{2}$ (denoted as S₃ for short) to be a unitary matrix.

Multiply S₃ by its Hermitian transpose, and substitute Eq. (9) into the product, then we obtain:

\begin{matrix} S_{3} \cdot {(S_{3})}^{*} = (S_{1} V_{1} D_{1}^{*} S_{2}) \cdot {(S_{1} V_{1} D_{1}^{*} S_{2})}^{*} \\ = S_{1} V_{1} D_{1}^{*} S_{2} S_{2}^{*} D_{1} V_{1}^{*} S_{1}^{*} \\ = S_{1} V_{1} D_{1}^{*} D_{1} V_{1}^{*} S_{1}^{*} \\ = S_{1} V_{1} V_{1}^{*} S_{1}^{*} \\ = I . \end{matrix}

In Eq. (12), we have proven that S₃ is also a unitary matrix, and thus Eq. (11) can be rewritten as:

(E^{o u t} \circ e^{j ϕ}) \cdot {(E^{o u t} \circ e^{j ϕ})}^{*} = S_{3} V_{3} D_{3}^{*},

where

S_{3} = D_{3} = S_{1} V_{1} D_{1}^{*} S_{2}

,

V_{3} = V_{2} V_{2}^{*}

is the singular value matrix of (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^*.

Return back to Eq. (10), the SVD of Eⁱⁿ · (Eⁱⁿ)^* can be written as:

\begin{matrix} E^{i n} \cdot {(E^{i n})}^{*} = (S_{2} V_{2} D_{2}^{*}) \cdot {(S_{2} V_{2} D_{2}^{*})}^{*} \\ = S_{2} V_{2} D_{2}^{*} \cdot D_{2} V_{2}^{*} S_{2}^{*} \\ = S_{2} (V_{2} V_{2}^{*}) S_{2}^{*} . \end{matrix}

That is, $V_{2} V_{2}^{*}$ is also the singular value matrx of Eⁱⁿ · (Eⁱⁿ)^*.

At this point, we have proven that, (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^* and Eⁱⁿ · (Eⁱⁿ)^* shares the same values singular value matrix. However, only the amplitude field of the speckle pattern (i.e., E^out) could be accessible in experiment. In a word, it’s necessary to find some relationship between the singular value matrix of E^out · (E^out)^∗ and (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^*.

We’ll prove that E^out · (E^out)^* has exactly the same singular value matrix as (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^*, as well as Eⁱⁿ · (Eⁱⁿ)^*.

Reformat the vector $e^{j ϕ} = {[e^{j ϕ_{1}}, e^{j ϕ_{2}}, \dots, e^{j ϕ_{M_{o u t}}}]}^{T}$ (where the (·)^T denotes transport operation) as a diagonal matrix (marked with φ), where each element in the diagonal is a corresponding element in e^jϕ, i.e.,

e^{j ϕ} = {[e^{j ϕ_{1}}, e^{j ϕ_{2}}, \dots, e^{j ϕ_{M_{o u t}}}]}^{T} \Rightarrow [\begin{array}{l} e^{j ϕ_{1}} \\ e^{j ϕ_{2}} \\ ⋱ \\ e^{j ϕ_{M_{o u t}}} \end{array}] = φ .

Then we have:

\begin{matrix} (E^{o u t} \circ e^{j ϕ}) \cdot {(E^{o u t} \circ e^{j ϕ})}^{*} = (φ \cdot E^{o u t}) \cdot {(φ \cdot E^{o u t})}^{*} \\ = φ \cdot E^{o u t} \cdot {(E^{o u t})}^{*} \cdot φ^{*} . \end{matrix}

Define the SVD of E^out as:

E^{o u t} = S_{4} V_{4} D_{4}^{*},

where S₄ and D₄ are unitary matrices, V₄ is the singular value matrix of E^out.

Then we have:

\begin{matrix} E^{o u t} \cdot {(E^{o u t})}^{*} = (S_{4} V_{4} D_{4}^{*}) \cdot {(S_{4} V_{4} D_{4}^{*})}^{*} \\ = S_{4} V_{4} D_{4}^{*} \cdot D_{4} V_{4}^{*} S_{4}^{*} \\ = S_{4} \cdot (V_{4} V_{4}^{*}) S_{4}^{*} . \end{matrix}

Apparently, $V_{4} V_{4}^{*}$ is a diagonal matrix, and is the singular value matrix of E^out · (E^out)^∗. Substitute Eq. (18) to Eq. (16), then we obtain:

\begin{matrix} (E^{o u t} \circ e^{j ϕ}) \cdot {(E^{o u t} \circ e^{j ϕ})}^{*} = φ \cdot E^{o u t} \cdot {(E^{o u t})}^{*} \cdot φ^{*} \\ = φ \cdot S_{4} V_{4} D_{4}^{*} \cdot {(S_{4} V_{4} D_{4}^{*})}^{*} \cdot φ^{*} \\ = φ \cdot S_{4} V_{4} D_{4}^{*} \cdot D_{4} V_{4}^{*} S_{4}^{*} \cdot φ^{*} \\ = (φ S_{4}) \cdot (V_{4} V_{4}^{*}) \cdot {(φ S_{4})}^{*} . \end{matrix}

It’s obviously that φS₄ is a unitary matrix since $(φ S_{4}) \cdot {(φ S_{4})}^{*} = φ S_{4} \cdot S_{4}^{*} φ^{*} = φ φ^{*} = I$ . That is to say, $V_{4} V_{4}^{*}$ is the singular value matrix of (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^*. Besides, considering the uniqueness of the singular value matrices (given the defination that the singular values in the singular value matrix displayed here are all arranged in an order from large to small), we have $V_{4} V_{4}^{*} = V_{2} V_{2}^{*}$ At this point, we can draw a conclusion that Eⁱⁿ · (Eⁱⁿ)^∗, (E^out ◦ e^jϕ) · (E^out ◦ e^jϕ)^* and E^out · (E^out)^*, share the same singular values or singular value matrix.

The conclusion can then serve as the principle of speckle pattern classification, that is, speckle patterns can be utilized for image classification and should have almost the same performance as object classification.

Funding

National Natural Science Foundation of China (Number:61601285).

Acknowledgments

The authors want to express their gratitude to editors and anonymous reviewers who gave their valuable comments and suggestions to this article.

Disclosures

The authors declare that there are no conflicts of interest related to this article.

References

1. Y. Wang, N. Cao, Z. Liu, and Y. Zhang, “Real-time dynamic MRI using parallel dictionary learning and dynamic total variation,” Neurocomputing 238, 410–419 (2017). [CrossRef]

2. R. Horstmeyer, H. Ruan, and C. Yang, “Guidestar-assisted wavefront-shaping methods for focusing light into biological tissue,” Nat. Photonics 9, 563 (2015). [CrossRef] [PubMed]

3. M. Kim, W. Choi, Y. Choi, C. Yoon, and W. Choi, “Transmission matrix of a scattering medium and its applications in biophotonics,” Opt. Express 23, 12648–12668 (2015). [CrossRef] [PubMed]

4. J. Jang, J. Lim, H. Yu, H. Choi, J. Ha, J. H. Park, W. Y. Oh, W. Jang, S. Lee, and Y. Park, “Complex wavefront shaping for optimal depth-selective focusing in optical coherence tomography,” Opt. Express 21, 2890–2902 (2013). [CrossRef] [PubMed]

5. B. R. Anderson, P. Price, R. Gunawidjaja, and H. Eilers, “Microgenetic optimization algorithm for optimal wavefront shaping,” Appl. Opt. 54, 1485–1491 (2015). [CrossRef] [PubMed]

6. L. Fang, X. Zhang, H. Zuo, and L. Pang, “Focusing light through random scattering media by four-element division algorithm,” Opt. Commun. 407, 301–310 (2018). [CrossRef]

7. L. Zhou, B. Zhuang, H. Sun, Z. He, M. Hu, and X. Qiao, “Speckle phase retrieval and transmission matrix obtaining of turbid media,” Opt. Int. J. Light Electron. Opt. 127, 9911–9916 (2016). [CrossRef]

8. I. M. Vellekoop and A. Mosk, “Focusing coherent light through opaque strongly scattering media,” Opt. Lett. 32, 2309–2311 (2007). [CrossRef] [PubMed]

9. A. P. Mosk, A. Lagendijk, G. Lerosey, and M. Fink, “Controlling waves in space and time for imaging and focusing in complex media,” Nat. Photonics 6, 283 (2012). [CrossRef]

10. I. M. Vellekoop, A. Lagendijk, and A. P. Mosk, “Exploiting disorder for perfect focusing,” Nat. Photonics 4, 320 (2010). [CrossRef]

11. H. He, Y. Guan, and J. Zhou, “Image restoration through thin turbid layers by correlation with a known object,” Opt. Express 21, 12539–12545 (2013). [CrossRef] [PubMed]

12. O. Katz, P. Heidmann, M. Fink, and S. Gigan, “Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations,” Nat. Photonics 8, 784 (2014). [CrossRef]

13. E. Edrei and G. Scarcelli, “Optical imaging through dynamic turbid media using the Fourier-domain shower-curtain effect,” Optica 3, 71–74 (2016). [CrossRef] [PubMed]

14. P. Wu, Z. Liang, X. Zhao, L. Su, and L. Song, “Lensless wide-field single-shot imaging through turbid media based on object-modulated speckles,” Appl. Opt. 56, 3335–3341 (2017). [CrossRef] [PubMed]

15. Y. Shechtman, Y. C. Eldar, O. Cohen, H. N. Chapman, J. Miao, and M. Segev, “Phase retrieval with application to optical imaging: a contemporary overview,” IEEE Signal Process. Mag. 32, 87–109 (2015). [CrossRef]

16. X.-H. Chen, Q. Liu, K.-H. Luo, and L.-A. Wu, “Lensless ghost imaging with true thermal light,” Opt. Lett. 34, 695–697 (2009). [CrossRef] [PubMed]

17. R. E. Meyers, K. S. Deacon, and Y. Shih, “Turbulence-free ghost imaging,” Appl. Phys. Lett. 98, 041801 (2011). [CrossRef]

18. Y. Xu, W. Liu, E. Zhang, Q. Li, H. Dai, and P. Chen, “Is ghost imaging intrinsically more powerful against scattering?” Opt. Express 23, 32993–33000 (2015). [CrossRef]

19. A. Zhang, Y. He, L. Wu, L. Chen, and B. Wang, “Tabletop x-ray ghost imaging with ultra-low radiation,” Optica 5, 374–377 (2018). [CrossRef]

20. R. Horisaki, R. Takagi, and J. Tanida, “Learning-based focusing through scattering media,” Appl. Opt. 56, 4358–4362 (2017). [CrossRef] [PubMed]

21. R. Horisaki, R. Takagi, and J. Tanida, “Learning-based imaging through scattering media,” Opt. Express 24, 13738–13743 (2016). [CrossRef] [PubMed]

22. H. Chen, Y. Gao, and X. Liu, “Speckle reconstruction method based on machine learning,” Proc. SPIE 10711, 107111U (2018).

23. Q. Wang, L. Ma, C. Li, Y. Zhou, and L. Tang, “Fast object recognition method from random measurements of compressive sensing camera,” Proc. SPIE 10679, 1067919 (2018).

24. Y. Cheng, Y. Cao, Q. Zhang, Z. Chu, and R. K. Wang, “Deep network for retinal disease classification based on limited clinical OCT angiography datasets (Conference Presentation),” Proc. SPIE 10474, 1047407 (2018).

25. G. Ran, M. F. Stuhlmacher, B. Tellman, N. Clinton, G. Hanson, M. Georgescu, C. Wang, F. Serrano-Candela, A. K. Khandelwal, and W. H. Cheng, “Using landsat and nighttime lights for supervised pixel-based image classification of urban land cover,” Remote Sens. Environ. 205, 253–275 (2018). [CrossRef]

26. S. H. Wang, P. Phillips, Z. C. Dong, and Y. D. Zhang, “Intelligent Facial Emotion Recognition based on Stationary Wavelet Entropy and Jaya algorithm,” Neurocomputing 272, 668–676 (2018). [CrossRef]

27. M. O. Faruqe and M. A. M. Hasan, “Face recognition using PCA and SVM,” in International Conference on Anti-Counterfeiting, Security, and Identification in Communication (2009), pp. 97–101.

28. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust Face Recognition via Sparse Representation,” IEEE Trans. Pattern Anal. Mach. Intell. 31, 210–227 (2009). [CrossRef]

29. M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Process. 54, 4311–4322 (2006). [CrossRef]

30. H. V. Nguyen, V. M. Patel, N. M. Nasrabadi, and R. Chellappa, “Kernel dictionary learning,” in IEEE International Conference on Acoustics, Speech and Signal Processing, (2012), pp. 2021–2024.

31. H. Van Nguyen, V. M. Patel, N. M. Nasrabadi, and R. Chellappa, “Design of non-linear kernel dictionaries for object recognition,” IEEE Trans. Image Process. 22, 5123–5135 (2013). [CrossRef] [PubMed]

32. Z. Wang, Y. Wang, H. Liu, and H. Zhang, “Structured Kernel Dictionary Learning with Correlation Constraint for Object Recognition,” IEEE Trans. Image Process. 26, 4578–4590 (2017). [CrossRef]

33. Vapnik and N. Vladimir, “The nature of statistical learning theory,” IEEE Trans. Neural Netw. 8, 1564 (1997). [CrossRef]

34. Y. Cai, D. Liu, J. P. Mathews, Z. Pan, D. Elsworth, Y. Yao, J. Li, and X. Guo, “Permeability evolution in fractured coal−combining triaxial confinement with X-ray computed tomography, acoustic emission and ultrasonic techniques,” Int. J. Coal Geol. 122, 91–104 (2014). [CrossRef]

35. Y. Kim and P. Nelson, “Optimal regularisation for acoustic source reconstruction by inverse methods,” J. Sound Vib. 275, 463–487 (2004). [CrossRef]

36. J. Wahr, M. Molenaar, and F. Bryan, “Time variability of the Earth’s gravity field: Hydrological and oceanic effects and their possible detection using GRACE,” J. Geophys. Res. Solid Earth 103, 30205–30229 (1998). [CrossRef]

37. M. Joshi, Chandrashakher, and K. Singh, “Color image encryption and decryption for twin images in fractional Fourier domain,” Opt. Commun. 281, 5713–5720 (2008). [CrossRef]

38. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE 86, 2278–2324 (1998). [CrossRef]

39. Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multi-scale structural similarity for image quality assessment,” in Asilomar Conference on Signals Systems and Computers (2003), pp. 1398–1402.

40. H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms,” (2017).

	0	1	2	3	4	5	6	7	8	9
Training set	5923	6742	5958	6131	5842	5421	5918	6265	5851	5949
Testing set	980	1135	1032	1010	982	892	958	1028	974	1009

Method	Classification accuracy (%)
	10,000 training samples		50,000 training samples

	P_o	P_s	P_o	P_s
PCA+SVM [27]	93.56	91.24	97.11	95.28
SRC [28]	93.68	95.28	96.79	97.55
KSVD [29]	95.40	95.80	97.00	97.00
Kernel KSVD [32]	96.40	96.00	97.50	97.88

Method	Classification accuracy (%)
	10,000 training samples		60,000 training samples

	P_o	P_s	P_o	P_s
PCA+SVM [27]	82.96	81.50	86.10	84.78
SRC [28]	83.82	84.54	87.18	87.99
KSVD [29]	85.40	84.80	86.30	87.30
Kernel KSVD [32]	86.20	85.90	87.70	88.70

	0	1	2	3	4	5	6	7	8	9
Training set	5923	6742	5958	6131	5842	5421	5918	6265	5851	5949
Testing set	980	1135	1032	1010	982	892	958	1028	974	1009

Method	Classification accuracy (%)
	10,000 training samples		50,000 training samples

	P_o	P_s	P_o	P_s
PCA+SVM [27]	93.56	91.24	97.11	95.28
SRC [28]	93.68	95.28	96.79	97.55
KSVD [29]	95.40	95.80	97.00	97.00
Kernel KSVD [32]	96.40	96.00	97.50	97.88

Imaging through scattering media using speckle pattern classification based support vector regression

Abstract

1. Introduction

2. Speckle pattern classification methodology

3. Object image regression methodology

4. Image reconstruction fidelity evaluation

5. Experimental results

5.1. Database establishment

5.2. limited imaging capability without classification first

5.3. imaging with speckle pattern classification based support vector regression

5.4. Comparisons

6. Conclusions

Appendix A - detailed derivations about the principle of speckle classification

Funding

Acknowledgments

Disclosures

References

Cited By

Figures (11)

Tables (3)

Equations (19)

Optics Express