1. Introduction

Optical microscopes often suffer from limited spatial resolution for imaging objects with nanoscale dimensions due to light diffraction. Over the past decades, the exploration of new optical super-resolution microscopies has attracted persistent interest with a goal of visualizing subwavelength features, to break through the diffraction limit. The most popular super-resolution approaches are the far-field fluorescence-based microscopies. The diversified fluorescence super-resolution microscopies, such as PALM [1], STORM [2], STED [3], and SIM [4,5], have been the most widespread bioimaging tools that achieve the state-of-the-art spatial resolution of a few nanometers. These fluorescence microscopies collect the sparse photoactivatable fluorophores to resolve spatial details of tightly packed molecules, thus the samples must be dyed with special fluorescent labels. This imposes restrictions on imaging inorganic samples. This drawback of current fluorescence microscopies hinders their applications into the fast-developing fields of nano-fabrications and integrated devices, which normally deal with inorganic artificial and regular nanostructures. For these artificial samples, a simple super-resolution optical imaging method working in a non-intrusive, wide-field and far-field way is still under exploration.

In principle, the conventional far-field optical imaging systems are constrained inevitably by the Abbe’s diffraction limit, which arises essentially from the loss of image information carried by exponentially decaying evanescent waves at the sample surface. Some optical “hardware” innovation, such as the super-lens [6–9] that can collect full-wave information, are promising to break the resolution constraint. However, at present, these metamaterial optical elements are still far from practical applications with many technical challenges unsolved, e.g., the quasi near-field work distance. In addition to these instrumental attempts, computational methods have also played an important role in the super-resolution reconstruction of optical micrographs [10,11]. For example, the optical blurring of fluorescent images can be formulated by the convolution of specimens’ scattering fields with the point spread function of an incoherent imaging system. Applying appropriate deconvolution algorithms, the fine structures of objects can be recovered inversely from their blurry images. For coherent imaging systems, many deconvolution algorithms, such as bandwidth extrapolation [12] and compressed sensing [13,14], have been proposed for super-resolution cognition of subwavelength structures based on the Fourier optics and information theory. The physical limit on the spatial resolution that can be recovered from far-field imaging is not necessarily close to the Abbe’s criterion, as discussed in a recent work [15]. However, these semi-analytical algorithms show limited super-resolving capability and applicability when addressing the highly nonlinear and ill-posed inverse problems of deep-subwavelength imaging.

In recent years, deep learning has emerged as a versatile and potent tool for tackling complex and previously intractable computational inverse problems in image processing, such as super-resolution reconstruction [16,17], classification or segmentation [18], phase retrieval [19–21], tomography [22,23], and holography imaging [24–26]. Deep learning methods are also very powerful for enhancing the spatial resolution of conventional optical microscopies, such as wide-field microscopy [27] and scanning confocal microscopy [28]. Nevertheless, the networks for the inverse image recovery must be pre-trained with extensive prior-known data. Therefore, the data-driven model lacks generality. Moreover, the “black box” nature of deep-learning networks raises concerns on the completeness of underlying physics that the network can conclude. Consequently, more studies aim to incorporate physical constraints into the deep neural architectures, with the goal of reducing the reliance on extensive training datasets [29–33]. For some nanostructures with regular shapes (nanowires, nanodisks, nanospheres, etc.), their far-field scattering fields depicted by intensity, phase, polarization, and spectral distributions can be precisely reconstructed by some physical models. This provides an ideal platform for modelling deep-learning networks with informed physics.

Here we present a deep-learning convolutional neural network (CNN) architecture for the deconvolution of optical microscopy images of both regularly arranged and randomly clustered metal nanostructures, achieving the super-resolution recovery of their real morphologies. Firstly, randomly distributed gold nanoparticles (AuNPs) are taken as an example to demonstrate the proposed CNN algorithm. These noble metal nanoparticles have many important applications in biosensing and bioimaging [34]. The optical micrographs of such AuNPs recording the grayscale intensities of their scattering fields are used as the input images, while their scanning-electron microscope (SEM) images captured at 10,000-fold magnification are treated as ground truths. Leveraging an extensive training dataset, the constructed CNN is learned to transform accurately the blurry optical images of randomly clustered AuNPs into SEM-like high-resolution images, where closely spaced nanoparticles can be well distinguished. Then we extend the same network to recover the optical images of randomly cross-linked silver nanowires. Most sections of these intricate nanowire nets can be reconstructed with only a slight discrepancy near their intersections. The deep learning method demonstrated here allows direct deconvolution of wide-field optical micrographs into super-resolution images with deep-subwavelength resolution.

2. Architecture design of our deep convolutional neural network

We take gold nanoparticles (AuNPs) as an example to demonstrate the deep-learning deconvolution model for the super-resolution reconstruction of optical images. The optical micrographs of the AuNPs were collected by a scientific camera mounted on a dark-field optical microscope, as schematically shown in Fig. 1(a). From the diffraction patterns of AuNPs (Fig. 1(b)), we can see that the closely spaced AuNP dimers are unable to be distinguished if the inter-particle distances are below the Rayleigh criterion (Δ = 0.61λ/NA). However, these intensity profiles of diffraction patterns still present many differences, providing much information for the reverse recovery of the nanoparticle arrangement. We designed a CNN to establish the nonlinear projection between optical diffraction images and particle dimensions. The proposed network architecture incorporates an explicit non-local prior by a non-local denoising module and an implicit non-local prior by a U-net structure, as shown in Fig. 2(a) [35], which is built on the four-scale encode-decoder U-net with a skip connection that exists in each scale. In the third scale, we plug the non-local denoising module for the encoder and decoder. For the operations of down-sampling and up-sampling in each scale, we simply use stride convolution layers of kernel size 3 × 3 and ConvTranspose layers of kernel size 2 × 2, respectively. The kernel sizes of other convolution layers are set to 3 × 3, following common settings. The number of feature maps is set to 32, 64, 128, and 256 in each scale. All the convolutional layers (except the last reconstruction layers) are followed by rectified linear units (ReLU) activation function.

 

Fig. 1. (a) Schematic of a conventional dark-field microscope for optical image acquisition. (b) Blurry optical images of AuNP dimers with varied spacing distance, collected by a CMOS camera mounted in the microscope. The light source used in the optical microscope is a halogen lamp which emits broadband light from 300 nm to 2000nm. Here a 633 nm narrow-band filter is added during image acquisition. (c) The corresponding SEM images of dimers.

Download Full Size | PDF

 

Fig. 2. CNN framework for deep-learning deconvolution microscopy. (a) The architecture of the proposed convolution neural network for image transformation and registration. (b) The input optical images of AuNPs with a field of view of 10.8 × 10.8 µm², the inset is the high-resolution TEM images of the AuNPs. The scale bar represents 200 nm. (c) The output SEM-like super-resolution images of AuNPs reproduced by the CNN network.

Download Full Size | PDF

The proposed network is then trained by the input optical images of randomly distributed gold nanospheres collected at 100X magnification (Fig. 2(b)). The illumination light source is a halogen lamp which emits broadband light from 300 nm to 2000nm. Note that the CMOS camera collecting the micrographs is only sensitive to the light in the wavelength range of 400-900 nm, which is broadband sufficient to eliminate the diffraction rings around nanoparticle for the convenience of network training. To obtain ground-truths, we acquired the SEM images of these AuNPs measured at 10,000X magnification. At this SEM magnification level, individual or clustered nanospheres can be well resolved, and the field of view of the SEM image is comparable to that of optical images. Thus, we can collect an adequate dataset of wide-field optical micrographs of randomly distributed AuNP multimers as input and their SEM images as ground-truths. The transmission-electron microscopic (TEM) image of the AuNPs illustrates their dimensional uniformity (the inset of Fig. 2(b)). The pre-processing of the experimental micrographs is performed for the purpose of image alignment and registration. The optical and SEM images were tailored into 480 pairs of non-overlapping subpictures of 400 × 400 pixels, among which 360 pairs are used for training and the remaining for testing. More details on sample preparation, image acquisition and renormalization are described in appendix section.

With the designed network architecture and the prior known datasets, the model parameters can be updated through the optimization of a loss function on a training set. We initialize the convolutional filters using the method of He et al [36], and adopt the Adam algorithm [37] to optimize the trainable parameters by minimizing the L1 loss function. Due to the local connectivity and shared weights properties of the proposed CNN model, we can use small patches for training. The network is then trained for 120 epochs. In each epoch, we randomly crop 32 × 3,000 patch pairs of size 96 × 96 from the training dataset. The learning rate starts from 10⁻⁴, then decreases by half every 30 epochs. The mini-batch size is set to 32 and other hyper-parameters of the Adam algorithm are taken by their default settings.

3. Super-resolution of nanoparticle multimers and nanowires

The proposed CNN network is well trained to transform the input optical micrographs of randomly distributed AuNP clusters into de-blurred output images that have the SEM-level resolution (Fig. 2(c)). We first select some AuNP dimers from the testing dataset for analysis. The input optical images of the dimers with inter-particle distances varied from 208 nm to 928 nm are shown in Fig. 3(a1-d1). In the optical micrographs, the closest-spaced dimer only exhibits one elliptical bright spot (Fig. 3(a1)), while two spherical spots gradually emerge and become distinguishable with the increase of dimer distances (Fig. 3(b1-d1)). Surprisingly, the output images from the trained CNN network present explicit non-diffracted spots (Fig. 2(a2-d2)), which are comparable to the ground truth SEM images (Fig. 3(a3-d3)). The spot size of an isolated nanosphere is ∼700 nm in the wide-field optical images (Fig. 3(d1)), whereas the corresponding sizes in the network outputs and the ground-truth SEM images are both ∼200 nm. Consequently, the deep-learning network can reconstruct diffraction-limited optical images of the AuNP dimers into SEM-level super-resolution images. To visually illustrate the accuracy of the deconvolution algorithm, we extracted the line profiles of grayscale intensities across the midpoint of each dimer in the input, output, and ground truth images, as compared in Fig. 3(a4-d4), further corroborating the super-resolving capability of our CNN model.

 

Fig. 3. Super-resolution imaging of AuNP dimers reconstructed by deep-learning algorithms. (a1-d1) The input blurry optical images of AuNP dimers with varying inter-particle distances, acquired on a conventional dark-field microscope. (a2-d2) The corresponding deconvolution images reconstructed by the trained CNN network. (a3-d3) The SEM images of AuNP dimers used as ground truths. These images are renormalized or rescaled. (a4-d4) Comparison of light-intensity profiles along the dimer axis between the input (black), output (green) and ground truth images (red).

Download Full Size | PDF

The proposed CNN deconvolution method is also qualified for more challenging cases of super-resolution reconstruction. As an example, we randomly select three groups of the input, output, and ground truth images containing the aggregated dimers, trimers, tetramers, etc., as shown in Fig. 4(a1-c1, a2-c2, a3-c3). Some representative zoom-in regions of multimers are depicted in Fig. 4(d). For most dimers and trimers with random orientations, their profiles and dimensions can be reconstructed well after the network computation. Especially, the CNN framework can recognize the minor difference in the optical images between the dimer (region 1) and the closely packed trimer (region 2), their far-field intensity profiles look like each other. For the catenulate trimer (region 3) and the tetramer (region 4), the bright spots in the optical micrograph become elliptic, exhibiting more differences for a higher identification efficiency. However, for AuNP clusters with more constituent nanospheres, such as the hexamer marked by the red square in Fig. 4(a2, a3), the discrepancy between the output images and the ground truths becomes apparent, particularly in the misjudgement of particle numbers and arrangement. It is reasonable to expect a decrease in accuracy as the number of particles in clusters increases for the CNN algorithm. This is because the optical patterns of multimers become more complex and there are fewer such clusters available for network training.

 

Fig. 4. Super-resolution imaging of randomly distributed AuNP multimers (N ≤ 4). (a1, b1, c1) The input wide-field optical images. (a2, b2, c2) The corresponding output deconvolution images and (a3, b3, c3) the ground-truth SEM images. (d) Zoomed-in regions of selected multimers bounded by green squares.

Download Full Size | PDF

To quantitatively evaluate the recognition accuracy of the proposed CNN model, we randomly chose 100 sets of isolated AuNP clusters tailored from the testing dataset for statistics. The output and SEM images of these complicated multimers (N ≥ 2) are merged into some kaleidoscopic images, as shown in Fig. 5. After careful observation, we can conclude that the computational discrepancies of the CNN model can be categorized into two classes - the inconsistency of particle positions and the misjudgement of particle number. Here, to include both kinds of mistakes, the correlation coefficient R between the network outputs and ground-truth SEM images is defined as the criterion of the recognition accuracy, which is formulated by

 

Fig. 5. The kaleidoscopes of 100-pair randomly selected AuNP multimers. (a1-d1) The output images from the trained CNN network for dimers (a1), trimers (b1), tetramers (c1), and more complicated AuNP clusters (d1). (a2-d2) The corresponding ground-truth SEM images.

Download Full Size | PDF

 

Fig. 6. The accuracy rate of transforming the optical images of AuNP multimers by the trained CNN framework. (a) The counted distributions of the correlation coefficient for multimers with varied particle numbers. (b) The average correlation coefficient between the output images and ground-truth SEM images of AuNP multimers. (c) The proportion of AuNP multimers with different particle numbers.

Download Full Size | PDF

So far, our deep-learning model has been demonstrated its efficiency in addressing image-recovering problems of AuNP clusters. However, the versatility of the proposed CNN algorithms needs further verification for other-type nanostructures. For this purpose, we have also employed our deep-learning model to reconstruct the morphologies of randomly cross-linked silver nanowires from their blurry optical images (Fig. 7). The silver nanowires have a diameter of ∼100 nm and a length of ∼10 µm. These nanowires were spin-coated on a glass substrate following the same experimental procedures described in Appendix A. Their optical and SEM images were measured under the same conditions as that of the AuNP experiments.

 

Fig. 7. The CNN reconstruction of nanowire morphologies from their optical images. (a) The input optical image of silver nanowires for testing, collected under the same conditions as that of nanoparticle experiments. (b) The output testing micrograph of nanowires. (c) The SEM images of silver nanowires. The scale bar represents 10 µm. (d, e) The zoomed-in images outlined by blue and red dashed squares in (b). (f) The loss curve during the network training.

Download Full Size | PDF

The silver nanowires were randomly distributed across the substrate surface as seen from the optical image shown in Fig. 7(a). The nanowire net exhibits numerous interactions which have inevitable influence on the resulting optical diffraction pattern. Using the trained CNN algorithm, as shown in Code 1 (Ref. [38]), the output image of testing nanowires can be reconstructed, as shown in Fig. 7(b). Compared with the ground-truth SEM image in Fig. 7(c), it is evident that majority of sections of the intricate nanowire net have been accurately recovered. However, minor discrepancies are noticeable near the intersection areas of nanowire net. Besides, thinner nanowires within certain densely packed regions are prone to misjudgement, resulting in subtle ripples in output nanowires. To illustrate these challenges more distinctly, we provide zoom-in pictures of two specific local regions, as shown in Fig. 7(d) and 7(e). Moreover, we present the loss curve observed during the network training, showing the gradual decreases and convergence of the loss with increasing epochs (Fig. 7(f)). Note that the motivation to design this CNN algorithm in this work is to improve the resolution of optical microscopies for common artificial nanostructures with regular shapes (disks, spheres, and wires, etc.), because these geometries are frequently used to construct more complex nano-patterns in the fields of nanomaterials, nanophotonics, nanoelectronics and even nano-mechanics. Additional modifications and optimizations on the present network are expected to produce better applicability to other nanostructures with irregular shapes and sizes.

4. Discussion and conclusion

Besides the problems of data acquisition for network training, well-designed CNN also plays a vital role to de-convolve computational images into high-resolution ones. Directly applying the common full CNN architectures may suffer from the problems of generality that the trained CNN models are heavily data-dependent and lack model interpretability. A solution to tackle this issue is to consider expert domain knowledge during the CNN design [39]. Such knowledge-based CNN is expected to have higher generality while using sparse training data, with known physics priors as a means to constrain the training [32]. Among various kinds of domain knowledge, non-local self-similarity has been extensively studied as it is a generic prior for most image restoration tasks [35]. Recently, it has been pointed out that U-net architectures with skip connections naturally impose self-similarity across multiple scales [40,41]. On the other hand, non-local denoising modules, which can be fused into existing CNN for end-to-end training, have been proposed to exploit this intrinsic property of non-local self-similarity across spatial domains.

Based on the aforementioned analysis, we present a deep non-local U-net which incorporates the non-local denoising module into the well-known U-net architecture. This integration enables our network to implicitly incorporate self-similarity priors across multiple scales with the U-net structure, and explicitly reveal the non-local self-similarity across the single scale via the non-local denoising module. Our proposed network proves especially effective for the specific task of transforming diffraction-limited optical images to SEM-like super-resolution images. Leveraging the robust learning capabilities of our well-designed deep CNN structure and an extensive dataset, the advanced deep CNN excels at accurately estimating images with SEM-level super-resolution from their diffraction-limited optical counterparts. To validate the capabilities of our deep learning model, we've employed aggregated AuNPs clusters and silver nanowires as a prime example for super-resolution imaging. The resulting images, showcasing the desired super-resolution of these AuNPs clusters, facilitate the study of homogeneity in plasmonic sensors without the need for electron microscopy, thereby preserving the integrity of sensing surface molecules. In summary, our work introduces a deep-learning-enabled microscopic technique of surpassing the diffraction limit, thus opening up new possibilities for the applications of deep-learning augmented computational microscopies in the nanoscience and nanofabrication industries.