Open Access
Add to BibSonomy
Abstract
We present an image-based autofocusing system applied in nonlinear microscopy and spectroscopy with a wide range of excitation wavelengths. The core of the developed autofocusing system consists of an adapted two-step procedure maximizing an image score with six different image scorings algorithms implemented to cover different types of focusing scenarios in automated regime for broad wavelength region. The developed approach is combined with an automated multi-axis alignment procedure. We demonstrate the key abilities of the autofocusing procedure on different types of structures: single nanoparticles, nanowires and complex 3D nanostructures. Based on these experiments, we determine the optimal autofocusing algorithms for different types of structures and applications.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Modern optical microscopy techniques include big varieties of physical principles to characterize structures with resolutions ranging from hundreds of microns to tens of nanometers [1]. One of the biggest challenges in optical microscopy nowadays is the automation of the measurement and characterization process. Almost all components, such as light sources, shutters, lenses, mirrors, stages, detectors, etc., can be coupled with an electronic controller or driver to control them remotely. The automation improves the stability of the system, reproducibility of the measurement procedure, time efficiency and throughput since the quality of a measurement will not be impacted by the operator’s movement or state. Therefore, in the development of modern microscopy techniques, fully automated measurement systems will be one of the key ways to improve accuracy, reproducibility and statistics of the results [2].
Among different optical microscopy techniques, nonlinear microscopy has some unique features. When the excitation laser beam is focusing in a medium, high intensities induce a nonlinear polarization response with respect to the electric field. This leads to multiple photon processes including multiphoton absorption, sum frequency generation (SFG), second harmonic generation (SHG), and third harmonic generation (THG) etc [3–6]. The multiphoton process occurs only at a single, diffraction-limited spot where the photon flux is enough to cause the absorption (or scattering) of more than one photon. This enables to perform 3D imaging as the signal emanates from a single spot. For processes like SHG multiple pump photons are combined into a single shorter wavelength signal photon. Compared to optical linear microscopy, the spatial resolution in nonlinear microscopy is higher as the wavelength of the photons is spectrally shifted. Additionally, the penetration depth is increased for materials and structures that absorb in the infrared regime.
Nonlinear (multiphoton) optical microscopy characterizations are usually applied in life sciences to study structural [7,8] and dynamical properties of different cells and proteins [9–12], but they also show a high potential in characterization of inorganic elements, such as semiconductors, metals, metamaterials, nanoparticles or nonlinear photonic components [13–19].
Another research field that employs methods of nonlinear optical characterization is nonlinear nanophotonics which studies the mechanisms of manipulating light at the nanoscale. The characterization methods in nonlinear nanophotonics require different approaches, such as the characterization of wavelength dependent resonances, emission enhancement or directionality of the generated nonlinear signals [20–24]. These types of studies require the recording of the nonlinear optical response over a broad range of wavelengths and polarizations of the incoming excitation laser beam. Current commercial multiphoton microscopes and corresponding software are not adapted for these types of measurements as they consist of achromatic objectives, adapted to certain precise wavelength ranges and adequate filters. Autofocusing systems are commonly used for routine measurements in life sciences, usually with attenuation, fluorescence or multiphoton imaging contrast. However, these systems do not allow to autofocus in a broad range of the laser excitation and collection wavelengths. That limits their application for nonlinear optical microscopy, especially in the fields of material analysis and nonlinear photonics structures characterization. The development of new approaches of automated focusing procedures will allow to overcome these limitations.
For an automated focusing procedure, there are many possible systems with various strategies and algorithms. Automated focusing procedures can be divided into two main classes: active/reflection-based or passive/image-based techniques. In the first category, active autofocusing systems measure the distance between a reference point near the sample and the objective with light or ultrasonic sound waves. For passive autofocusing systems a camera captures the signal at different sample or objective positions’ and evaluates the focused position with an image analysis algorithm. On the one hand, active/reflection-based techniques are faster and more direct because no images need to be taken, on the other hand, they are less accurate because the distance is measured with respect to a reference point (or reference plane), not the sample itself. Passive image-based systems rely on algorithms to determine if a sample is focused and, on a strategy, to find the best position of the focus. The focusing strategy governs the choice of images that are captured and given an image, the scoring algorithm returns a figure of merit called the focus score. This numerical value indicates the focus quality.
In our work, we present an image-based autofocusing system developed for high-resolution nonlinear optical microscopes. This system allows in a fully automated regime to characterize the nonlinear optical responses while sweeping the excitation laser wavelength, polarization power and position of the sample. To perform the autofocusing for a broad region of changing excitation parameters, we developed an autofocusing procedure with an image-based technique. This procedure includes a modified two-step hill-climbing strategy (coarse to finer search) with six different robust focusing algorithms also called focus measure functions. In our work we combined one strategy to maximize the focus score with the methods of multiphoton imaging microscopy. This approach allows to perform in a fast and reliable manner the nonlinear optical characterizations in a fully automated regime without any limitations to a specific wavelength region. We developed a software that allows to control all essential components such as laser, camera, stages for excitation and collection objectives, and stages to control the sample’s position. The autofocusing system is incorporated in this software as a plug-in and allows to perform nonlinear optical characterization in a fully automated regime for different types of structures and in a fully automated regime for a broad range of wavelengths. Furthermore, we developed a procedure to control position of the sample by controlling the different stages (axes) to keep the sample focused and aligned with the laser beam during the entire measurement procedure. In the next sections, we describe the autofocusing procedure and demonstrate how we applied it to characterize different types of structures. We tested the developed autofocusing system for different excitation, specifically laser and white light excitation, and different types of signals, such as a transmitted laser beam, bright field image or a generated SHG signal. We demonstrate the abilities of the system by characterizing structures with different dimensions and geometry. The InGaN nanowires were used as an example of an elongated 1D structure, single barium titanate (BaTiO3 or BTO) nanoparticles as an example of small single spot structures, layers of an epitaxially grown GaAs film and a woodpile photonic crystal structure, as an example of a layered structure with complex SHG signal. Our approach of the autofocusing can be easily adapted to different types of optical systems without limitations of wavelength range or contrast mechanism. The developed method can significantly reduce the duration measurements’ time and increase the quality as well as the reproducibility of the nonlinear optical characterizations in material science and other nonlinear photonics research areas.
2. Method
To demonstrate the abilities of the developed autofocusing system, we used a home-built nonlinear optical microscope with transmission and reflection measurement configurations. This setup is equipped with a tunable Ti:sapphire laser and with an optical parametric oscillator (OPO) system (wavelengths 700 – 1080 nm for laser region and 1080 – 1600 nm for the OPO region). Figure 1 presents the typical schematic of the home-built microscope [25–27], which was used to test the autofocusing procedure. The nonlinear microscope allows to perform the characterization with wide-field excitation in transmission regime, using a low magnification objective or lens to excite the studied structure.
Fig. 1 Schematic of the nonlinear optical microscope a) in transmission and b) in reflection. To switch from transmission to reflection, a beam splitter and a mirror are added. The focusing and collection is then performed with the same objective.
The wide-field excitation principle has two advantages over the scanning system. It is significantly faster and allows to study wavelength interference effects inside the structure [28]. As shown in Fig. 1(a), the excitation laser beam passes firstly through the power and polarization control unit, then it is focused with the excitation objective on the sample. The nonlinear response generated by the sample is collected with an objective, filtered to remove the pump wavelength and finally recorded with a camera. The core part of the autofocusing system consists of controlling the position of the sample, the excitation and collection objectives. The excitation and collection objectives are placed on motorized translation stages that move in horizontal direction (along the optical axis) while the sample holder is mounted on translation stages moving in the plane perpendicular to the optical axis. Similarly, the nonlinear microscope can be built in a reflection configuration (Fig. 1(b)). The main difference here is that only one objective is required for excitation and collection. All controllers/drivers for the translation stages, camera and tunable sources are combined in one software environment. This software environment allows to tune the excitation wavelength, to reposition the objectives and the sample, and capture the corresponding nonlinear optical signal. All components for the autofocusing are combined in the software in one loop, that allows for all hardware devices to communicate with each other to perform the measurements with the procedure set by the operator.
2.1 Wavelength sweep
A key challenge during a wavelength sweep is to focus the excitation laser on the sample and maximize the collected SHG signal for each wavelength and, as the optics, especially the objectives, have a wavelength dependent focal distance. Our autofocusing system in the automated regime adjusts the positions of the objectives distance between the objectives and the sample for each excitation wavelength during the wavelength sweep. This adjustment is performed at each wavelength until the signal is maximized. Figures 2(a) and 2(b) show images of the laser spot and corresponding focus scores (corresponding to the maximum pixel intensity of the image) at three different wavelengths (700, 800 and 900 nm) for various distances between the objectives. Figure 2(b) demonstrates the simultaneous change in excitation and collection translation stages’ position during the wavelength sweep to get the corresponding normalized focus score curves (here by the maximum pixel intensity) for each wavelength. When the wavelength is swept from 700 to 900 nm, the focused position shifts as indicated by the maximum in the focus score curve. Figure 2(c) illustrates the steps performed during the automated focusing procedure. Different positions during the autofocusing procedure are shown with purple dots in Fig. 2(c). After each wavelength change, the beam spot is focused by moving the objectives to the correct position.
Fig. 2 Illustration of the autofocusing procedure a) Images of the laser beam at three different wavelengths (700, 800 and 900 nm) for various distances between the excitation and collection objectives. The corresponding normalized focus score was given by the maximum pixel intensity of the captured image and shown for 700 nm. b) The normalized focus score curves are shown for the wavelength range 700 to 900 nm with 10 nm wavelength sweep step. The maximum in each curve indicates the focused position and shifts during a wavelength sweep. c) For an automated measurement procedure, the wavelength sweep step is followed by an autofocusing step. The different positions being evaluated during the autofocusing procedure are shown with purple dots.
2.2 Autofocusing procedure
During the autofocusing procedure every image of the signal from the sample gets a focus score indicating the focus quality. There are many autofocusing algorithms, also called focus measure functions, to perform this task. In their work, Sun et al. group these algorithms in four categories: (1) derivative-based, (2) statistical, (3) histogram-based and (4) intuitive algorithms. Various authors compared and ranked focus measure functions of these algorithms [29–32]. However, there is not one optimal algorithm for all possible configurations and focusing scenarios, as their performance depends on the specific situation, although some algorithms perform very well in multiple situations.
In our nonlinear optical microscope, we implemented six focus measure functions to cover a wide variety of different scenarios, for autofocusing on the different types of structures. These six focus measure functions algorithms are listed in Table 1. For more tuning in the autofocusing, a threshold can be set for the Brenner Gradient and the Image Power so that only terms bigger than and respectively are taken in the summation, where is the intensity at a given pixel. The threshold allows these methods to be used in a wider variety of focusing scenarios. We also evaluated the performance of other algorithms, for example the neighbor pixel difference algorithm (Squared Gradient), normalized Variance and Autocorrelation [33] algorithms, but they demonstrated a lower performance for our autofocusing method, both in terms of quality of results and speed. Therefore, we excluded these algorithms from further implementations. We have selected six different algorithms that demonstrated the best efficiency for different studies. The choice of the focusing criteria, is based on the sharpness of the image that we can see by eye or the shape of the focus curve, given by the optimal algorithm and threshold. That will give a focus score curve, in which the focused and non-focused positions can be clearly distinguished.
Table 1. Description of the six implemented algorithms. H and W are the height and width of the image, is the intensity at a given pixel, is the mean intensity of the image and B is the measured background obtained by averaging the pixel intensity of the border pixels of the image. They The algorithms can be categorized as statistical, derivative-based or intuitive algorithms.
Regarding the convergence of the algorithms, the intensity of the SHG varies with the peak power and the pulse duration of the excitation, which depends on the alignment and the related material dispersion in the optical path. We implemented in our system different algorithms to be able to focus on different aspects of the signal in case of intensity fluctuations. For example, the Maximum Pixel Intensity algorithm will be affected the most by such fluctuation, while the Image Power of Brenner Gradient algorithms will allow to overcome this problem.
The autofocusing strategy, also called search algorithm, describes how the highest focus scores can be found. In their work Zhang et al. group these search algorithms in six different categories: (1) global search, (2) ruled-based search, (3) model-based search, (4) coarse to finer search, (5) curve-fitting-based methods and (6) machine-learning-based methods [37]. In our method, every time when the laser excitation wavelength is changed, the positions of the objectives and the sample need to be readjusted. Our autofocusing system in order to be precise and fast, uses a modified hill-climbing strategy, which belongs to the category of coarse to finer search. The autofocusing strategy steps are illustrated in Figs. 3-4. Figure 3 presents a schematic overview of the autofocusing strategy. During coarse focusing, the autofocusing system actively searches for the region where the image is focused by comparing three successive focus score values (three points comparison) as shown in Fig. 4(a). If the three measured focus scores indicate a maximum region, this region is evaluated more closely in the fine focusing step. In order not to incorrectly identify a local maximum as the region of interest, the autofocusing system performs the three points comparison with a running average of n points (n points average). As can be seen from Fig. 4(b), in order to autofocus at the correct position and to avoid falling in a local maximum, the user can select either big step sizes for coarse focusing with n = 0, or by averaging over many points with smaller steps which will correspond to big values of n. In our case, the value n = 3 was chosen as a tradeoff between going fast with big step sizes for coarse focusing and analyzing a bigger range during the fine focusing, or with smaller steps that will make the coarse focusing procedure much longer. This procedure of coarse focusing allows us to find a region where we have to perform the finer focusing with very small steps to distinguish the neighboring maxima that are very close to each other.
Fig. 3 Schematic representation of the autofocusing strategy. If the Brenner Gradient or the Image Power algorithms are selected, the maximum threshold θ is first calculated from several images around the starting point. The coarse focusing step is divided in several loops, focus score values are measured with defined coarse step intervals, n consecutive values are averaged to reduce fluctuations (n points average) and three of these averages are compared to find the region of interest (three points comparison). In the fine focusing step, this region of interest is evaluated with finer step intervals and in the end, the stage is moved to the position with the maximum focus score.
Fig. 4 Schematic representation of the autofocusing strategy. a) During the coarse focusing step (star shaped markers), the global maximum is searched using the hill-climbing strategy. A wrong search direction (red points) is identified by checking for three consecutively lower focus scores. During the fine focusing step (square points), the region of interest around the identified global maximum is scanned to find the focused position. b) Averaging over n points during the coarse focusing step to smooth out the focus score curve, which ensures the global maximum (labeled “Corrected”) is found instead of a local one (labeled “Early”).
The parameter is a value between 0% to 100% defined only for the Brenner Gradient and Image Power, and it is a relative parameter to the maximum possible threshold. To define the value of the user does not need to know the signal intensity, the background noise, average or maximum pixel intensity. At the beginning, the threshold parameter is set to a default value, in our case 10% of the maximum possible threshold. Lower signal to noise ratios typically requires higher values for the threshold. The value can be experimentally optimized by observing the performance of the autofocusing system for different values.
We offer the possibility to select two different focus measure functions for the coarse and fine focusing. The focus score measured with the coarse algorithm should continuously increase as the image gets more and more focused. Since the autofocusing algorithm actively searches for a region of interest, this guarantees the first step to succeed even if the starting point is far from the final one, by continuing to move until the system will reach the correct region of interests. Concerning the fine focusing algorithm, the measured focus score curve should possess a global maximum that corresponds indeed to the desired final position. The coarse and fine focusing step size is defined by the user and depends on the depth of focus of the objectives. As an example, for the nonlinear microscope presented in Fig. 1, we advise to use a ratio of 1 to 10 for the two step sizes. For the position of the collection objective and the sample position we limit the fine focusing step size to the translation stage incremental step sizes which is 0.1 µm, while for the excitation lens we usually use 10 µm and 1 µm for the coarse and fine focusing step size. For other type of the nonlinear optical systems the steps for coarse and fine focusing can be adapted depending on the required task.
2.3 Multi-axis autofocusing
During the autofocusing procedure, the sample should be in the focal position for each wavelength change. To autofocus with multiple translational stages (axes), the same procedure as described in the previous sections is applied in a loop, repeatedly optimizing the positions of one stage after the other. This algorithm of multi-axis autofocusing is illustrated in Fig. 5 and it shows, how each axis is cyclically optimized after the other using the autofocus routine. The process is completed once the positions of all stages change is smaller than a given tolerance.
Fig. 5 Schematic of the multi-axis focus procedure. In each step, the position of the indicated axis is optimized using the autofocusing routine. The position of each stage is cyclically optimized until the change of position is smaller than a given tolerance.
An example of recorded positions of four translation stages during a multi-axis focus is shown in Figs. 6(a)-(d). In this case, we analyzed the SHG signal generated from a single BTO nanoparticle for different wavelength excitations. The wavelength was swept from 780 to 1080 nm (forward, red) and from 1080 to 780 nm (backward, blue). The position of the stages was adjusted at each wavelength step to keep the signal focused. For the structures with sizes of hundreds nm (BTO nanoparticles with diameters of 200 nm) the shift in the objectives positions even for 100 nm will change the result signal significantly due to the drift of the structure from the focused laser spot. As we can see from the recorded positions for x and y axes, the multi-axis autofocusing method allows to reposition the structure inside the excitation beam at each step of the wavelength sweep to keep the signal maximized. Another example of the recorded positions during the autofocusing procedure is presented in the Figs. 6(e) and 6(f), in which the positions of the excitation and collection objectives are recorded during the autofocusing of a 3D woodpile photonic crystal.
Fig. 6 Wavelength sweep with multi-axis autofocusing performed every 10 nm a)-d) from 780 to 1080 nm (red) and from 1080 to 780 nm (blue) for a BTO nanoparticle and e)-f) from 1040 to 1590 nm (orange) and from 1590 to 1040 nm (green) for a woodpile structure. Position recorded for the translational stages for the BTO nanoparticle for the a) excitation objective, b) the collection objective, and c)-d) the sample in x- and y-direction. Position recorded for e) the excitation and f) the collection objective for the woodpile structure.
To demonstrate how the developed multi-axis autofocusing method can be used to characterize different samples, we applied it to different categories of the structures. Depending on the ratio between the sizes of the excitation laser spot and the dimension of the studied structure, we can split the structures in four main categories: 0D, 1D, 2D and 3D. In our work, we experimentally tested it on the following structures: a spherical nanoparticle with sizes much smaller than the beam diameter (3-5 µm) as an example of a 0D structure, a long and thin nanowire as an illustration of a 1D structure, a layer of GaAs material to show how the autofocusing system will work for a 2D structure, and a photonic crystal as an example of a complex 3D structure. The number of stages that should be involved in the autofocusing procedure is given by the shape and morphology of the investigated structure. In the transmission mode of the setup, objectives with different magnification can be used for the excitation and the collection. This allows to achieve both wide-field excitation and high resolution in collection simultaneously. In reflection mode, the excitation and collection are usually performed with the same objective, which decreases the number of axes involved in the multi-axis autofocusing procedure. For the excitation, we used a focusing lens, that allows to perform wide focusing. The changes in the focused position during the wavelength sweep can be calculated with the equation for the wavelength dependency of the focal length of a thick lens [38]. The analytical solution could be used as a first step in the autofocusing procedure if the wavelength was swept with big steps.
Table 2 presents an overview of the axes (translational stages) that should be involved in the multi-axis autofocusing procedure depending on the type of structure. For instance, measuring 0D particles in transmission mode requires autofocusing by moving the collection and excitation objective plus the x and y position of the sample. In reflection mode, the excitation and collection are performed by the same objective. Also, in nanowire-like materials (1D), the motion of the x and y translational stages of the sample can be coupled to optimize the position perpendicular to the direction of the wire.
Table 2. Axes that are involved in the focusing procedure depending on the dimensionality (class) of the sample. In reflection mode, the collection and excitation are performed by the same objective. The axes x and y represent the position of the sample, z1 and z2 represent positions of excitation and collection objectives correspondently.
3. Experimental demonstration
The autofocusing system was tested in real conditions on samples with different geometries and dimensionalities. We demonstrate autofocusing for the nanowire under bright light illumination, under 800 nm laser beam and similarly with 400 nm SHG recorded. After that, we concentrate on focusing SHG from a nanoparticle, a layer and a woodpile structure. This show the capability of the developed system to work first with very different types of signals, and second with very different geometries and dimensions. To evaluate the performance of autofocusing, the true focused position was first determined for each axis. The focused image in a first step, when the sample is placed in the system, can be determined manually based on the sharpness perceived by human eye and after that the position of the objectives and the sample can be set precisely from the final position returned by the algorithm. Images were recorded around this position for some axes (with 0.1 µm step size) and then the focus score was measured at each position with each algorithm. Since the focus score curves reflect the performance of the autofocusing system using the corresponding algorithm, we can discuss the quality for each algorithm independently from the parameters such as the start position or the coarse and fine step size. As mentioned in the previous section, on the one hand the focus score measured with the coarse algorithm should continuously increase as the image gets more and more focused, because this indicates the direction of the maximum region. On the other hand, the focus score curve measured with the fine focusing algorithm should reach a global maximum corresponding to the true focused position. Additionally, the sharpness of the maximum defines the uncertainty of the algorithm and the wings around the maximum the size of the search range.
3.1 Nanowire
We tested the autofocusing system on a 20 µm long InGaN nanowire [39,40] with a hexagonal cross section of 500 nm high between two sides. Images were taken in transmission for three different situations: under bright light, under 800 nm laser beam and similarly with 400 nm SHG being recorded. In the first two situations, the excitation and collection objectives were moved simultaneously in the same direction by the same amount (equivalent to moving the sample along the optical axis) and in the last one, only the collection objective was moved around the focused position. The focus score curves for the images of the nanowire around the focused position for different autofocusing algorithms are shown in Fig. 7. In the first situation (see Fig. 7(a)), the intuitive algorithms (Maximum Pixel Intensity and Image Power) could not be used to focus as the calculated focus score was almost constant over the whole range of images, as can be seen from the difference between the true focused position and the one returned by the algorithm is indicated by an error arrow and a shaded region. These two algorithms were not suitable because all the captured light signal was coming from the transparent substrate. Similarly, the Image Power algorithm with very high thresholds ( > 90%) returned a wrong focused position as the signal originated from a bright spot in the image. The Variance and Brenner Gradient (low threshold ) algorithms could not focus the image correctly. The focus score curve indicated indeed a broad and almost flat maximum. However, these algorithms could be used for coarse focusing (only) as the measured focus score curve possessed decreasing wings (offset > 2 µm) which indicated the focused region. The Variance & Edge and Brenner Gradient (high threshold ) algorithms returned a correct position. The measured focus score curve indicated a clear maximum that also corresponded to the true focused position. The calculated focus curve showed two other peaks that did not correspond to any sharp image. To avoid a fine focusing step around one local maximum only, the coarse focusing step was set to 10 µm.
Fig. 7 Images and focus score curves calculated with different focus measure functions for the nanowire images around the focused position under a) bright light, b) under laser spot and c) under laser with SHG signal being captured. In the first situation, only the Brenner Gradient and the Variance & Edge filter algorithms were well suited for the autofocusing system. In the second one, only the Brenner Gradient algorithm could return the two focused positions. In the last situation, all algorithms are suitable for autofocusing. The colored vertical lines and frames indicate the true focused positions and the corresponding images. The difference between the true focused position and the one returned by the algorithm is indicated by an error arrow and a shaded region.
Focus score curves and images of the nanowire under a laser spot are shown in Fig. 7(b). Two clear positions appeared focused on the camera, either a thin line or a thicker still sharp line. This is caused by the particular hexagonal cross section of the nanowire [39,40], because the laser light can be focused either on the top facet or on the edges of the nanowire with hexagonal cross section. Only the Brenner Gradient algorithm was able to correctly identify the two positions. The two different threshold parameter values (20% and 50%) enable to focus on one or the other focused position of the nanowire, that correspond to the top and side facets of the nanowire. The other methods found neither of the two positions. Indeed, the Image Power and Variance algorithms returned a position in-between the two focused positions, as can be seen from the difference between the focused position and the one returned by the algorithm. This is indicated by an error arrow and a shaded region. They could still be used in the coarse focusing step to find the region of interest. The Variance & Edge algorithm could return a position close to one of the true focused positions.
In the last situation, the SHG signal was detected but the focused position was difficult to evaluate by hand with the same precision as in the previous situations. All algorithms could maximize the SHG signal and return a focused position. In the focus score curve measured by the Brenner Gradient and Image Power algorithms (with high thresholds ), the maximum was sharper, indicating a position with higher precision that what was achieved manually. In summary, the Brenner gradient demonstrated high performances when autofocusing a nanowire in all three different situations.
3.2 Single nanoparticle and thin layer
The autofocusing system was tested on a BTO nanoparticle (radius around 100 nm) in transmission. Figure 8(a) demonstrates the performance of the autofocusing system. It shows SHG images captured with the excitation objective moved around the focused position and the focus score curves measured with the different focus measure functions. For this point-like structure, every algorithm could return the same correct focused position and the peak in the focus score curves were sharp. We note that, in case of drift, the nanoparticle can be repositioned in the laser beam center to maximize the SHG response for each wavelength, by using the multi-axis autofocusing on the x and y stages.
Fig. 8 The SHG images around the true focused position and the focus score curves calculated with different focus measure functions for a) a BTO nanoparticle in transmission. All the algorithms are suited to autofocus the signal. b) SHG images for an Al0.2Ga0.8As layer in reflection mode. The Variance and Image Power algorithms return a different position than the other methods, which appears to be incorrect upon visual inspection. The green vertical lines and frames indicate the focused positions and corresponding images. The difference between the true focused position and the one returned by the algorithm is indicated by an error arrow and a shaded region.
The system was also tested on an Al0.2Ga0.8As layer in reflection mode. Figure 8(b) shows SHG images captured with the objective moved around the true focused position. The Variance and Image Power algorithms returned a position with a high SHG signal, which was not the true focused position. This can be caused by the fact that the position of the objective cannot match the excitation at pump’s wavelength with the collection at SHG’s wavelength. The SHG signal was spread and collected for positions around the focused position. At the position with offset −2 µm (Fig. 8(b)), the total signal intensity was still high enough, that the Variance and Image Power algorithms returned a wrong focused position as indicated by an error arrow and a shaded region. Nevertheless, the other three algorithms returned the correct focused position. The Brenner Gradient (with medium to high threshold ) algorithm showed the best performance as the focus score curve was not affected by the SHG signal measured at an offset of −2 µm.
3.3 Woodpile structure
The autofocusing system was tested on a complex structure, a 3D nonlinear photonic crystal with woodpile assembly structure. The sample was built up of single spherical nanoparticles which made it difficult structure to focus precisely. The SHG signal was captured in transmission mode. Images of the SHG signal were taken around the true focused position by varying the position of the excitation and then the collection objective. All algorithms were well suited to focus the sample, yet it was visually more difficult to evaluate the best excitation position. This was also visible in the focus score curve calculated by the focus measure functions, the maximum was not sharp. This photonic crystal has no localized or specific structure that could be focused on. Figure 9 demonstrates the ability of the developed autofocusing system to focus complex 3D structured samples. It shows images of a woodpile structure (photonic crystal) around the focused position and the focus score curves calculated with different focus measure functions. Figure 9(a) shows the autofocusing, when the excitation objective was moved, Fig. 9(b) shows images when the collection objective was moved. All algorithms were able to return the focused position, with much better precision in the second case, as expected as it was also clear to distinguish the focused image. For the case shown in Fig. 9(a), the focused position was difficult to determine by eye with a precision better than 5 μm. For the Maximum Pixel intensity and Image Power (with high threshold ) algorithms, the focus score curves possess a smaller maximum plateau region indicating a slightly better precision by focusing automatically.
Fig. 9 The SHG images of a woodpile photonic crystal structure around the true focused position and the focus score curves calculated with different focus measure functions. a) Either the excitation objective or b) the collection objective was moved. All algorithms were suited in both cases, yet it was more difficult to evaluate the best focused position in a). The green vertical lines and frames indicate the focused positions and corresponding images.
4. Conclusion
In this paper we demonstrated a two-step image-based autofocusing system for nonlinear optical microscopy with a broadband range of excitation wavelengths. We showed that with the adequate algorithms, the system was able to focus correctly in different scenarios on different types of sample geometries like nanowires, nanoparticles, layers or photonic crystals.
The key feature of the presented approach is the ability to autofocus for a broad range of excitation wavelengths. This feature was demonstrated for the focusing on the BTO nanoparticle, excited in the range 780 to 1080 nm, but we should notice that the developed system allows to autofocus for any other excitation wavelength ranges. The focusing procedure evaluates in general around 60 images (for the coarse and fine step). The main limitation of the autofocusing routine is the speed of the translation stage and the exposure time of the camera, which directly depends on the intensity of the signal. For typical structures, the autofocusing time can be estimated as 5 to 10 seconds for each wavelength, when the signal is very low the autofocusing procedure takes maximally 30 seconds. The modified two-step hill-climbing strategy with six different robust algorithms (focus measure functions) allows to focus the excitation and collection objectives on structures with different geometries from single 0D nanoparticles to complex 3D photonic crystals. The experimental results presented in our paper allows to select the optimal algorithms for coarse and fine focusing depending on the geometry of the studying structure. On top of that, the multi-axis focusing procedure enables the automation of an entire measuring sequence so that it keeps several optical components aligned as well as the sample focused. The autofocusing approach developed in our work can be also implemented without significant changes to any other focusing device.
Funding
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (SNF) (150609, 163916, 179966); Horizon 2020 European Research Council (ERC) (714837, 639052); H2020 Marie Skłodowska-Curie Actions (722176); Government Council on Grants, Russian Federation (075-00780-19-00); Ministry of Education and Science of the Russian Federation (16.9789.2017/BC).
Acknowledgements
This work was supported by the Swiss National Science Foundation grant 150609, 163916 and Ambizione grant no 179966, by the European Union’s Horizon 2020 research and innovation program from the European Research Council under the Grant Agreement No 714837 (Chi2-nano-oxides). M. Tchernycheva acknowledges the financial support from EU ERC project “NanoHarvest” (grant no. 639052) and from H2020 ITN project “INDEED” (grant no. 722176). I. S. acknowledges the support by State Project Nº 075-00780-19-00 (Subject Nº 0074-2019-0007) and A. B. acknowledges grant of Minobrnauki RF no. 16.9789.2017/BC
References
1. P. Török and F.-J. Kao, Optical Imaging and Microscopy, 2nd Edit. (Springer, 2007).
2. M. B. Y. Zeiss, “Adding efficiency to your fluorescence imaging .,” Nat. [inside view] (26th October), (2017).
3. D. Fröhlich, “Aspects of Nonlinear Spectroscopy,” Festkörperprobleme 21, 363–381 (1981). [CrossRef]
4. R. Li, X. Wang, Y. Zhou, H. Zong, M. Chen, and M. Sun, “Advances in nonlinear optical microscopy for biophotonics,” J. Nanophotonics 12(03), 033007 (2018). [CrossRef]
5. F. Helmchen and W. Denk, “Deep tissue two-photon microscopy,” Nat. Methods 2(12), 932–940 (2005). [CrossRef] [PubMed]
6. J. Squier, M. Müller, G. Brakenhoff, and K. R. Wilson, “Third harmonic generation microscopy,” Opt. Express 3(9), 315–324 (1998). [CrossRef] [PubMed]
7. L. Abdeladim, K. S. Matho, S. Clavreul, P. Mahou, J. M. Sintes, X. Solinas, I. Arganda-Carreras, S. G. Turney, J. W. Lichtman, A. Chessel, A. P. Bemelmans, K. Loulier, W. Supatto, J. Livet, and E. Beaurepaire, “Multicolor multiscale brain imaging with chromatic multiphoton serial microscopy,” Nat. Commun. 10(1), 1–14 (2019). [PubMed]
8. K. R. Campbell and P. J. Campagnola, “Wavelength-Dependent Second Harmonic Generation Circular Dichroism for Differentiation of Col I and Col III Isoforms in Stromal Models of Ovarian Cancer Based on Intrinsic Chirality Differences,” J. Phys. Chem. B 121(8), 1749–1757 (2017). [CrossRef] [PubMed]
9. W. R. Zipfel, R. M. Williams, and W. W. Webb, “Nonlinear magic: Multiphoton microscopy in the biosciences,” Nat. Biotechnol. 21(11), 1369–1377 (2003). [CrossRef] [PubMed]
10. K. König, “Multiphoton microscopy in Life Sciences,” J. Microsc. 200(2), 83–104 (2000). [CrossRef] [PubMed]
11. S. Brasselet, “Polarization-resolved nonlinear microscopy: application to structural molecular and biological imaging,” Adv. Opt. Photonics 3(3), 205–271 (2011). [CrossRef]
12. P. J. Campagnola and L. M. Loew, “Second-harmonic imaging microscopy for visualizing biomolecular arrays in cells, tissues and organisms,” Nat. Biotechnol. 21(11), 1356–1360 (2003). [CrossRef] [PubMed]
13. M. Timofeeva, A. Bouravleuv, G. Cirlin, I. Shtrom, I. Soshnikov, M. Reig Escalé, A. Sergeyev, and R. Grange, “Polar Second-Harmonic Imaging to Resolve Pure and Mixed Crystal Phases along GaAs Nanowires,” Nano Lett. 16(10), 6290–6297 (2016). [CrossRef] [PubMed]
14. L. Huang and J.-X. Cheng, “Nonlinear Optical Microscopy of Single Nanostructures,” Annu. Rev. Mater. Res. 43(1), 213–236 (2013). [CrossRef]
15. A. Prylepa, C. Reitböck, M. Cobet, A. Jesacher, X. Jin, R. Adelung, M. Schatzl-Linder, G. Luckeneder, K. H. Stellnberger, T. Steck, J. Faderl, T. Stehrer, and D. Stifter, “Material characterisation with methods of nonlinear optics,” J. Phys. D Appl. Phys. 51(4), 043001 (2018). [CrossRef]
16. J. Sun, Y. J. Gu, D. Y. Lei, S. P. Lau, W. T. Wong, K. Y. Wong, and H. L. W. Chan, “Mechanistic Understanding of Excitation-Correlated Nonlinear Optical Properties in MoS2 Nanosheets and Nanodots: The Role of Exciton Resonance,” ACS Photonics 3(12), 2434–2444 (2016). [CrossRef]
17. K. Y. Yang, J. Butet, C. Yan, G. D. Bernasconi, and O. J. F. Martin, “Enhancement Mechanisms of the Second Harmonic Generation from Double Resonant Aluminum Nanostructures,” ACS Photonics 4(6), 1522–1530 (2017). [CrossRef]
18. R. A. Myers, N. Mukherjee, and S. R. J. Brueck, “Large second-order nonlinearity in poled fused silica,” Opt. Lett. 16(22), 1732–1734 (1991). [CrossRef] [PubMed]
19. D. Krause, C. W. Teplin, and C. T. Rogers, “Optical surface second harmonic measurements of Isotropic thin-film metals: Gold, silver, copper, aluminum, and tantalum,” J. Appl. Phys. 96(7), 3626–3634 (2004). [CrossRef]
20. D. Smirnova and Y. S. Kivshar, “Multipolar nonlinear nanophotonics,” 3(11), 1241–1255 (2016).
21. S. S. Kruk, R. Camacho-Morales, L. Xu, M. Rahmani, D. A. Smirnova, L. Wang, H. H. Tan, C. Jagadish, D. N. Neshev, and Y. S. Kivshar, “Nonlinear optical magnetism revealed by second-harmonic generation in nanoantennas,” Nano Lett. 17(6), 3914–3918 (2017). [CrossRef] [PubMed]
22. F. Timpu, J. Sendra, C. Renaut, L. Lang, M. Timofeeva, M. T. Buscaglia, V. Buscaglia, and R. Grange, “Lithium Niobate Nanocubes as Linear and Nonlinear Ultraviolet Mie Resonators,” ACS Photonics 6(2), 545–552 (2019). [CrossRef]
23. S. Kühn, U. Håkanson, L. Rogobete, and V. Sandoghdar, “Enhancement of single-molecule fluorescence using a gold nanoparticle as an optical nanoantenna,” Phys. Rev. Lett. 97(1), 017402 (2006). [CrossRef] [PubMed]
24. S. Palomba and L. Novotny, “Nonlinear excitation of surface plasmon polaritons by four-wave mixing,” Phys. Rev. Lett. 101(5), 056802 (2008). [CrossRef] [PubMed]
25. M. Timofeeva, A. Bouravleuv, G. Cirlin, I. Shtrom, I. Soshnikov, M. Reig Escalé, A. Sergeyev, and R. Grange, “Polar Second-Harmonic Imaging to Resolve Pure and Mixed Crystal Phases along GaAs Nanowires,” Nano Lett. 16(10), 6290–6297 (2016). [CrossRef] [PubMed]
26. C. Renaut, L. Lang, K. Frizyuk, M. Timofeeva, F. E. Komissarenko, I. S. Mukhin, D. Smirnova, F. Timpu, M. Petrov, Y. Kivshar, and R. Grange, “Reshaping the Second-Order Polar Response of Hybrid Metal-Dielectric Nanodimers,” Nano Lett. 19(2), 877–884 (2019). [CrossRef] [PubMed]
27. R. Grange, G. Brönstrup, A. Sergeyev, J. Richter, T. Pertsch, A. Tünnermann, S. Christiansen, C. Leiterer, W. Fritzsche, C. Gutsche, A. Lysov, W. Prost, and F.-J. Tegude, “Imaging of waveguiding and scattering interferences in individual GaAs nanowires via second-harmonic generation,” in Nanophotonics IV (2012), 8424, pp. 1–8.
28. R. Grange, G. Brönstrup, M. Kiometzis, A. Sergeyev, J. Richter, C. Leiterer, W. Fritzsche, C. Gutsche, A. Lysov, W. Prost, F. J. Tegude, T. Pertsch, A. Tünnermann, and S. Christiansen, “Far-field imaging for direct visualization of light interferences in GaAs nanowires,” Nano Lett. 12(10), 5412–5417 (2012). [CrossRef] [PubMed]
29. Y. Sun, S. Duthaler, and B. J. Nelson, “Autofocusing in computer microscopy: selecting the optimal focus algorithm,” Microsc. Res. Tech. 65(3), 139–149 (2004). [CrossRef] [PubMed]
30. L. Firestone, K. Cook, K. Culp, N. Talsania, and K. Preston Jr., “Comparison of autofocus methods for automated microscopy,” Cytometry 12(3), 195–206 (1991). [CrossRef] [PubMed]
31. F. C. A. Groen, I. T. Young, and G. Ligthart, “A comparison of different focus functions for use in autofocus algorithms,” Cytometry 6(2), 81–91 (1985). [CrossRef] [PubMed]
32. Y. Zhang, H. Wang, Y. Wu, M. Tamamitsu, and A. Ozcan, “Edge sparsity criterion for robust holographic autofocusing,” Opt. Lett. 42(19), 3824–3827 (2017). [CrossRef] [PubMed]
33. D. Vollath, “Automatic focusing by correlative methods,” J. Microsc. 147(3), 279–288 (1987). [CrossRef]
34. T. Yeo, S. Ong, Jayasooriah, and R. Sinniah, “Autofocusing for tissue microscopy,” Image Vis. Comput. 11(10), 629–639 (1993). [CrossRef]
35. J. F. Brenner, B. S. Dew, J. B. Horton, T. King, P. W. Neurath, and W. D. Selles, “An Automated Microscope for Cytologic Research,” J. Histochem. Cytochem. 24(1), 100–111 (1976). [CrossRef] [PubMed]
36. A. Santos, C. Ortiz De Solórzano, J. J. Vaquero, J. M. Peña, N. Malpica, and F. Pozo, “Evaluation of Autofocus Functions in Molecular Cytogenetic Analysis,” 188(3), p. 264−272.
37. Y. Zhang, L. Liu, W. Gong, H. Yu, W. Wang, C. Zhao, P. Wang, and T. Ueda, “Autofocus System and Evaluation Methodologies : A Literature Review,” 30(5), 1165–1174 (2018).
38. E. Hecht, Optics, 4th Edi. (Pearson Addison-Wesley, 2002).
39. L. Mancini, D. Hernández-Maldonado, W. Lefebvre, J. Houard, I. Blum, F. Vurpillot, J. Eymery, C. Durand, M. Tchernycheva, and L. Rigutti, “Multi-microscopy study of the influence of stacking faults and three-dimensional In distribution on the optical properties of m-plane InGaN quantum wells grown on microwire sidewalls,” Appl. Phys. Lett. 108(4), 042102 (2016). [CrossRef]
40. R. Koester, J. S. Hwang, D. Salomon, X. Chen, C. Bougerol, J. P. Barnes, D. S. Dang, L. Rigutti, A. de Luna Bugallo, G. Jacopin, M. Tchernycheva, C. Durand, and J. Eymery, “M-Plane Core-Shell InGaN/GaN Multiple-Quantum-Wells on GaN Wires for Electroluminescent Devices,” Nano Lett. 11(11), 4839–4845 (2011). [CrossRef] [PubMed]
