1. Introduction

Advanced scientific imaging applications always drive us to pursue ever higher resolution, not only in spatial resolution, but also in temporal resolution for fast dynamic events on very short time scales. Generally, the temporal resolution is restricted by two respects of the imaging system. One is from the technical parameters of image sensor [1], such as photo sensitivity, pixel array, pixel digitalization, pixel readout rate, transfer mode, interface and so on. The other is from the bandwidth (i.e., response time) of optical modulation devices in the system, such as the shutter, optical switching, mechanical flipping, spatial light modulator (SLM) [2], digital micro-mirror device (DMD) [3], electrically tunable lens (ETL) [4], multi-axis motion and positioning, and etc. Once the shortage of frame rate occurs, imaging will lose the temporal information.

Following the manufacturing trend of the image sensor to decrease the pixel size and to expand the full sensor area, the pixel readout rate becomes a bottleneck after the quadratic increasing of pixel number. Therefore, when we use a scientific grade camera (e.g., sCCD, sCMOS and EMCCD camera), as well as high speed devices, in low-light imaging systems, for high spatio-temporal resolution, we are accustomed to choose the region of interest (ROI) readout mode to enhance the frame rate. However, this ROI hyper-frame-rate method is limited by the readout mode of image sensor. Frame rate is only proportional to the number of rows, regardless of the number of column [5,6]. Because the pixel data in each row is transferred serially, a considerable part of the pixel readout bandwidth is wasted. Another reported hyper-frame-rate method proposes to temporally multiplex neighboring pixels of the image sensor based on the DMD at the expense of low spatial sampling [7,8]. For high spatial resolution microscopic imaging, this method will be inapplicable.

In this paper, we present a new temporal multiplexing hyper-frame-rate method that uses a truncated field of view (FOV) and optically shift it to different sub-region of the image sensor during a full frame exposure. Following this framework, the frame rate of scientific camera at width-limited ROI will increase by making full use of the pixel readout rate. In the system, a rectangular field stop and a dual-axis scanning galvanometer are localized at the relay image plane and the pupil plane, respectively, and a high quality scan lens is utilized to form the image-side telecentric path. By customizing a FPGA based real-time control hardware, multiple sub-frame regions under different exposure time can be high-speed recorded from a single full frame. For validation, we implement a fluorescence microscope and use it to observe the high dynamic process of blood cells during the regular heartbeat of zebrafish larvae. Comparing with the DMD based hyper-frame-rate method, our hyper-frame-rate method maintains the pixel-level sampling and has a much easier alignment without the intensity loss from DMD filling factor and diffraction effect. Therefore, it is very suitable for the high dynamic fluorescence microscopic imaging over width-limited FOV.

The paper is organized as follows. Our hyper-frame-rate method is presented in section 2. In section 3, we set up a fluorescence microscope system. Imaging calibrations and validations is detailed in section 4. Section 5 presents the high spatio-temporal resolution results for imaging the heartbeat of zebrafish larvae. Section 6 concludes the paper.

2. Method

Considering that the ROI readout mode wastes bandwidth of pixel readout rate of the image sensor, we insist on using full-row or full-frame readout mode for our hyper-frame-rate imaging.

As illustrated in Fig. 1(a), the coherent optical fields from the point object propagated to the front focal plane and back focal plane of scan lens, have the Fourier transform relationship [9]. Let the front focal plane be conjugated to the pupil plane, and mount a dual-axis scanning galvanometer at this plane. While the dual-axis galvanometer rotates in ${\theta }_x$ and ${\theta }_y$ angle, it will bring an additional tip and tilt wavefront to the pupil field $U_P(x,y)$,

 

Fig. 1 (a) Temporal multiplexing method by the dual-axis scanning galvanometer, scan lens and their image-side telecentric path. (b) By active shifting the image, different sub-frame can be exposured at different time points over a single full-frame exposure.

Download Full Size | PDF

After Fourier transform, the back focal plane field $U_I(x\text{'},y\text{'})$ becomes

If the scan lens is the F-Theta lens, ${U^{\prime }}_I(x\text{'},y\text{'})$ approximates to $U_I(x\text{'}-f\cdot 2{\theta }_x,y\text{'}-f\cdot 2{\theta }_y)$. Differences between the image of ${\left|U_I(x\text{'},y\text{'})\right|}^2$ and image of ${\left|{U^{\prime }}_I(x\text{'},y\text{'})\right|}^2$ are the two offsets only. Therefore, we can use the dual-axis scanning galvanometer and scan lens to shift the image on the sensor.

Moreover, using this image-side telecentric path can maintain the planar magnification well and avoid the image distortions [10]. After eight times shifting, the sub-FOV will move from region (1) to region (9). In order to save response time, only one axis is acted upon in each shift. Furthermore, by synchronizing the light source off on the scanner action, as shown in Fig. 1(b), the above temporal multiplexing method can use the full frame mode to exposure sub-frame 9 times at different time points, keeping the maximum pixel readout rate of image sensor without any waste. Besides, a rectangular field stop with determined size corresponding to the sub-frame region on the image sensor, is mounting at the former relay image plane to prevent all sub-frames overlapping during the full frame exposure.

3. System implementation

3.1 A proof-of-principle microscope system

As shown in Fig. 2, we implemented an epifluorescence microscope system for the principle validation. Two-channel laser (Coherent) excitations with 488nm and 561nm wavelength are both single mode incident and respectively collimated. After coupled by a dichroic mirror (DM, Semrock), the beams converge through a lens, reflect on the second dichroic mirror (DM, Semrock) and focus at the back-plane of objective (Zeiss Epiplan-Apo 20 × /0.7, Zeiss EC Epiplan 40 × /0.6, Olympus UMPlanFL W 20 × /0.5). Finally, the excitation beams become the collimated illumination at the object plane of the objective. The two-channel emission fluorescence is then collected by the same objective, transmitted through a dichroic mirror and a multi-narrow-band filter (Semrock), and arrived at the flipping in/out mirror (FM, Thorlabs). When the FM is out of the optical path, the fluorescence converges through the tubelens and directly images at the detector plane of scientific grade camera (SGC, Hamamatsu, 2048 × 2048). This path allows the ordinary wide field imaging over full FOV. When the FM is in the optical path, the fluorescence reflects and converges at the relay image plane, where a fixed size field stop is mounted. As described in section 2, only the central FOV beams can be propagated to the dual-axis scanning galvanometer (Cambridge Tech, PS1-7), and finally image at the detector plane of SGC (Andor Neo, 2560 × 2160) through a scan lens (Thorlabs, CLS-SL) with lateral displacements. This temporal multiplexing path allows the hyper-frame-rate imaging over width-limited FOV. The total magnification of our microscope system under the hyper-frame-rate imaging is 17.02, 34.04, and 15.56, corresponding to the 20 × Zeiss Objective, 40 × Zeiss Objective and 20 × Olympus Objective, respectively.

 

Fig. 2 Schematic of our validation microscope system.

Download Full Size | PDF

3.2 System real-time control

In the system, in order to strictly control the timing of two lasers, external trigger exposure of SGC and step action of the dual-axis scanner, as presented in Fig. 1(b), we developed a timing control system based on a FPGA board (Xilinx). This timing control system outputs multiple digital and analog signals to the high-speed camera, dual-axis scanner, and two lasers without involving a host computer. This real-time control system and the host computer communicate via the serial port, and a useful graphical user interface (GUI) is programmed via QT in C + + on the host computer to import the user-defined timing of all devices and to configure the FPGA at initialization.

3.3 Other practical issues

Since one axis of the dual-axis scanner is not orthogonal to the optical axis, the coordinates passing through the scanner will lead to a fixed angle rotation along the optical axis [11]. This may generate image rotation around the origin of optical axis at the detection plane. Therefore, we pre-rotate the field stop at the proper angle and let the sub-frame region on the image sensor be the regular rectangular area.

Moreover, it is practical that the integrated dual-axis galvanometer product can’t place two mirrors both at the conjugated pupil plane. Being different from the point scanning galvanometer applications, the dual-axis galvanometers here is working step-by-step and we make the wide field image during the static state of scanner. Therefore, the field curvature and other low order aberrations from scanning will not exist. Once the galvanometer axes are not across the optical axis, the image plane inclination at different rotations will exist. This inclination influence could be negligible sometimes relative to the depth of field (DOF) at the detection plane (e.g., ± 0.42mm at 525nm wavelength and 20 × /0.5 objective). Meanwhile, we should place two galvanometer mirrors symmetric with the relay pupil plane to minimize the inclination.

4. System calibration

4.1 Single-step response of galvanometer

Our hyper-frame-rate imaging method requires us to calibrate the actual response of the dual-axis scanner to determine the sub-frame exposure interval as short as possible and filling up more sub-frame images per second. In practice, by sending the single-step motion command to each galvanometer by a signal generator and recording the readout of the galvanometer position from its control board by an oscilloscope [12], we can obtain the actual response of two galvanometers at different single-steps.

As shown in Fig. 3, we sent such a step signal (in blue line) to the first galvanometer for rotating 2.12 degree, which is corresponding to 800 pixels shifting at the image plane. This action takes almost 190μs. Other shifts from 100 to 1000 pixel have been also calibrated on this galvanometer, before turning to another galvanometer. These response curves are quite similar to the indicated curve in Fig. 3. Therefore, as summarized in Table 1, this 200μs can be treated as the possible minimum interval of sub-frame exposure.

 

Fig. 3 Response (red line) of single axis galvanometer to a single step motion Input (blue line).

Download Full Size | PDF

Table 1. Response Time of Galvanometer Single Step Motion under Different Increments.

View Table

4.2 Sub-frame position

Based on the rectangular field stop, we can precisely put the central FOV falling on each designated sub-frame region of image sensor by adjusting dual-axis scanner positions. This calibration also requires us to evaluate the position repeatability of the sub-frame scanning. The specification letter of the dual-axis scanner claims the repeatability as 20μradian, which transfers 1.4μm in RMS (root of mean square) at the image plane. Later experimental results have demonstrated that the position error is less than one pixel (6.5μm).

4.3 Spatial resolution in multiplexing imaging

First, we use a high magnification dry objective lens (40 × /0.6 Zeiss EC Epiplan) and a resolution test target under diascopic illumination to evaluate the spatial resolution of our system, both from the ordinary imaging and the temporal multiplexing imaging path, as shown in Fig. 2. Bright field images by temporal multiplexing imaging were acquired and an example shown in Fig. 4(a); the center region is enlarged in Fig. 4(b). Figure 4(c) is from the ordinary imaging result (with the center region enlarged in Fig. 4(d)). Profiles in Figs. 4(b) and 4(d) are presented in Figs. 4(e) and 4(f), respectively. Here, the line pattern was designed in 1.1μm period with 50% duty cycle, and the diffraction limit is 559nm at 550nm wavelength. After calculation, the mean FWHM (full width at half maximum) of line profile in Fig. 4(e) is 470nm while the sampling is 190nm/pixel, and the mean FWHM of line profile in Fig. 4(f) is 403nm while the sampling is 178nm/pixel. Both methods have achieved the desired line resolution under the 550nm line-width, and the temporal multiplexing imaging is a little worse than the ordinary imaging. Meanwhile, we calculated the mean contrast of profiles at the same position in each sub-frame images, and the ratio of RMS (root of mean square) to mean of 9 mean contrasts is 8.3%. Except for the aberrations involved from the galvanometer mirror unflatness, the scan lens will bring more sensible aberrations to different sub-frames along the individual focusing path [13].

 

Fig. 4 Line resolution test target images under the ordinary imaging and hyper-frame-rate imaging. (a) Hyper-frame-rate image. (b) Enlargement of the region (in yellow block) in the 5th sub-frame of (a). (c) Ordinary wide field image. (d) Enlargement of the region (in green block) in (c). (e) Profile along the purple line in (b). (f) Profile along the blue line in (d).

Download Full Size | PDF

In order to evaluate the point spread function (PSF) at each sub-frame, we use fluorescence microspheres (20nm in diameter, Thermofisher F8787) immobilized in the agarose layer to generate idealized point sources under 488nm excitation and Olympus 20 × /0.5 water dipping objective, which is the imaging condition for our zebrafish heartbeat imaging. By moving the sample stage to search for the smallest point-like object, the full frame image by temporal multiplexing was been acquired (see Fig. 5(a)). Here, a 3 × 3 sub-frame structure is used and the PSF distribution will locate at the same position of each sub-frame, as shown in Fig. 5(b). The FWHM of PSFs are obviously larger than the diffraction limit (640nm) here, mainly due to the agar diffusion and long exposure time. Sub-pixel position errors (discussed in section 4.2) are sensed in Fig. 5(b).

 

Fig. 5 PSF images and extended scene images under the temporal multiplexing imaging. (a) Hyper-frame-rate fluorescence image of the microsphere. (b) Enlargement of PSFs at the same region (in red circle) in nine sub-frames of (a). (c) Hyper-frame-rate image of the resolution test target. (d) Enlargement of the sub-region (in yellow block) in the 5th sub-frame (in green block) of (c). (e) Cross-correlation matrix between nine sub-frames in (c).

Download Full Size | PDF

Replacing with the resolution test target and diascopic illumination, bright field image in temporal multiplexing under 20 × /0.7 Zeiss Objective was acquired (see Fig. 5(c)), and the central sub-region was enlarged in Fig. 5(d). The point array was designed in 2μm period with 50% duty cycle. The normalized cross-correlation of these nine sub-images is calculated in matrix (see Fig. 5(e)), and all coefficients are more than 0.93. The 1st, 3rd, and 7th region both from the edge of FOV contribute to the negligible inconsistency. Aberration differences between these sub-frames are mainly from the different propagating path on the aperture of scan lens.

5. Experiment

Our dynamic imaging experiment focuses on the heartbeat process of the zebrafish larvae. The transgenic fish Tg(gata1:GFP) was used after 60 hours post fertilization (hpf), and was embedded in 1% low-melting agarose for imaging. Our transgenic zebrafish expressed GFP in its erythroid cells.

In the system, 20 × Olympus objective lens was used, and the 488nm excitation was emitted in 120mW power. The central FOV at the image sensor is limited to 820 × 700 pixel array by the field stop, corresponding to 350μm × 300μm at the object plane, thus fully covering the zebrafish heart as well as the aortic region. According to the conventional ROI hyper-frame-rate method, the Neo sCMOS camera can provide a 148 Hz frame rate.

In the timing control, the exposure time of each sub-frame is set to 2ms, and interval between them for the scanner action is set to 334μs. The full frame rate of Neo sCMOS camera is set to 48fps (its maximum full frame rate), which could be filled with 9 sub-frame exposure. Depending on our method, the practical frame rate is up to 432 fps, which triples the ROI frame rate method, as shown in Fig. 6. This temporal resolution is enough for capturing the cellular movements during the in vivo hemodynamics process. Note that, in order to make the single-step as short as possible, each odd and even numbered frame are designed to have different trajectories, as shown in Fig. 6(b).

 

Fig. 6 Hyper-frame-rate imaging of dynamic zebrafish heartbeat (see Visualization 1). (a) Raw full frame image acquired by Andor Neo sCMOS camera. (b) Different multiplexing trajectory in odd and even frames for saving the response time of the galvanometer scanning. (c) Sub-frame video sequence is extracted from the full frame video sequence according to the calibration results.

Download Full Size | PDF

After cropping the sub-frames from the full frame video sequence based on the calibration results (see Fig. 6(c)), the hyper-frame-rate video sequence can be reconstructed, as shown in Visualization 1, under the 0.05 × display speed. This video shows the high spatio-temporal resolution achieved by our method. During a single heartbeat, the movement of blood cell labeled by green fluorescence can be recorded in more than 120 frames.

Notice that, due to the limited aperture at input and output face of dual-axis scanner, an unexpected vignetting exists at the upper left edge of 1st sub-frame. We can choose a larger aperture dual-axis scanner in the future experiments to ameliorate this effect.

6. Discussion

6.1 System applicability

Our hyper-frame-rate imaging method aims to enhance the effective performance of camera and make the temporal multiplexing before the detection plane. Therefore, it is applicable in other types of high spatio-temporal resolution imaging system, not only in fluorescence microscope systems. Because of the independence with the upstream optical path, we can integrate the optical elements into a hyper-frame-rate unit (HFRU). As shown in Fig. 2, based on the known back length of scientific camera, the extended HFRU can stand behind the last focal lens at fixed distances.

6.2 Limitation of hyper-frame-rate

Except for the maximum pixel readout rate of the scientific grade camera, the exposure time and the FOV are the other two limitations in applying our hyper-frame-rate method. At practice, it is impossible to shorten the exposure time in few hundreds of microsecond, regardless of the shortage of signal-to-noise ratio (SNR) and image contrast.

Taking the Neo sCMOS camera as an example, if the possible minimum exposure time in application is 800μs, by adding 200μs for the single-step response of scanner, each sub-frame will take 1ms and the hyper-frame-rate imaging can achieve 1000fps. Considering the full frame rate of Neo camera is 48fps, therefore the number of multiplexed sub-frame could be 20. Dividing 2560 × 2160 pixel array into 20 sub-frame regions, each sub-frame becomes more than 500 × 500 pixel, keeping in square area for the most applications. Figure 7(a) illustrates the possible frame rate at different sub-frame exposure time in our system, and Fig. 7(b) illustrates the possible frame rate at different sub-frame pixel array, comparing with the conventional ROI hyper-frame-rate method. While the frame rate increases to 2500Hz (see Fig. 7(b)), the galvanometer response time will be comparable to the exposure time. Under this circumstance, the galvanometer action time will also limit the performance of our hyper-frame-rate method.

 

Fig. 7 (a) Possible frame rate at different sub-frame exposure time under the temporal multiplexing imaging and Andor Neo sCMOS camera. (b) Possible frame rate at different sub-frame region under the temporal multiplexing method and Andor Neo sCMOS camera.

Download Full Size | PDF

6.3 Shutter modes of image sensor

Our temporal multiplexing hyper-frame-rate method is suitable for many different scientific cameras. In general, there are two shutter modes of image sensor, i.e., the global shutter and the rolling shutter mode. For example, the Andor Neo sCMOS chip uses global shutter mode, and Hamamatsu Flash4.0 sCMOS chip uses the rolling shutter mode.

In the rolling shutter mode, by controlling the sub-frame illumination timing to be silent at the beginning and ending section of full frame exposure, when the asynchronized exposure of each row starts, our hyper-frame-rate method can still work. Nevertheless, the exposure interval of each sub-frame would be unequal. That could be troublesome for some imaging applications. Therefore, we recommend the hyper-frame-rate imaging method to work on global shutter cameras.

6.4 Comparing with other high-speed cameras

Our hyper-frame-rate method is an equipment improvement technique based on the currently available scientific grade cameras. Other than the non-scientific grade high-speed cameras, it maintains all the high performances of scientific grade image sensor, such as quantum efficiency, dark current noise, readout noise, dynamic range and etc. We only sacrifice the FOV of image sensor to enhance the effective temporal resolution of image sensor.

7. Conclusion

Based on the temporal multiplexing hyper-frame-rate method, we successfully recorded the rapid dynamic process of blood cells during the regular heartbeat of zebrafish larvae at 432fps over 820 × 700 pixel array. Once the exposure time decreases to 800μs and the FOV shrinks to 500 × 500, the effective frame rate of our imaging can achieve 960fps, which would be the best square field imaging performance of the scientific grade camera in the world, especially for the global shutter cameras. Due to the high independency and general applicability, our hyper-frame-rate imaging method can benefit numerous imaging systems to maximize the spatio-temporal resolution of camera at width-limited ROI.