Open Access
Add to BibSonomyAbstract
Rotating mirror cameras represent a workhorse technology for high speed imaging in the MHz framing regime. The technique requires that the target image be swept across a series of juxtaposed CCD sensors, via reflection from a rapidly rotating mirror. Employing multiple sensors in this fashion can lead to spatial jitter in the resultant video file, due to component misalignments along the individual optical paths to each CCD. Here, we highlight that static and dynamic fiducials can be exploited as an effective software-borne countermeasure to jitter, suppressing the standard deviation of the corrected file relative to the raw data by up to 88.5% maximally, and 66.5% on average over the available range of framing rates. Direct comparison with industry-standard algorithms demonstrated that our fiducial-based strategy is as effective at jitter reduction, but typically also leads to an aesthetically superior final form in the post-processed video files.
© 2014 Optical Society of America
1. Introduction
Rotating-mirror framing cameras readily facilitate high-speed imaging in the MHz framing regime: a capability that has been exploited for defence, aerospace and medically-related studies. In this last category, the recent realisation that ultrasound-stimulated microbubble cavitation may be utilised for therapeutic purposes [1] has led to an intense research effort to ascertain the details of cavitation processes at [sub-]microsecond temporal resolutions [2–7].
The benefits of rotary mirror systems, relative to their non-optomechanical counterparts, include the superior image resolution, high intrinsic dynamic range, and the relatively large sequence record depth, all of which are preserved even at maximum framing rate [7]. Annoyingly, spatial 'jitter' is present in the recompiled video files, so that countermeasures are required to restore the inter-frame alignment to its optimal aesthetic presentation. To address this, industry standard 2D image stabilization algorithms can be implemented, including: optimized cross-correlation of discrete Fourier transforms [8]; and global motion estimation methods [9,10]. However, in circumstances where target scenes exhibit non-uniform exposures from frame to frame; or during periods of rapid in-scene change, these orthodox curative strategies can be found wanting. Moreover, other technical issues regarding the spatio-temporal accuracy of rotary cameras can also present problems [11–14]. The purpose for the present study was to address such issues by making innovative use of native fiducials.
We evaluated images captured on a commercial rotary mirror framing camera over its full range of framing rates. Camera operation served to expose each of the 62 CCD imaging sensors in a sequential manner, whereupon their data are compiled to form a video file. Inspection of resultant videos showed two extraneous characteristics: (i) relatively small, static features, superimposed on the image and present on all individual frames; (ii) dynamic background features, usually elongated in nature, which were seen to sweep in the y-direction due to the intrinsic movement of the rotating mirror. The static features are presumed to be dust particles that have settled on a conjugate optical component [5,15], whereas the dynamic features appeared to be microscopic imperfections on the facets of the rotating mirror. We chose to term these naturally occurring features 'fiducials', given that they can be exploited as true points of reference within the resultant image files, thus providing a convenient alternative to orthodox system calibration approaches [5,13–18].
2. Methodology and apparatus
The hardware comprises a rotating mirror high speed camera (Cordin 550-62) (Fig. 1) fitted with an objective lens (magnification: 63x, numerical aperture: 0.9) and Leica Monozoom 7. A pentagonal beryllium mirror driven by a gas turbine is designed to rotate at a maximum of 12,500 revolutions per second (enabling framing rates of up to 4 MHz) [12,19]. Internally, the camera is constructed around 62 CCD image sensors, each comprised of a 1000 x 1000 pixel array, and arranged in two radial and symmetric 72° arcs either side of the mirror. The CCDs are housed in consecutive image frames labeled F0 to F63. CCDs are not present at locations F15 and F47: rather, these regions serve as apertures to allow direct image beam entry to the rotary mirror chamber - the image from the target having been divided by an upstream beam-splitter so that light is directed into the housing from two separate locations (blue arrows in Fig. 1) and towards two different facets (horizontal red rays) on the rotating mirror (labelled M). At any instant, two images of the target are thus circulating within the mirror chamber, one of which will lie on an active CCD bank, the other being formed somewhere within the two 108° arcs of 'dead-space' between the arrays of CCD banks. This situation is illustrated in Fig. 1 where the left hand image is reflected onto the first CCD at frame F0 just at the instant when the right hand image has finished being exposed at frame F63. The geometry of the mirror and nature of the recording process is such that a continuous 64 frame movie can be taken without any temporal interruption other than the presence of 'black-out' frames at the entrance aperture locations (F15 & F47). Imaging was undertaken with frame rates, Ω, ranging from 0.1 to 2.0 million frames per second (Mfps) after initial spatial calibration (viz inset to Fig. 1).
Fig. 1 Schematic overview of the optical arrangement within the sensor array housing of the Cordin 550-62 camera. (inset) Representative image of two 10-micrometer diameter glass calibration beads, and indication of the coordinate system used in the methodology.
3. Results
Sequences thus recorded were found to exhibit 'jitter', i.e., the image presented to each CCD appears somewhat displaced from frame to frame, which is aesthetically non-optimal. We determined to ascertain the nature of this resultant jitter, as well as developing a countermeasure to its occurrence. Static fiducials, as described earlier, were exploited towards this end. The process involved capture of multiple sequences across a range of frame rates (125kfps, 400kfps, 700kfps, 1.5Mfps and 2Mfps). As sequence recording starts randomly at any one of the 8 available CCD banks (Fig. 1), we first reformatted all sequences in terms of the absolute CCD position used to acquire specific frames within each video: this in a bid to quantify whether the inter-frame misalignments arising were directly linked to misaligned optical components. Rather than show the results for all 62 channels, only a subset of individual images, from banks 2 through to 4 (i.e. 24 frames in all, from F16 through F40 (viz Fig. 1)), are used for illustrative purposes for the rest of this paper. In each sequence file, a random static fiducial was selected, and its pixel coordinates were tracked across the following 24 frames using commercial image processing software (ImageJ v.1.46r using the MTrackJ plug-in [20]). Subsequently, in any particular frame i (where i = 0-23), the horizontal and vertical pixel positions of that pre-selected fiducial were defined as xi and yi respectively (the pixel position of the reference frame F16 (or frame i = 0) is thus designated (x0, y0).
Similarly, the relative displacements for the static fiducial (in pixels) for successive frames with reference to frame i = 0 are:
andThe inter-frame displacements for static fiducials obtained at different frame rates are displayed in Fig. 2. Evidently, measurements on dxi and dyi usually exhibited small variances as a function of frame rate. This suggested that misalignment of optical components along the beam path to each CCD sensor was the dominant factor contributing to image jitter, rather than internal vibrations. Here, vibration related deviations (relative to an arbitrary origin: dxi = 0 & dyi = 0) appear minimal (i.e. displacements were <10 pixels, over all framing rates) compared to the few more severely misaligned frames (at i = 2 or 8 for example (Fig. 2)) This resulted in |dxi| and |dyi| having maxima of 57 and 16 pixels respectively (which corresponds to 5.7% and 1.6% of the associated CCD lateral dimension), again underscoring the need for an alignment compensation algorithm.
Fig. 2 Relative displacements of a static fiducial (in pixels) over frames i = 1-23, as referenced to frame 0. Experiments were conducted over a range of framing rates, as indicated in the inset legend. (a) (left) horizontal relative displacement dxi; together with (right) an expanded vertical scale plot over frames 12-23 of the same data. (b) (left) Plot of vertical relative displacements dyi; together with (right) expanded vertical scale equivalent data over frames 12-23. Media 1, shows an animated comparison with an industry-standard jitter reduction routine [8]: evidently, both methods can be as effective at pin-pointing misaligned frames.
Consistent dependence of these image displacements on the absolute frame number (F) became obvious, facilitating direct implementation of a correction algorithm. An arbitrary reference frame (i = 0, corresponding to F16) was selected, and all subsequent frame displacements were assigned relative to this. Specifically, we measured the x- & y-displacements for a static fiducial over all subsequent frames, then mathematically compensated for these in the recompiled sequence. This is equivalent to a standard tracking algorithm where the displacement of some specific feature between each image frame is monitored relative to the Cartesian position of a nominated static fiducial - after which we then calculate dxi and dyi over all frames using Eqs. (1) and (2). The x-y coordinates of the target are designated as, xi,o and yi,o for the ith frame, and similarly, the uncorrected relative displacements (dxi,o, dyi,o) for the target fiducial are:
andwhere i is restricted to specific frames of interest and x0,o and y0,o represents target coordinate position of the arbitrary reference frame. Jitter compensated relative displacements (dxi*, dyi*) are then represented thus:andThe presence of dynamic fiducials (Fig. 3) was also exploited by selecting both a static fiducial and a dynamic fiducial for each individual sequence set, and continually monitoring their respective pixel coordinates. Using the jitter correction algorithm (Eqs. (3)–(6)), we obtained the corrected coordinate displacements of the dynamic fiducial, and in turn, its Cartesian velocity components Vx and Vy respectively (note that a hardware-borne diagnostic read-out on frame rate is used to calculate this and the units are expressed as [m.s−1]). The dynamic fiducial shown in Fig. 3 is seen to displace along the y-axis (Figs. 3(a)–3(h)). Its relativedisplacement can be jitter-corrected by subtracting the relative displacement of a nominated static fiducial. Subsequent calculations return values of |Vx| and |Vy| at 0.01 m.s−1 & 1.00 m.s−1 respectively. As |Vx| values in all cases were less than 2% of |Vy|, they are presumed negligible. A graph of |Vy| against Ω for both raw uncorrected data, and its jitter corrected counterpart are shown in Fig. 4. Here, it is evident that uncorrected data can have a large deviation from the mean. The fiducial-based jitter-correction procedure reduced such deviations significantly. Moreover, Fig. 4 supports the view that the average dynamic fiducial velocity is indeed correlated with mirror rotational speed. The superior aesthetic enhancement arising via our approach, is underscored in Media 2, where sequence excerpts from: (a) a raw data file; (b) the fiducial-based corrected version of that same raw data file; and (c), an industry-standardalgorithm [8] corrected version of the raw data in (a), are displayed together for ease of comparison.
Fig. 3 (a-h): A sequence captured at 0.33 Mfps and highlighting the characteristic difference in native fiducials. A static fiducial is shown (red circle 1 in frame (a)) with a dynamic fiducial (marked with a yellow circle numbered 2 in frame (a), and subsequently tracked as a green circle to denote its position in previous frames). A 10 μm scale bar is shown at the bottom of frame (h). Media 2 illustrates a direct comparison of such data, (with static fiducial highlighted) for the cases of: (a) raw data; (b) fiducial corrected data; & (c) industry standard [8] corrected data.
Fig. 4 Plot of the absolute average dynamic fiducial velocity, |Vy| [m.s−1] against frame rate Ω [Mfps] for selected experimental case studies. |Vy| data in grey represent the uncorrected raw data; whereas |Vy| in black are the corresponding jitter-corrected data emerging from the exploitation of static fiducals and averaged across all frames. Standard deviation bars are included for each measurement.
4. Conclusions
Having observed image jitter within image sequences acquired on a rotating mirror camera (Media 2 inset (a)), we showed that this apparent frame-to-frame displacement arose mainly due to small geometric misalignments in the optical hardware. Judicious choice of static and dynamic fiducials allowed us to execute a correction algorithm that compensated for camera jitter (Media 2 inset (b)) which performed comparably with an industry standard processing routine [8] (Media 2, inset (c)) but also produced a superior aesthetic result. Moreover, closer inspection over many imaging sequences showed that the five mirror facets each exhibit characteristic features (scratches/minute imperfections): i.e., each facet has a signature fiducial pattern, a realisation that facilitates analytical discrimination of sequence records on a facet-by-facet basis: useful information for undertaking downstream data diagnostics, and especially with identifying sources of aberration. This effectively adds an additional level of inspection.
We also demonstrated that jitter-corrected velocities of dynamic fiducials are correlated to frame rates recorded by the camera (Fig. 4), and moreover, that implementation of a jitter correction algorithm suppressed the standard deviation, relative to uncorrected data, by 88.5% maximally, and 66.5% on average over all framing rates. Analysis suggests that native fiducials are exploitable as accurate embedded scales for micro-imaging scenarios and moreover, that users should be able to accurately (and independently from the hardware) predict instantaneous framing rates. Whilst we have used the serendipitous appearance of native fiducials for our present analyses, we of course recognise that deliberately introduced fiducials, such as optical graticules or embossed mirrors could perform the same duties.
Acknowledgments
Infrastructural support via EPSRC (EP/D048958 & EP/H045368/1), an MRC Milstein Award, a Royal Society Industry Research Fellowship (IF09010 (to PC)), and a Wellcome Trust Strategic Award (098439/Z/12/Z) are gratefully acknowledged. BHTG thanks the NUS Graduate School for scholarship support. Adrian Walker (STFC) supported us with camera loans via the EPSRC instrument pool. We appreciate the assistance of both Charlie Main and Michael Conneely in critiquing the manuscript.
References and links
1. P. A. Campbell and M. R. Prausnitz, “Future directions for therapeutic ultrasound,” Ultrasound Med. Biol. 33(4), 657 (2007). [CrossRef] [PubMed]
2. P. A. Prentice, A. Cuschieri, K. Dholakia, M. R. Prausnitz, and P. Campbell, “Membrane disruption by optically controlled microbubble cavitation,” Nat. Phys. 1(2), 107–110 (2005). [CrossRef]
3. S. T. Thoroddsen, T. G. Etoh, and K. Takehara, “High-speed imaging of drops and bubbles,” Annu. Rev. Fluid Mech. 40(1), 257–285 (2008). [CrossRef]
4. P. A. Prentice, “Membrane disruption by optically controlled cavitation,” Ph.D. Thesis (University of Dundee, 2006).
5. H. O. Rolfnes, “Sonoptics: Applications of light and sound in the context of biomedicine,” Ph.D. Thesis, (University of Dundee, 2012).
6. V. Garbin, B. Dollet, M. Overvelde, D. Cojoc, E. Di Fabrizio, L. van Wijngaarden, A. Prosperetti, N. de Jong, D. Lohse, and M. Versluis, “History force on coated microbubbles propelled by ultrasound,” Phys. Fluids 21(9), 092003 (2009). [CrossRef]
7. C. T. Chin, C. Lancée, J. Borsboom, F. Mastik, M. E. Frijlink, N. de Jong, M. Versluis, and D. Lohse, “Brandaris 128: A digital 25 million frames per second camera with 128 highly sensitive frames,” Rev. Sci. Instrum. 74(12), 5026 (2003). [CrossRef]
8. M. Guizar-Sicairos, S. T. Thurman, and J. R. Fienup, “Efficient subpixel image registration algorithms,” Opt. Lett. 33(2), 156–158 (2008). [CrossRef] [PubMed]
9. S. Erturk, “Digital image stabilization with sub-image phase correlation based global motion estimation,” IEEE Trans. Consum. Electron. 49(4), 1320–1325 (2003). [CrossRef]
10. A. Litvin, J. Konrad, and W. Karl, “Probabilistic video stabilization using kalman filtering and mosaicking,” Proc. Image Video Commun. IS&T/SPIE Symp. Electron. Imaging, Santa Clara, CA., 663–74 (2003). [CrossRef]
11. E. A. Igel and M. Kristiansen, Rotating Mirror Streak and Framing Cameras. (SPIE Press, 1997).
12. A. Dubovik, The Photographic Recording of High Speed Processes (Wiley Interscience, 1981).
13. M. Conneely, H. O. Rolfsnes, C. Main, D. McGloin, and P. A. Campbell, “On the accuracy of framing-rate measurements in ultra-high speed rotating mirror cameras,” Opt. Express 19(17), 16432–16437 (2011). [CrossRef] [PubMed]
14. M. Conneely, H. O. Rolfsnes, D. McGloin, C. Main, and P. A. Campbell, “Role of mirror dynamics in determining the accuracy of framing rate in an ultra high speed rotating mirror camera,” Proc. SPIE 8125, 812512 (2011). [CrossRef]
15. Cordin 550 User’s Manual (Cordin Company, Inc, 2004).
16. H. M. Graham and G. A. Leavitt, “Air spark fiducial for ultra-high speed photography,” Rev. Sci. Instrum. 44(11), 1630–1632 (1973). [CrossRef]
17. T. Huen, “Programmable 10 MHz optical fiducial system for hydrodiagnostic cameras,” Proc. SPIE 832, 63–71 (1988). [CrossRef]
18. L. L. Shaw, S. A. Muelder, and A. T. Rivera, “Slit-mounted LED fiducial system for rotating mirror streak cameras,” Proc. SPIE 1539, 230–236 (1992). [CrossRef]
19. S. Ray, High Speed Photography and Photonics (SPIE, 1997).
20. E. Meijering, O. Dzyubachyk, and I. Smal, “Methods for cell and particle tracking,” Methods Enzymol. 504, 183–200 (2012). [CrossRef] [PubMed]
