1. INTRODUCTION

Frame rate and resolution are two intertwined performance metrics of an imaging system. Frame rate describes the system’s temporal sampling capability using the frequency with which the field of view is imaged. Resolution, in a digital image processing context, describes the system’s spatial sampling capability using the number of samples within a single field of view. Depending on its needs, an imaging application may require a high frame rate (e.g., video) [1–3], high resolution (e.g., microscopy) [4,5], or both [6,7]. Not all combinations of resolution and frame rate are achievable, however, depending on the underlying imaging method. In particular, scanned imaging modalities, which use a predetermined pattern to encode the spatial position with time, exhibit a fundamental tradeoff between resolution and frame rate, governed by the imaging system’s hardware [8]. Optical coherence tomography [1,2,7,9,10] and ultrasound [3] are two common and relevant such modalities in biomedical imaging. Suboptimal performance, temporally or spatially, results when the interaction between frame rate and resolution is not understood and optimized.

Scanned imaging modalities use time-division multiplexing to assemble a larger image or frame by collecting samples throughout the field of view in a predetermined sequence (i.e., a scan pattern). A nearly universal goal of scanning is to accomplish this task in the minimum amount of time, either to maximize the frame rate or minimize the rolling shutter artifact. The frame time scales with the number and spatial extent of samples such that an increase in the resolution produces a compensatory decrease in the imaging frame rate and vice versa. A maximum achievable frame rate exists for each resolution, with the highest frame rate obtained at the lowest resolution. The central challenge in scanning is to distribute samples across the scene efficiently so as to maximize the frame rate. Although advances in acquisition and scanning technology have increased sampling rates [11] and scanner efficiency [12,13], none have fundamentally altered the trade-off between the frame rate and the resolution.

The desire to maximize the frame rate through efficient scanning has produced many techniques to allocate samples across an imaging scene, including resonant [14], Lissajous [15], and constant angular [16] and linear [17] spiral scan patterns. Each technique seeks to better match the scan pattern with the scanner technology’s capabilities than traditional raster and radial patterns [18]. On the hardware front, similar efforts have sought to develop faster imaging sources [11], detectors [19,20], and scanner technologies [21–23] that lift restrictions on the sampling rates and imaging scene geometry. Where the underlying technology is maximally exploited, attention frequently turns to hybrid or cascaded imaging systems that are complex and expensive [17]. When the performance limit is eventually reached and both the scan pattern and hardware options are exhausted, a compromise as dictated by the frame rate–resolution limit becomes inevitable.

Here, we introduce a scene-reactive adaptive scanning paradigm to imaging that decouples the frame rate from the resolution by selectively scanning the scene, which escapes the fundamental trade-off that otherwise relates the two. Our approach differs from existing compressed sensing techniques for scanned modalities [24,25] by altering the selective scan pattern in response to scene dynamics on a per-frame basis. With a real-time implementation of adaptive scanning, we demonstrate frame rate speedups in scenes with biomedical relevance, including simulated ophthalmic microsurgery and imaging of multiple moving organisms or targets. Notably, this algorithm requires no knowledge of the underlying scene structure, seamlessly reduces to uniform scanning for highly active scenes, and generates feasible time-optimal scanner trajectories on the fly.

2. BACKGROUND

A. Scanned Imaging Modalities

Scanned imaging modalities exhibit both spatial and temporal performance characteristics as governed by their acquisition field and sampling rate, respectively. These two properties, as further defined below, govern the speed with which a scanned imaging system can accomplish its task of scanning the imaging scene.

The acquisition field describes the spatial extent of simultaneous data collection, as determined by the system’s light collection and detection scheme. In most cases, the detector’s geometric dimensionality corresponds to the acquisition field’s geometry, as shown in Figs. S1(a)–S1(c) in Supplement 1. For example, a point detector (0D) is naturally employed to transduce light collected from a point (or “spot”) acquisition field. Line-scan (1D) and area-scan (2D) imaging systems similarly exhibit the use of line and area cameras, respectively. From a spatial standpoint, the scanning performance is improved through increased detection parallelism rather than an increased acquisition field area. The sampling rate describes how rapidly the system can collect data from the acquisition field. This rate is predominantly determined by the system’s sample interrogation and signal digitization scheme; however, many other factors, such as computational requirements and sample power limits [26,27], may influence this rate. In optical coherence tomography (OCT), common determinants of the sampling rate are the swept-source repeat rate, the camera line-scan rate, and the reference arm sweep rate for swept-source [28], spectral-domain [29], and time-domain OCT [9], respectively. From a temporal standpoint, the scanning performance is improved through increased sampling rates, allowing the detection of more acquisition fields per unit time.

For many imaging tasks, the desired field of view (i.e., the spatial extent of data collection) is much larger than the acquisition field (i.e., the spatial extent of the simultaneous data collection). This situation is almost universal for point- and line-scan systems, but may also occur for area-scan systems, where the terms montaging [30] and mosaicking [31] are commonly applied. The well-established solution to achieve a field of view larger than the acquisition field is to move the acquisition field to incrementally populate the imaging scene, as shown in Figs. S1(d)–S1(f) in Supplement 1. Certain applications may instead move a sample through a stationary acquisition field, but this analysis applies equally well in either rest frame. The scene is the set of positions throughout the field of view to which the acquisition field is moved for data collection, as shown in Fig. S1 in Supplement 1. A frame is the image or volume that results from having acquired data once from each position within the scene, and the time required to assemble the frame determines the frame rate. The number of positions within the scene (i.e., its cardinality) determines the scene resolution. We consider this term in the imaging sense (e.g., pixel count) rather than the optical sense (e.g., resolvable spot size). For example, the raster scan pattern in Fig. S1(d) in Supplement 1, which uniformly samples a rectangular field of view using 2D grid of points with $n$ rows and $m$ columns, has a resolution of $n \times m$. This definition of scene resolution is sufficiently general to accommodate other scan morphologies, including radial, Lissajous [15], or spiral patterns [16,17].

B. Frame Rate–Resolution Trade-off

For a scanned imaging modality, the frame rate $F$ achieved when populating an imaging scene with image resolution $R$ at a sampling rate of $S$ is expressed as

In practice, $FR$ is an ideal upper bound because the imaging system cannot instantaneously reposition the acquisition field. For systems with fixed sampling rates, if the time required to move the acquisition field from one position to another exceeds the sampling interval $1/S$, off-target (or “inactive”) samples are acquired while the acquisition field is transitioning. In common practice, such off-target samples are considered “flyback artifacts” or “warping” and are ultimately discarded [32,33]. For systems with dynamically adjustable sampling rates, the same phenomenon occurs, except such systems postpone the next sampling event to avoid collecting the undesired data (e.g., [34] in reverse). These off-target samples, acquired or not, prolong the frame time $T$ like an on-target (or “active”) sample and can therefore be considered as increasing the resolution without contributing to the output frame. The theoretical limit in Eq. (2) can be restated more realistically to distinguish between on- and off-target samples as

 

Fig. 1. Frame rate-resolution trade-off for a fixed sampling rate $S = 100\;{\rm kHz}$ and varying scan efficiencies. The maximum efficiency scan ($\beta = 1$) defines the boundary between feasible (white) and infeasible (gray) frame rate–resolution pairs. Increasing scan efficiency toward 100% shifts the frame rate–resolution curve closer toward the limit (red → blue → green → purple), but the limit remains unchanged (orange). Therefore, for every on-target resolution ${R_{{\rm on}}}$, there exists a maximum frame rate; higher on-target resolutions necessitate a reduction in frame rate.

Download Full Size | PDF

3. SCAN EFFICIENCY

The positions of on-target samples during a scan are designed to satisfy the requirements of an imaging task. In contrast, the positions of off-target samples are determined by the interaction between the scan pattern and scanner hardware. The scan pattern is the sequence in which on-target positions are visited, subject to dynamics limits imposed by the scanner hardware. For a given imaging task, maximization of the scan efficiency therefore amounts to the design of scan patterns and the selection of scanner hardware so as to minimize ${R_{{\rm off}}}$. The ability to minimize ${R_{{\rm off}}}$ varies with the scan pattern, and we classify patterns as monolithic, as described in Sec. S1 in Supplement 1, or segmented based on this property.

A. Segmented Scan Patterns

We consider a scan pattern segmented when the design process for the on- and off-target samples is loosely coupled or completely decoupled. For these scans patterns, the imaging task determines the on-target sample positions, but has limited influence on the off-target sample positions. The resulting scan pattern consists of alternating groups of contiguous on-target samples (“active” segments) and contiguous off-target samples (“inactive” segments). The 2D raster scan is a prototypical segmented scan pattern, consisting of a sequence of parallel and linear active segments (i.e., B-scans) separated by inactive segments for scanner turnaround. Another example is the unidirectional constant linear velocity spiral scan [17], which consists of a single active segment (the spiral) followed by a single inactive segment that returns to the spiral center to begin the next frame. The design of these inactive segments, which connect each active segment with the next, is left unspecified by the overall pattern.

The design of inactive segments infrequently receives considerable attention except in specific contexts [34–37]. This is unfortunate because the combined length in samples of all inactive segments is precisely ${R_{{\rm off}}}$, the primary tunable variable in scan efficiency. As a concrete illustration, consider again the ubiquitous 2D raster scan as executed by a point-scan imaging system. This scan pattern is achieved by driving the fast-axis scanner with a sawtooth waveform [Fig. 2(a)]. Such a waveform is simple to generate, but infeasible to execute due to discontinuities in position and its time derivatives that command the scanners to instantaneously reposition between fast-axis sweeps. These discontinuities undergo smoothing first by the scanner controller, which must respect drive current or voltage limits, and second by the scanner itself, which is subject to its own mechanical or electrical dynamics (e.g., mirror inertia). Smoothing manifests as distortion since the scanner’s failure to execute the prescribed pattern yields data acquisition at unintended positions. Cropping is the nearly universal approach [32,33] to handling distortion, by empirically inspecting that data to determine the number of samples to discard. Some have sought to improve scanner turnaround between fast-axis sweeps by replacing the sawtooth waveform with a triangular one, as shown in Fig. 2(d) [39,40]. The pattern remains infeasible, however, as the discontinuities persist in velocity and higher-order derivatives, but the distortion is nevertheless improved. Still others have tuned scanner hardware or augmented scan waveforms to improve turnaround [33], but the empirical nature of these approaches generalizes poorly when new imaging requirements demand adjustments in the waveform period or amplitude. In general, prior work has emphasized scanner tuning and conceded the distortion that results from infeasible patterns. Like [35], we instead consider what waveforms would accomplish the scan with minimal time and distortion given well-tuned hardware.

 

Fig. 2. Sawtooth (a–c), triangular (d–f), and limit-satisfying (g–i) scan waveforms for a raster scan’s fast axis. The sawtooth waveform exhibits (a) position and (b) velocity discontinuities that yield (c) acceleration spikes, rendering it infeasible to execute. The triangular waveform avoids (d) position discontinuities, but still exhibits (e) velocity discontinuities that require (f) alternating acceleration spikes, also rendering it infeasible to execute. The limit-satisfying waveform designs (g) a piecewise-quadratic return segment that operates precisely at (h) scanner velocity and (i) acceleration limits to achieve a minimum turnaround time. In this illustration, the forward segment (black) is twice the duration of the return segment (red). Extension to piecewise-cubic segments that satisfy jerk (e.g., $\mathop{x}\limits^{.\,\!.\,\!.}$) limits also is possible [38].

Download Full Size | PDF

B. Time-Optimal Segmented Scan Patterns

We propose an alternate approach to scan pattern generation that extends the state of the art [41] by explicitly accounting for scanner dynamics limits in a general-purpose manner and subsequently minimizes ${R_{{\rm off}}}$. The result is the generation of intrinsically feasible scan patterns that execute with negligible distortion in minimal time [Fig. 2(g)] for properly chosen limits. We assume without loss of generality that every active segment is already feasible, because any infeasible active segment may be decomposed into feasible ones separated by inactive segments. The optimal scan pattern design task thus reduces to generating inactive segments with minimal duration. Minimal inactive segment durations ensure a minimal number of off-target samples because samples occur at fixed time intervals. Moreover, these inactive segments must match the exit position and velocity of the prior active segment as well as the entry position and velocity of the subsequent active segment to avoid discontinuities, as shown in Figs. 2(g) and 2(h).

To pose the inactive segment design problem more precisely, let ${p_i}(t)$ represent the position of the $i$th active segment with duration $\Delta {t_i}$ for $t \in [0,\Delta {t_i}]$. Further, since each active segment is intrinsically feasible as above, we have

 

Fig. 3. Illustration of the scan pattern design for three segments of a bidirectional raster scan in the $x {-} y$ plane. The on-target segments (black) are connected by off-target transition segments (red) that are designed to match entry and exit velocities (blue) at zero acceleration.

Download Full Size | PDF

For multidimensional scanner hardware, these requirements apply to each degree of freedom independently. With such a waveform and well-determined velocity and acceleration limits, the acquisition field arrives at every sample position at the designed time, which eliminates distortion due to infeasible scan waveforms. In addition, the scanner enters each active segment having already accelerated to the appropriate velocity for sampling near the segment endpoints.

This problem of finding the optimal waveform ${q_i}(t)$ is precisely a point-to-point trajectory generation problem that is commonly encountered in motion planning for robots [38,42]. Well-established solution techniques exist for computing such trajectories for arbitrary degrees of freedom in sub-millisecond times [38]. Briefly, the time-optimality of a given trajectory is achieved by operating at least one degree of freedom at its velocity or acceleration limit at all points along the trajectory, an approach known as bang-bang control ([43], Section ${\sim}{14.6.3}$). Such a trajectory accelerates and travels as fast as possible while respecting the dynamics limits of the system, as illustrated in Figs. 2(g), 2(h), and 2(i). A shorter duration trajectory is not possible because it would imply a violation of velocity and/or acceleration limits for that degree of freedom [42]. This technique is directly suitable to calculate inactive segment waveforms, which we subsequently refer to as trajectories, between active segments. With this approach, we then have ${R_{{\rm on}}} = S\sum\nolimits_i \Delta {t_i}$, as chosen for the imaging task, and ${R_{{\rm off}}} = S\sum\nolimits_i \Delta t_i^\prime $, as determined by scan geometry, resolution, and scanner limits. The scan efficiency becomes

C. Efficiency of Grid Scan Patterns

Point-scan imaging systems employ a variety of common scan patterns, including grid (raster and radial) [18], Lissajous [15], and constant angular [16] and linear [17] spiral patterns, each of which exhibits its own efficiency considerations. This section specifically considers the efficiency of 2D grid scan patterns that find utility in the context of adaptive scanning. 2D grid patterns form a uniformly spaced grid in a 2D parametric space. This class of patterns naturally includes raster and radial scans, which create grids in Cartesian ($x {-} y$) and polar coordinates ($r {-} \theta$), respectively. Resonant raster scans [14] (i.e., raster scan with sinusoidal fast sweep) also can be considered under this formalism because the desirable portion of linear sweep approximates uniformly spaced samples. For clarity of presentation, we now focus on a point-scan imaging system and a traditional raster scan pattern, although an equivalent analysis may be carried out for other systems and segmented scan patterns.

Consider a raster scan pattern for a rectangular scene with dimensions $h \times w$ and resolution $n \times m$ (${\rm rows} \times {\rm columns}$) that is performed using scanner hardware capable of maximum velocity ${v_{{\rm max}}} = 80 \times {10^3}^ \circ {{\rm s}^{- 1}}$ and acceleration ${a_{{\rm max}}} = 25 \times {10^6}^ \circ {{\rm s}^{- 2}}$. These limits were hand-tuned as per Section 3.B for ScannerMax Saturn 1B galvanometers (Pangolin Laser Systems, Orlando, FL) operated in factory tuning #4 [33]. Traditionally, the raster pattern visits the scene row by row, from left to right and top to bottom, as shown in Fig. S2(a) in Supplement 1, although other orientations are possible. Each row is an active segment with a constant scan velocity ${v_s} = Sw/m$ and a duration $\Delta t = w/{v_s} = m/S$, such that ${v_s} \lt {v_{{\rm max}}}$, yielding no off-target samples. Between each active segment, however, ${I_s}$ many off-target samples are acquired while the scanner repositions to start the next row, and ${I_v}$ many off-target samples are similarly acquired while the scanner repositions to start the next volume. ${I_s}$ and ${I_v}$ are therefore the lengths of the intersegment and intervolume inactive segments. All segments share the same ${I_s}$ since translation-only shifts between successive segments do not change the dynamics for trajectory generation. For this rectangular scan, ${R_{{\rm on}}} = nm$ and ${R_{{\rm off}}} = n{I_s} + {I_v}$, yielding an overall scan efficiency of

Therefore, for fixed ${I_s}$ and ${I_v}$, the scan efficiency is maximized ($\beta \to 1$) as the scan resolution is increased ($nm \to \infty$), with a particular sensitivity to increased columns $(m)$. Furthermore, for scenes with approximately square aspect ratios ($n \approx m$), the effect of ${I_s}$ dominates the effect of ${I_v}$.

In practice, ${I_s}$ and ${I_v}$ exhibit a complex dependence on scanner dynamics limits (${v_{{\rm max}}}$ and ${a_{{\rm max}}}$), scan velocity (${v_s}$), resolution ($n$ and $m$), and pattern geometry ($h$ and $w$). Clearly, $\beta \to 1$ as ${I_s} \to 0$ and ${I_v} \to 0$, as previously observed, so designing the scan with minimum ${I_s}$ and ${I_v}$ as consistent with Section 3.B is appropriate. For this two degrees-of-freedom scan pattern, the time-optimal inactive segment trajectory ${q_i}(t) = (x,y)$ for $i \in [0,n - 1]$ is described by

This analysis applies to both unidirectional and bidirectional 2D grid scan patterns. Extension to bidirectional scans, as shown in Figs. S2(c) and S2(d) in Supplement 1, where the active segments alternate directions, requires only alternating the $x$ positions of Eq. (14) and the signs of Eq. (15), such that

D. Influence of Scanner Technology

The technology that underlies a given scanner indirectly enters into the considerations of scan efficiency through dynamics limits. Higher limits, particularly in acceleration, support shorter inactive segments that yield faster scan execution times, as shown in Fig. S3(b) in Supplement 1. The largest determinant of a scanner’s dynamics limits is the whether its deflection technique is inertial or noninertial [44,45]. Inertial approaches involve the physical movement of a mirror or another optical element to deflect the incident beam. Examples include galvanometric, resonant, MEMS, piezoelectric, and polygonal scanners. The scanner dynamics limits arise immediately as a consequence of mirror inertia and available force/torque to manipulate that mirror. By contrast, noninertial approaches employ nonmechanical optical phenomena to deflect the incident beam. Examples include electro-optic and acousto-optic deflectors. The achievable scan rates with these deflectors are frequently $10 {-} 100 \times$ faster than inertial techniques [44]. As seen in Fig. S3(b) in Supplement 1, the differences in the dynamics limits of these orders can substantially affect the overall performance. Non-inertial scanners exhibit a smaller range of deflection angles, however, that limit the number of individually addressable positions and therefore the final scan resolution. These differences give rise to the “inertial gap” [44] or “inertial limit” [45] that separate inertial and noninertial scanner technologies. The needs of current clinical imaging, especially with OCT in ophthalmology, has strongly favored high addressability over high dynamics limits. This is in part because high-density scan patterns or slow acquisition speeds [46,47] did not push the scanner hardware to its limits. Nevertheless, understanding and operating at the scanner limits without violating them is essential to achieve maximum scan performance.

 

Fig. 4. Overview of the adaptive scanning approach for a 2D scene on a rectangular grid. Based on (a) the priority of each sample position, (b) a subset is selected to scan during the next frame. (c) A scan pattern is then generated to efficiently visit the select sample positions. (d) The change between successive frames drives (a) the priority for the subsequent frame.

Download Full Size | PDF

4. ADAPTIVE SCANNING

Adaptive scanning escapes the frame rate–resolution limit by visiting each scan position only as needed instead of in rigid sequence, as shown in Fig. 4. The full frame is then reconstructed for both visited and unvisited positions to maintain the scene resolution. After each frame, the scene is analyzed, based on current and previously acquired data, to determine which positions to visit during the subsequent frame. This sparse acquisition approach is suitable for imaging scenes that contain dynamic features on an otherwise static background and stands in contrast to dense acquisitions that sample the complete scene for each frame. Potential and ubiquitous applications include imaging ophthalmic surgical instruments (e.g., forceps) that interact with tissue at specific points (e.g., grasped tissue), targets suspended in solution (e.g., cells in aqueous humor), or organisms moving throughout a confined space. Rather than distributing samples uniformly across the scene, adaptive scanning seeks to concentrate them on areas of interest and thereby reduce the number of samples required to refresh the frame. We first discuss the performance benefits of adaptive scanning in Section 4.A and then present a concrete implementation for scene analysis, update, and reconstruction in Section 4.B.

A. Efficiency of Adaptive Scanning

Whereas a traditional scan densely samples all on-target positions ${R_{{\rm on}}}$, an adaptive scan instead visits a subset of those positions

For those cases with ${\beta _a} \gt 1$, adaptive scanning produces performance (i.e., a frame rate and resolution product) with an effective sampling rate of ${S^\prime} = {\beta _a}S$, which would have otherwise required faster hardware to obtain. Consequently, one can consider it to offer an effective frame rate of ${F^\prime} = {\beta _a}F$, an effective resolution of $R_{{\rm on}}^\prime = {\beta _a}{R_{{\rm on}}}$, or any mixture of the two. ${\beta _a}$ subsequently functions like a frame rate speedup factor or a resolution multiplier over a traditional dense scan. It is by these means that adaptive scanning breaks the frame rate–resolution limit to operate in the infeasible region of Fig. 1. This capability is necessarily predicated on the selection of proper scenes that permit $\alpha \ll 1$ and exhibit spatial update patterns that promote ${R_{{\rm off}}} \ll {R_{{\rm on}}}$. Notably, when such conditions do not exist, ${\beta _a} \to \beta$ as $\alpha \to 1$ such that adaptive scanning transparently reduces to traditional dense scanning.

B. Probabilistic Adaptive Scanning

We now present a modular and general-purpose approach to adaptive scanning using a probabilistic technique for scene analysis, update, and reconstruction. Notably, this approach requires no knowledge of the underlying scene structure. We again consider a rectangular scene with dimensions $h \times w$ at resolution $n \times m$, as this satisfies the vast majority of imaging tasks, but alternative scene topologies work equally well. Probabilistic adaptive scanning operates cyclically in the four phases despicted in Fig. 4. Below, individual scene positions are identified by the subscript $i$ whereas values that vary with the frame number are identified by the subscript $j$.

1. Update Selection

The update selection phase determines which scene positions to visit during the next frame, as shown in Fig. 4(b). This is based on the identification of dynamic positions or areas of interest within the scene, primarily using measures of signal change. Each scan position is assigned a scan probability

An $n \times m$ Boolean scan map ${[{{V_j}}]_i} = {p_i} \le {u_i}$ is created from these probabilities by sampling the uniform random variable $u \sim U(0,1)$ to determine which positions to visit during the current frame. Positions exhibiting a dynamically changing signal are therefore selected more frequently for re-imaging than positions with a comparatively static signal. A key consideration is balancing the re-imaging of known dynamic positions with exploration to identify undiscovered dynamic positions. It is clearly desirable that all positions are visited eventually so that unknown areas of interest are identified. This probabilistic approach naturally accommodates random exploration within the adaptive scanning framework. Alternatively, the periodic insertion of dense scans by setting ${[{{V_j}}]_i} = 1$ can enforce a minimum visitation interval for desired positions.

2. Scan Generation

The scan generation phase produces a scan trajectory to visit the selected positions using the time-optimal technique in Section 3.B, as shown in Fig. 4(c). We apply binary dilation to the raw scan map so that neighborhoods around each selected position also are included. This accommodates object movement, collects context from adjacent scene positions, and consolidates selected positions. We then perform connected-component analysis to cluster selected positions, which reduces off-target scanner travel across the scene and thereby improves scan efficiency. Raster segments are generated for each connected component and dispatched to a time-optimal scan trajectory planner based on the Reflexxes motion library [42].

The use of time-optimal trajectories in scan patterns is essential in adaptive scanning for multiple reasons. First, the probabilistic selection of positions to visit yields highly variable and seemingly random scan waveforms, rendering conventional by-hand or waveform-specific tuning of scanner turnaround impractical. Second, the scanners must reach every target position at the pre-planned time to acquire the correct data. These irregular scan patterns violate the assumptions of traditional techniques such as frame cropping for handling scan distortion. Third, adaptive scanning seeks to unlock frame rates that were previously unachievable for a given resolution. It is therefore critical that the scan pattern is executed as quickly as possible so the resultant frame rate is maximized.

3. Scene Reconstruction

The scene reconstruction phase assembles a full frame from the sparse acquisition time series. The first frame ${S_0}$ is either unpopulated (e.g., ${S_0} = 0$) or initialized with a uniform acquisition of the complete scene. Subsequent sparse acquisitions (i.e., $\alpha \lt 1$) update only the acquired positions within the scene. This is accomplished by aligning the acquired data with the time-shifted scan trajectory to account for the fixed scanner response latency. Using trajectory information, the data for each position is mapped into the intended coordinates within the frame. Scan positions that are not visited carry forward their current data into the next frame such that ${S_j}(p) = {S_{j - 1}}(p)$, where $p$ represents an unvisited position. This constitutes a zero-order hold for unvisited positions, although other strategies are possible.

4. Priority Propagation

The priority propagation phase transforms the signal change between frames into the priorities that drive the selection of the next scan positions. The choice of transformation depends on the scene composition. For large differences in signal between static and dynamic scene regions, an appropriate intensity-based interframe change at each position is

Using the interframe change $\Delta$, the change map ${C_j}$ is updated in two steps, as shown in Fig. 4(d). The first step,

5. METHODS

We evaluated probabilistic adaptive scanning in simulated and physical imaging scenes, including living organisms. Simulation experiments enabled us to establish the ground truth and therefore directly calculate the adaptive scanning scene reconstruction error. Physical imaging scenes provided more realistic imaging tasks and exposed the algorithm to real-time execution constraints.

A. Computational Experiments

We developed a Python-based environment for adaptive scanning that simulated the acquisition of data with and without noise from a scene with moving objects. We chose a scene with the dimensions ${9^ \circ} \times {16^ \circ}$ and a resolution of $250 \times 1000$ because this represented a typical imaging task for OCT applications. Objects and empty space within the scene reported signal values of 1 and 0, respectively. We did not simulate full-depth profiles (e.g., A-scans) because our selected change metric in Eq. (25) did not require that information. We added uniformly distributed noise $U(0,0.2)$ into the simulated data to account for acquisition noise.

Using this simulation environment, we designed two imaging tasks. The first task exercised the ability of adaptive scanning to distinguish dynamic from static scene positions. This scene featured a single homogeneous object moving along a Lissajous trajectory at the edge of an otherwise empty scene. The second task exercised the ability of adaptive scanning to handle many diverse objects moving in different directions and the appearance of new objects. This scene included many objects of varying sizes that moved along Lissajous or random walk trajectories. These objects moved throughout the scene in addition to entering and exiting the field of view, such that approximately 50–60 objects were visible within the scene at any given instant. We performed adaptive scanning in the simulation of these tasks for 500 frames each.

For scan pattern generation, we used maximum velocity $v_{{\rm max}}=80\times 10^{3^\circ} {{\rm s}^{- 1}}$ and acceleration ${a_{{\rm max}}} = 25 \times {10^6}^ \circ {{\rm s}^{- 2}}$, which we had previously hand-tuned for our galvanometers. We dilated the raw scan mask with a kernel of size 0.25° prior to scan trajectory generation. Connected components within the final scan mask were individually rasterized and visited sequentially, as shown in Fig. 4(c). Notably, the simulator did not model the evolution of the scene while the scan pattern executed to avoid interaction between the scan execution time and the acquired object positions (i.e., rolling shutter). Simulated frames consequently occurred at fixed time intervals with a global shutter. Each simulated task began with two fully acquired frames to initialize the scene change map.

Since our simulator did not produce depth profiles, we adopted the change metric in Eq. (25) that relies only upon en face projections to compute interframe change. We selected a diffusion kernel of size 0.25° with a standard deviation of 0.125° to match the scan mask dilation kernel size. We used ${\lambda _g} = 0.0001$, ${\lambda _d} = 1$, ${\lambda _e} = 0.5$, ${\lambda _m} = 0$, and ${\lambda _w} = 1$ after tuning to obtain error fractions of nominally 1%. We determined the adaptive scanning error rate as the fraction of all positions within the reconstructed scene (visited and unvisited) that differed from the ground truth, under the assumption of no acquisition noise. These parameters prioritized the scanning of neighboring positions to best adapt to moving objects. During computational experiments, we recorded the error rate, scan fraction $\alpha$, and adaptive scan efficiency ${\beta _a}$. We excluded the initial 50 frames from analysis while the adaptive scanning algorithm converged on dynamic scene positions.

B. Physical Experiments

We performed physical experiments to validate scan trajectory generation (Section 5.B.2) and to demonstrate probabilistic adaptive scanning in realistic OCT imaging tasks (Section 5.B.3). All such experiments were performed with the same OCT system described in Section 5.B.1.

1. OCT System and Engine

We acquired OCT data using a custom telecentric scanner with a ScannerMax Saturn 1B galvanometers (Pangolin Laser Systems, Sanford, FL) and a 26 mm working distance. These galvanometers were chosen for their low-resistance coils and low-inertia rotors, which provide favorable dynamics for rapidly varying scan patterns. We optimized the telecentric scanner design to achieve diffraction-limited OCT performance at 1060 nm over a ${\pm}7\;{\rm mm}$ field of view, yielding a theoretical lateral OCT resolution of 25 µm for a 3.2 mm input beam. We operated the scanner with a custom-built swept-source (SS) OCT engine using a 1060 nm swept frequency source (Axsun Technologies, acquired by Excelitas Technologies, Waltham, MA, in 2018) with 100 nm bandwidth at a 200 kHz A-scan rate. The optical signal detection chain used a $1.8\;{\rm GS}\;{{\rm s}^{- 1}}$ digitizer (AlazarTech, Pointe-Claire, Quebec, Canada) to measure the output of a balanced photoreceiver (Thorlabs, Newton, NJ). We performed real-time OCT processing and display on the graphics processing unit (GPU) using custom Python software developed with the open-source Vortex OCT library [48]. An extended imaging depth with A-scans of $2048 \times$ was obtained through $2 \times$ spectral upsampling.

2. Scan Pattern Trajectory Validation

We validated the ability of our galvanometers to faithfully follow designed trajectories by concurrently recording the position command and feedback signals from their driver. We configured the driver with factory tuning #4 [33], which favors a low response time and high scan speeds. Time-optimal adaptive scanning waveforms with ${v_{{\rm max}}} = 80 \times {10^3}^ \circ {{\rm s}^{- 1}}$ and ${a_{{\rm max}}} = 25 \times {10^6}^ \circ {{\rm s}^{- 2}}$, as previously tuned, were then generated and executed. After correcting for the fixed response latency, we compared the command and feedback waveforms and computed the rms tracking error during the active segments.

3. Imaging in Model Systems

We compared traditional raster scanning and adaptive scanning through imaging the “coffee,” “aphids,” and “intrasurgical” model systems described below with both scanning techniques. The coffee model consisted of finely ground coffee suspended in water and subjected to gentle agitation. Numerous small grains not only floated throughout the imaging scene, necessitating higher resolution to detect each one, but also periodically coalesced into larger clusters on a time-scale that required higher frame rates to be best observed. The aphids model featured several aphids (superfamily Aphidoidea) exploring a culture plate. This task required high resolution to resolve minute features, such as the aphid antennae, and a high frame rate to capture rapid movements. Indeed, we briefly subjected the aphids to cold temperatures immediately before imaging to reduce their activity to levels appropriate for the scanner’s field of view. The intrasurgical model involved a 25-gauge vitreoretinal forceps (Bausch + Lomb, Vaughan, Ontario, Canada) suspended over an open-globe porcine retina to simulate ophthalmic surgery maneuvers. A motorized stage moved the instrument at $0.1\;{\rm mm}\;{{\rm s}^{- 1}}$ across the scene to simulate surgeon motion, which provided near-perfect repeatability between both imaging techniques, unlike the other two models.

For all systems, we chose a scene with dimension of $14 \; {\rm mm} \times 14\;{\rm mm}$ and resolution of $1000 \times 1000$ for high-resolution imaging tasks at our scanner’s maximum field of view. This imaging configuration operated at nearly 90% of Nyquist sampling at our scanner’s lateral resolution. We used the same scan pattern generation, scene reconstruction, and scanner limits as in Section 5.A with two exceptions. First, we dilated the raw scan mask with a kernel of size 0.56 mm. Second, we rotated the scene positions by 45° with respect to the scanner axes to balance the turnaround between the two galvanometers. This provided a theoretical $\sqrt 2$ increase in maximum velocity and acceleration because each galvanometer needed only to contribute $1/\sqrt 2$ of the scan velocity or acceleration. In this configuration, uniformly scanning one volume required 5.640 s (0.177 Hz) for our 200 kHz source when accounting for scanner dynamics during turnaround, yielding a baseline scan efficiency $\beta = 0.887$. To meet real-time requirements without dead time between frames, we completed the stages of adaptive scanning in bands across the scene using an incremental, pipelined approach. If the computation overran the available time, such as may happen during very low scan fractions (i.e., $\alpha \lt 0.05$), the scanners briefly held their position until a new trajectory became available.

Since these model systems exhibited changes in both depth and reflectivity, we adopted the axial change metric in Eq. (26) to compute the interframe change. We selected a diffusion kernel of size 0.56 mm with a standard deviation of 0.056 mm to match the scan mask dilation kernel size. We used ${\lambda _g} = 0.0001$, ${\lambda _d} = 1$, ${\lambda _e} = 0.1$, and ${\lambda _m} = 0$ after tuning to yield a good performance. Each scene required its own ${\lambda _w}$; however, due to differences in object reflectivity, manual tuning yielded ${\lambda _w} = 10$ for the coffee grounds, ${\lambda _w} = 100$ for the aphids, and ${\lambda _w} = 30$ for the intrasurgical model systems. We implemented probabilistic adaptive scanning for online operation in pure Python with GPU acceleration.

We conducted imaging sessions for 60 s in each model system using both traditional raster scanning and adaptive scanning. Each adaptive scanning imaging session began with two densely acquired frames to initialize the scene change map. We measured the number of source sweeps required to complete each frame to determine the instantaneous frame rate. This calculation included any delays incurred due to the computational overruns described above. Since the frame rate varies as adaptive scanning responds to scene dynamics, we reported the effective frame rate by averaging the volume intervals for the final 45 s of imaging time. This allowed the adaptive scanning algorithm the opportunity to scan the complete scene initially and for the priority map to stabilize before computing the effective frame rate. As with computational experiments, we recorded the scan fraction $\alpha$ and adaptive scan efficiency ${\beta _a}$ for each volume. To explicitly compare to traditional raster scanning, we computed the speedup factor for each frame individually using

 

Fig. 5. Scan fraction ($\alpha$, top), adaptive efficiency (${\beta _a}$, middle), and error rate (bottom) trends and means (right) for the single- and multitarget scenes during adaptive scanning in simulation. The first 50 frames are excluded from the mean to allow the algorithm to converge on dynamic regions. Adaptive scanning maintained low error rates (${\lt}0.05$) despite scanning substantially less than the complete frame ($\alpha \lt 0.25$) for both scenes.

Download Full Size | PDF

6. RESULTS

Figure 5 summarizes the performance results of adaptive scanning of the single-target and multitarget tasks in simulation, as shown in Figs. S4 and S5, respectively, in Supplement 1. In both cases, the algorithm quickly identified the dynamic scene positions and reduced the frequency of visits elsewhere. For the single target task, adaptive scanning performed with a mean efficiency ${\beta _a} = 2.43$, visiting on average 15% of the scene positions, after recognizing that the object’s homogeneous interior did not require frequent updates (Visualization 1). During scene reconstruction, the error rate oscillated with a mean of 0.9% as the target changed directions and peaked at 3.2%. Similarly, for the multiple target task, adaptive scanning performed with a mean efficiency ${\beta _a} = 1.84$, visiting on average 21% of scene positions, despite large size and velocity variations between the targets (Visualization 2). It furthermore detected the entry of previously unseen targets into the scene through random exploration. During scene reconstruction, the error rate stabilized to a mean of 0.8% and occasionally peaked up to 3.8% for previously stationary targets or ones entering the field of view.

Figures 6–8 summarize the performance of adaptive scanning in, respectively, the coffee, aphids, and intrasurgical model systems. In all three cases, the algorithm detected the dynamic scene elements and equilibrated to a low scan fraction without compromising image quality within $\approx 5\;{\rm s}$. The scan maps for each task reflect identification of the floating coffee grounds, crawling aphids, or moving intrasurgical forceps on which the algorithm focused the scanning effort. For the coffee model, adaptive scanning achieved a mean adaptive scan efficiency and speedup factors of $\beta = 4.79$ and $\gamma = 5.40$, respectively, by selectively visiting the central cluster and smaller clumps of coffee grounds (Visualization 3). When imaging aphids, adaptive scanning achieved a mean adaptive scan efficiency and speedup factors of $\beta = 5.98$ and $\gamma = 6.75$, respectively, by identifying and scanning only those aphids that exhibited movement, even after several volumes of inactivity (Visualization 4). In the intrasurgical model, adaptive scanning achieved a mean adaptive scan efficiency and speedup factors of $\beta = 5.75$ and $\gamma = 6.48$, respectively, by distinguishing the dynamic forceps from the static retina (Visualization 5). In addition, the algorithm updated scattered debris dragged along as the forceps moved through the viscoelastic gel preserving the retina, although the smallest pieces intermittently escaped tracking. Visualization 3, Visualization 4 and Visualization 5 demonstrate the frame rate improvement and adaptive image quality for each model system with a side-by-side comparison to conventional raster scanning.

 

Fig. 6. Adaptive scanning of coffee grounds suspended in gently agitated water (top right), showing reconstructed scene and scan map (top) for specific volumes as well as the speedup factor $\gamma$ (middle) and adaptive scan efficiency ${\beta _a}$ (bottom) for each volume. The algorithm identified the sparse dynamic structure shortly after initialization and followed multiple targets throughout the scene without neglecting the central cluster. Image quality was preserved despite low scan fractions of $\alpha \in [0.10,0.15]$, which yielded mean speedup factor of $\gamma = 5.40$ and mean adaptive scan efficiency of ${\beta _a} = 4.79$. Dashed red lines indicate effective performance of a uniformly scanning 1 MHz system (middle) and the maximum efficiency of uniform scanning (bottom). See Visualization 3 for full demonstration.

Download Full Size | PDF

 

Fig. 7. Adaptive scanning of aphids (top right), showing reconstructed scene and scan map (top) for specific volumes as well as speedup factor $\gamma$ (middle) and adaptive scan efficiency ${\beta _a}$ (bottom) for each volume. The algorithm identified which aphids were moving and ceased frequent scanning of the static ones near the bottom of the scene. Image quality was preserved despite low scan fractions of $\alpha \in [0.05,0.09]$, which yielded mean speedup factor of $\gamma = 6.75$ and mean adaptive scan efficiency of ${\beta _a} = 5.98$. Dashed red lines indicate effective performance of a uniformly scanning 1 MHz system (middle) and the maximum efficiency of uniform scanning (bottom). See Visualization 4 for a full demonstration.

Download Full Size | PDF

 

Fig. 8. Adaptive scanning of vitreoretinal surgical forceps over a porcine retina (top right), showing reconstructed scene and scan map (top) for specific volumes as well as speedup factor $\gamma$ (middle) and adaptive scan efficiency ${\beta _a}$ (bottom) for each volume. The algorithm identified the forceps as the primary scene element and subsequently skipped the static background. Image quality was preserved despite low scan fractions of $\alpha \in [0.11,0.14]$, which yielded a mean speedup factor of $\gamma = 6.48$ and a mean adaptive scan efficiency of ${\beta _a} = 5.75$. Dashed red lines indicate the effective performance of a uniformly scanning 1 MHz system (middle) and the maximum efficiency of uniform scanning (bottom). See Visualization 5 for a full demonstration.

Download Full Size | PDF

Figure S6 shows galvanometer position command and feedback waveforms during adaptive scanning with the aphids model systems. With time-optimal trajectory generation to respect galvanometer dynamics limits, we observed appropriate tracking despite the highly variable nature of the scan pattern. The $x$ and $y$ galvanometers exhibited rms tracking errors of 0.0235° and 0.0241°, respectively, computed across all three adaptive scanning imaging sessions.

7. DISCUSSION

Compared to uniform scanning at the same scene resolution, our adaptive scanning technique achieved an unprecedented performance level as measured by the frame rate. For physical imaging experiments, we obtained a real-time mean speedup factor of $\gamma \gt 5$ in all three model systems, including losses due to computational overruns. Based on these results, our 200 kHz SS-OCT system achieved a frame rate that would have otherwise required a 1 MHz or faster source. This is seen in Fig. 9, where adaptive scanning breaks the frame rate–resolution limit to operate at traditionally infeasible frame rates beyond the $\beta = 1$ limit for uniform scanning. As a software-only approach, adaptive scanning shows particular promise for improving frame rates of point-scan OCT systems without mandatory hardware changes.

 

Fig. 9. Frame rate resolution limit for the unidirectional raster scan chosen for physical experiments in Sec. 5.B.3. For each model system (red, green, blue), adaptive scanning operated well within in the traditionally infeasible reason (gray) from which the uniform raster scan (black) is excluded.

Download Full Size | PDF

The ability to adapt scanning in real-time to the imaging scene is relevant for many imaging tasks, especially when this capability can push the frame rate beyond traditionally achievable frequencies. A prominent example is capturing the temporal dynamics of evolving scenes, such biological (Fig. 7) or moving (Fig. 6) specimens, where the tradeoff between the frame rate and resolution is acutely felt. Scan pattern adaptation to match scene dynamics promises to capture faster phenomena while maintaining the resolution necessary to resolve small structures, such as the aphid antennae in Fig. 7. Another example is an image-guided procedure, such as the intrasurgical model system (Fig. 8), in which acquired frames serve as the primary source of feedback for both human and artificial agents. By boosting the frame rate, the feedback latency and motion artifacts are reduced, thereby promoting stability for closed-loop instrument control.

A particular strength of the probabilistic adaptive scanning approach is its scene independence. Despite no prior knowledge of and large inter-experiment variation in the scene structure, the algorithm effectively tracked and revisited the identified targets in subsequent frames. We observed this both in computational and physical imaging, which shared all parameters, respectively, except for change weights (${\lambda _w}$) for physical experiments due to the differences in target optical properties between the model systems. While more sophisticated techniques for structure extraction and scene reconstruction may yield better performance in selected imaging tasks, the generality and simplicity of this current approach present it as a general-purpose, computationally efficient algorithm to expedite scanning for appropriately selected scenes. Indeed, updated selection, scene reconstruction, and priority propagation for our physical experiments were implemented in pure Python, yet nonetheless achieved real-time performance.

Notably, the limiting factor for adaptive scanning as implemented here was the scanner dynamics. The acceleration limit ${a_{{\rm max}}}$ in particular greatly influenced the galvanometer turnaround time, thereby the inactive segment length ${R_{{\rm off}}}$. Such was this effect that scenes requiring $\alpha \mathbin{\lower.3ex\hbox{$\buildrel \gt \over{\smash{\scriptstyle\approx}\vphantom{_2}}$}} 0.15$ could not reliably achieve $\gamma \geq 5$ since larger $\alpha$ tends to require more inactive segments. A key determinant of which imaging tasks are well-suited for adaptive scanning is therefore the anticipated scan fraction $\alpha$. Densely populated or “busy” scenes with a predominance of dynamic scene positions ($\alpha \uparrow$) will experience smaller speedups than sparsely populated scenes ($\alpha \downarrow$). For the physical imaging experiments shown here, we consequently kept scan fractions low. Nevertheless, as the multitarget computational imaging experiment demonstrated (Fig. 5, blue), “busy” scenes can still benefit from adaptive scanning, albeit with a more modest speedup. Adaptive scanning will not yield worse performance than uniform scanning because it reduces to the latter as $\alpha \to 1$, except in pathological cases.

Although adaptive scanning escapes the frame rate–resolution limit, it remains bounded by scanner dynamics. This is particularly relevant with increasingly available MHz sources that permit higher frame rate–resolution products in Eq. (2). For a fixed imaging task (i.e., unchanged scene morphology and resolution), the gains of adaptive scanning diminish with faster sources unless advances in scanner technology relax dynamics limits. For example, if using a 1 MHz source instead of our 200 kHz one, the speedup factors demonstrated here would require a $5 \times$ decrease in scene dimension or an increase in scene resolution to achieve. Such scaling of the imaging task preserves the active segment velocity and thereby avoids prolonging the already minimally short inactive segments. For a given scanner hardware, a faster source thus translates better to a higher resolution than to a faster frame rate in the adaptive scanning sense; however, a user is still able to trade off between the two.

From a detection parallelism standpoint, point-scan imaging as employed here is both the most impoverished and most flexible of the acquisition techniques. Line- and area-scan systems sample thousands or millions of scene positions simultaneously and, in doing so, avoid many of the challenges of point-scan imaging. It comes, however, at the expense of reconfigurability due to the fixed spatial arrangement of elements within the detector arrays. Adaptive scanning is thus particularly well-suited to point-scanning, where the need for performance enhancement is great and the ability to scan arbitrarily is present. Nevertheless, a role for adaptive scanning exists for line- and area-scan systems when the desired field of view exceeds the acquisition field. In such cases, the probabilistic paradigm shown here would provide a structured method for scene-reactive montaging.

A limitation of our analysis is the absence of quantitative image quality comparisons between uniform and adaptive scanning for physical experiments. Each of our model systems evolved with time (e.g., moving aphids) since a key application of adaptive scanning is a reduction in the frame interval to better capture scene dynamics. Combined with the large difference in rolling shutter intervals between scanning modes, these scene dynamics prevented the frames acquired via uniform and adaptive scanning from exhibiting a pixel-level correspondence that would have facilitated a quantitative comparison. For this reason, we quantitatively assessed the computational experiments (Fig. 5), where such a correspondence existed, and qualitatively evaluated the physical experiments through side-by-side comparisons (Visualization 3, Visualization 4, and Visualization 5).

8. CONCLUSION

For properly selected scenes, adaptive scanning changes the familiar trade-offs between the frame rate and resolution for scanned imaging modalities. By improving the scan efficiency for sparse scenes, adaptive scanning permits an increased frame rate, better resolution, or a combination of the two without hardware changes. The time-optimal trajectory generation approach to designing scan patterns is a key enabler of this capability. Results from adaptive scanning in computational and physical experiments indicate its utility in enhancing the scan efficiency for common inanimate, biological, and medical imaging applications. In particular, we demonstrate the effective of performance of a 1 MHz swept-source OCT system while using only a 200 kHz source. Adaptive scanning is a general-purpose, software-only approach to enhance the performance of scanned imaging systems.