Open Access
Add to BibSonomyAbstract
We introduce an optical time-of-flight image sensor taking advantage of a MEMS-based laser scanning device. Unlike previous approaches, our concept benefits from the high timing resolution and the digital signal flexibility of single-photon pixels in CMOS to allow for a nearly ideal cooperation between the image sensor and the scanning device. This technique enables a high signal-to-background light ratio to be obtained, while simultaneously relaxing the constraint on size of the MEMS mirror. These conditions are critical for devising practical and low-cost depth sensors intended to operate in uncontrolled environments, such as outdoors. A proof-of-concept prototype capable of operating in real-time was implemented. This paper focuses on the design and characterization of a 256x64-pixel image sensor, which also comprises an event-driven readout circuit, an array of 64 row-level high-throughput time-to-digital converters, and a 16Gbit/s global readout circuit. Quantitative evaluation of the sensor under 2klux of background light revealed a repeatability error of 13.5cm throughout the distance range of 20 meters.
© Optical Society of America
1. Introduction
Consumer electronics applications have recently created a great need for low-cost three-dimensional (3D) image sensors [1,2] and the sensor community is responding with a high number of compelling technologies. Applications relying on 3D-enhanced computer vision are consequently rapidly emerging. Interactive gaming applications have led the trend by taking advantage of less demanding specifications, in particular with respect to the rejection to external and uncontrolled background light – a major challenge to most active near-infrared (NIR) depth sensors. In this context, a fast-growing class of light detection and ranging (LIDAR) technologies are the so-called time-of-flight (TOF) 3D image sensors [3–14]. In these systems, arrays of light emitting diodes (LEDs), or laser diodes (LDs), are utilized to diffusely illuminate targets within a field-of-view (FOV) that preferably matches the FOV of the typically employed standard imaging lens. Owing to their simple electro-optical design and the utilization of conventional integrated circuit technologies, such as CMOS, these techniques commonly benefit from the obvious potential for low-cost and compact implementations. In these systems, however, the pixels need to cope with extremely low optical signal intensity due to the diffused light approach. While they operate reasonably well over a range of several meters in controlled environments, such as indoors, their performance tends to rapidly deteriorate in severe background light conditions. In these conditions, higher optical signal to background ratio (SBR), and consequently better performance, is typically achieved by laser scanner approaches, which take advantage of collimated laser beams. Not until recently, scanner laser systems exhibited rather limited vertical resolutions and their use has been mostly relegated to scientific, military and some specific – typically low-volume – applications. With the introduction of commercially available and relatively high vertical resolution LIDAR [15], 3D-enhanced computer vision seems to have broken through the indoor barrier into fully uncontrolled environments. The most actively researched applications for that technology have been in the automotive field and, in particular, fully autonomous driving [16] and the detection of vulnerable road users such as pedestrian and cyclists [17]. The high-resolution LIDAR of [15] thus serves a very important purpose of providing the required level of performance to enable the research to progress in these applications. However, its construction, based on an array of 64 discrete 905nm LDs and 64 carefully aligned photodetectors, makes its cost prohibitive for mass production. Our research group has recently introduced a scanning LIDAR sensor achieving comparable resolution and performance [18], yet utilizing a single LD coupled with a CMOS photodetector. Accurate range imaging was obtained outdoors with a resolution of 340x96 pixels at 10 frames per second (fps) within a range of 100 meters. This performance was partly enabled by taking advantage of arrays of avalanche photodiodes (APD) manufactured in a readily available CMOS technology. Biased above their characteristic breakdown voltage and therefore operating in the so-called Geiger mode, APDs become digital pixels [6], thus enabling direct and virtually noise-free signal processing. These devices, commonly referred to as single-photon avalanche diodes (SPADs), output a train of digital pulses upon the detection of individual photonic events with a timing resolution on the order of one hundred picoseconds [19]. The ability to digitally perform a combined spatial and temporal correlation of photons within a pixel contributed significantly to the results achieved in [18]. While continued optimization of that polygonal mirror-based approach is underway in our group, in particular for applications requiring long range and higher immunity to background light, the approach has limitations related to system size and cost, thus potentially narrowing its applicability.
In this work, we propose a new concept of imaging LIDAR. In particular, in order to allow for significant gains in system size and ultimately cost, we proposed a new micro-electromechanical system (MEMS) scanning concept. The design flexibility and resilience against electronic noise of SPAD signals played a key role towards the newly proposed sensor architecture.
Another MEMS-based scanned LIDAR system capable of 256x128 pixels has been recently introduced [20]. Based on a commercially available MEMS mirror, the sensor consists of fully independent transmitting and receiving optical paths. On the receiving end, a single InGaAs PIN photodiode allows for the detection of back-reflected laser pulses. Lateral image resolution is therefore obtained sequentially as the laser beam scans the scene of interest both in the horizontal and vertical directions. In that concept, the MEMS mirror is not involved in the receiver path and, hence, a MEMS mirror small enough just to accommodate the collimated transmitting beam may be effectively employed. A careful analysis of the concept, however, reveals an important limitation with respect to SBR. Ideally, the receiver optics should have a very narrow FOV so as to image only the point – or area – currently being illuminated by the scanner. Only that point benefits from higher optical SBR. In [20], since the receiver optics has a fixed and wide FOV, background light reflected from unilluminated areas on the target are also received, thus practically revoking the advantages of scanning over diffused light approaches.
Yet another type of MEMS scanned LIDAR concept was introduced in [21,22], whereby the sensor benefits from a quasi-coaxial transmitting and receiving optics design. In coaxial LIDAR sensors, transmitting and receiving paths share the same optical axis. Back-reflected optical signals are thus imaged onto the optical receiver through the same scanning mirror. The receiver optics may therefore have the desired FOV to match the laser spot size on the target surface, thus leading to an advantageous SBR [18]. Unlike the previous situation, however, large area MEMS mirrors are necessary to obtain sufficiently large aperture. Indeed, TOF ranging performance is also limited by the amount of back-reflected light that is collected in the receiver channel, thus calling for large aperture designs. In [21,22], a monolithic array of synchronized MEMS mirrors is proposed to enable one-dimensional and relatively large-aperture MEMS scanners. Provided that all the mirrors are accurately synchronized, the ensemble fulfils the same function of a single larger mirror. The authors also suggested that one synchronized MEMS mirror may be exclusively used for the deflection of the laser beam on the transmitting path, thus leading to a quasi-coaxial design. Although the complete LIDAR sensor has yet to be reported, this approach theoretically addresses the main limitation of [20]. Matching properties between individual mirrors, especially with respect to their resonance frequency, as well as the large scale manufacturability of such large MEMS mirrors are still open issues too.
In this work, we propose a new concept of MEMS-scanned LIDAR that benefits from a high optical SBR capability, yet employing a small area MEMS mirror. In our approach, the design of the transmitting and receiving channels are fully independent. Similarly to [20], in our system, the MEMS mirror may be small enough just to accommodate collimated laser beams. Unlike [20], however, we take advantage of a 256x64 SPAD-based image sensor whose pixels are actively gated in synchronization with the laser scanning pattern. Owing to this synchronized gating approach, each pixel on the image sensor does not integrate background light when its point imaged on the target surface is not illuminated by the laser beam. Likewise, owing to the utilization of a standard imaging lens, background light reflected on the remaining areas on the target surface do not interfere with the illuminated pixel, to the extent that lens non-idealities may be neglected. Furthermore, since the imaging lens may be chosen independently, large aperture, or equivalently low f-number, small size lenses may be effectively employed. These benefits are enabled by the introduction of a novel image sensor architecture based on an event-driven resource sharing principle, which, in turn, is almost exclusively made possible by SPAD-based pixels. We believe that the concept proposed here presents a good trade-off between high-performance scanning LIDARs and their diffused light counterparts in terms of performance, compactness and cost.
While the overall system design issues and detailed ranging performance are reported in a companion paper [23], this paper focuses on the design aspects and chip characterization of the proposed CMOS SPAD-based image sensor. The paper is organized as follows. In the next section, we introduce our depth sensor architecture; in section 3, we comprehensively describe the architecture and design of the single-photon imager. Sensor characterization and measurement results are reported and discussed in section 4. The paper is then concluded in section 5.
2. Depth sensor architecture
Figure 1 illustrates the overall imaging rangefinder architecture. The transmitting optical system consists of three NIR pulsed LDs, collimating lenses, and a two-axis MEMS scanner. In order to minimize the oscillation amplitude of the 8x4mm2 MEMS mirror in the horizontal direction, the FOV was divided in three adjacent scanning regions. The collimated beam from each LD is then directed to its own region for raster scanning. 870nm LDs emitting pulses of 4ns at a repetition rate of 100 kHz, were utilized. Each LD outputs a maximum peak power of 40W, while its average power is approximately 16mW. The collimating optics consists of three individual small lenses and a larger common lens.
The MEMS device and its driving electronics, introduced in [24] and fully redesigned for this work, performs uniform scanning within a FOV of 15 degrees horizontally and 11 degrees vertically, thus leading to a combined FOV of 45 by 11 degrees. Raster scanning is performed along the Y-axis at the resonance frequency of the MEMS mirror, i.e. 1.3 kHz, and then driven along the X-axis in a non-resonant mode at 10 Hz. Actuation of the MEMS mirror is based on a moving magnet configuration. The MEMS driving electronics continuously measures the actual mirror deflection angle in both directions, by means of Hall sensors placed underneath the mirror, and provides a digital feedback signal for the main system controller. On the receiving path, the design complexity is encapsulated into the SPAD-based CMOS image sensor as much as possible so as to reduce system components. A single f/1.0 lens was therefore utilized to image the targets within the FOV onto the image sensor chip. The latter also comprises an array of high-throughput time-to-digital converters (TDCs) in order to measure the arrival time of photons impinging on the photo-sensitive area.
In addition to the MEMS and laser driver boards, the system electronics comprises a sensor interface board, a digital board featuring a low-cost field-programmable gate array (FPGA), and a USB transceiver to send data to and from a standard personal computer. Although most of the digital interface circuits are designed in dedicated digital blocks on the FPGA device, the latter also embeds a soft 32-bit microprocessor that acts as the main system controller.
3. Image sensor design
The architecture of the image sensor was carefully devised so as to mitigate the number of critical specifications in other sensor components. For instance, the maximum measurement error on the actual MEMS mirror deflection angle is a critical specification as our system relies on a virtually perfect pixel gating synchronization. This constraint stems from the fact that an erroneous feedback signal would cause the image sensor to select pixels that are not illuminated by any laser beam, thus resulting in wrong TOF evaluations. Alternatively, an exceedingly strict error specification would call for more complexity at the system level, thus rendering the approach less applicable.
Another constraint that is unrelated, yet also specific to our concept, arises and should be addressed. On the photo-sensitive area of our image sensor, silicon real-estate should as much as possible be devoted to the design of the SPADs so as to maximize the fill factor. Yet, there is a need to measure the time interval between the laser pulse emission and the instant the same pulse is detected inside a pixel. An area-inexpensive and very precise circuit to achieve this function in CMOS is a TDC; however, it still may take significant silicon area when compared to the size of a pixel. It is therefore more effective in terms of fill factor to place these circuits on the periphery of the image sensor. The introduction of an asynchronous readout circuit capable of sharing resources is therefore required. In order to mitigate these constraints, in particular the error constraint on the Hall sensor circuitry, we introduce a hybrid pixel gating pattern simultaneously based on a raster scan and on an event-driven pixel readout mechanism. The underlying idea is to select a slightly broader pixel area around the location where the laser beam is expected to be imaged, and, at the same time, allow peripheral TDC resources to be shared by the selected pixels on an effective event-driven basis. This architecture is feasible provided that pixels featuring gating capability on the nanosecond scale, and capable of outputting large and robust signals on extremely weak incoming optical power, exist. To the best of our knowledge, only SPAD-based pixels currently qualify for these functions in any solid-state technology. An event-driven readout principle for single-photon pixels was introduced in [25], and was also utilized in other sensors, e.g. in [26], where this resource sharing scheme is more effective than a simple raster scan.
Figure 2 shows the architecture of our image sensor based on the proposed MEMS-scanner and pixel synchronization approach. The sensor consists of an array of 256x64 SPAD-based pixels, a flexible column decoder circuit, an event-driven pixel readout circuit, an array of 64 TDCs, a phase-locked loop (PLL), and a sensor readout circuit. Taking into account our MEMS scanning pattern, we have opted for a row-based TDC layout. As the number of TDCs match the number of rows in the pixel array, it is not necessary to disable unused rows synchronously with the scanning pattern as enough resources are available. This choice is also motivated by the much faster vertical scanning speed as well as by the fact that the laser beams have a rather small aspect ratio of approximately 1:12. Synchronous column activation is nonetheless necessary and it is implemented by means of a multi-phase column decoder, explained in section 3.2. As photons are detected in the activated pixels, precise timing trigger signals propagate horizontally to the TDC array, which, in turn, performs time interval to digital conversions with a resolution of 208ps. A third-order PLL was implemented on-chip to generate precise timing reference signals for the TDC array, thus keeping process, voltage, and temperature (PVT) timing variations down to a minimum. The 12-bit TOF data from the TDCs, together with an 8-bit pixel coordinate stamp indicating the pixel from which the TOF evaluation originates, is synchronously read out off-chip by means of a pipelined readout circuit. The latter is based on eight channels that operate as time-multiplexed readout circuits, each one outputting the data of eight TDCs. In order to cope with the data rate, the readout channels operate at a frequency eight times faster than the throughput of an individual TDC.
Fig. 2 SPAD-based image sensor architecture. The image sensor is capable of achieving a throughput of 16Gbit/s, or equivalently, 800 million TOF evaluations per second.
3.1 Single-photon pixel
Individual photonic events are detected inside a pixel by means of a p+/p-well/deep n-well SPAD whose cross-section is shown in Fig. 3 . The device, a fill-factor-optimized version of the SPAD presented in [27], measures 10.5µm in diameter, which leads to 13.9% of fill factor. Its peak photon detection efficiency is 20% at 470nm. At the central wavelength of the LD, i.e. 870nm, it is 3%. Premature edge breakdown is prevented by means of p-well guard-rings. The overall pixel circuit, depicted in Fig. 4 , performs passive quenching and recharge, typical functions required in any SPAD front-end circuit, and implements an output buffer.
Fig. 3 Cross-sectional view of the p+/p-well/deep n-well SPAD. The left half-section shows the device along a vertical imaginary cut, whereas the right half-section shows it along a horizontal one. The avalanching p+/deep n-well junction has nonetheless a circular shape.
Fig. 4 In-pixel front-end and readout circuit. The 7-transistor circuit performs passive quenching and recharge functions. M7 signals a photon detection by pulling the row output line down for the duration of the SPAD recharge pulse.
Upon photon detection, the latter drives the output line down, thus generating an accurate trigger signal for the TDC on the same row. All the transistors in the pixel are thick-oxide, except for the output buffer M7, which requires higher driving strength and is fully isolated from voltage swings larger than the 1.8V power supply voltage. The cathode terminal VAPD of all the SPADs in the pixel array is biased at a constant voltage, typically at 25.6V, which is 3.3V of excess bias (VE) above the SPAD’s breakdown voltage at room temperature. Since the cathode terminal remains at a constant voltage, the parasitic deep n-well/p-substrate junction does not contribute to any increase in the parasitic capacitance of the SPAD [27]. The anode terminal VS, however, is rapidly raised by approximately VE upon photon detection as the avalanche current is much larger than the maximum current set by transistors M1 and M2. These transistors, biased as a cascode current source, achieve the quenching and recharge functions. Column selection is performed by transistors M2 and M3 via the column-common signal VSEL. When VSEL is set to zero logic state, the SPAD is unable to recharge, and, simultaneously, the gate of M7 is driven to ground, thus fully disabling the pixels in the corresponding column. When the pixel is selected, transistors M3 and M4 together with the resistor R1 form a gated NMOS inverter, which is then followed by a CMOS inverter formed by M5 and M6. These inverters reshape the triangular digital pulse generated on the node VS, see [27], into a rectangular digital pulse that drives the output buffer. The utilization of a NMOS inverter is justified by miniaturization constraints. PMOS transistors require a relatively large clearance distance with respect to the SPAD’s deep n-well layer, so we have minimized their utilization inside the pixel to just one device. While it would have been possible to replace M6 by another resistor, resulting in a potential reduction of pixel size [28], that modification would make the pixel draw quiescent current in our implementation, even when the pixel is inactive. We believe our solution is a good trade-off between area and power consumption since no static current is consumed, should the pixel be selected or not.
3.2 Programmable column decoder and event-driven pixel readout
In order to effectively share TDC resources on a row basis and yet allow for some tolerance on the actual column position of the laser beams, the multi-phase column decoder and the event-driven readout were designed to operate together. According to our estimates, a maximum of eight adjacent columns is wide enough to accommodate the uncertainty on the laser beam position as well as its worst case divergence. Furthermore, in our setup, shown in Fig. 1, three laser beams are simultaneously directed to the scene. However, in order to optimally share the same TDC on a given row, the lasers are pulsed at different times. Since the repetition rate of the lasers is 100 kHz, i.e. with a period of 10µs, each LD is pulsed with a delay of 3.33µs with respect to the other two. This delay gives enough time so the longest TOF of interest corresponding to any given LD does not ambiguously enter the time slot of the remaining ones. As shown in Fig. 2, the column decoder consists of four master-slave shift registers and a 4:1 multiplexer. Each master-slave shift register is then programmed with the desired column selection pattern associated with one LD. Once all the shift registers are programmed, the 4:1 multiplexer allows for fast switching between them in synchronization with the laser drivers. Figure 5 illustrates the timing diagram of the LDs and their relation with the column activation pattern. Since our optical setup requires only three laser beams, the fourth master-slave shift register is optionally used to actively disable all the columns. Each master-slave shift register has 256 bits for column selection, and additional 8 bits that can be programmed with a unique code stamp identifying the selection state of that particular shift-register. This information is also synchronously readout off-chip at a later stage, as explain in section 3.5. As the MEMS mirror deflects the laser beams horizontally, it is necessary to continually reprogram the contents of the shift registers. The new state of each slave shift register chain is serially programmed during operation without affecting the state of the master one, thus keeping the selection state of the columns intact during this time. It takes 264 clock cycles of 10ns, i.e. 2.64µs, to reprogram all the four slave shift registers, and a single clock cycle to transfer their state into the master shift registers. The column decoder ensures that, at any given time, there is a maximum of eight contiguous columns on a single row that may utilize the TDC circuit. Based on this assumption, the row event-driven readout of [25] was significantly simplified in this work. Figure 6 depicts the readout circuit of a single pixel row, whereby only the first nine pixels are illustrated. In order to convey asynchronous time interval, i.e. TOF, requests to the TDC, each row utilizes eight column address lines. These lines are pulled up by properly biased PMOS transistors at the end of the pixel row, shown as resistors on the left-hand side of Fig. 6. As can be seen in the figure, the lines are connected to the output buffer of pixels following a regular pattern. The first line is connected to the 1st, 9th, 17th, 25th pixel, and so on. The second line is then connected based on the same pattern, i.e. connected to the 2nd, 10th, 18th, 26th pixel, and so on. Since there are 256 pixels on a row, each address line is connected to 32 pixels. Despite the multiple connections on a single line, since a maximum of eight contiguous pixels are activated at any point in time, it is ensured that each line is connected to a single activated pixel.
Fig. 5 Timing diagram of the LDs and illustration of the column activation pattern. Since each LD emits a laser pulse with a delay of 3.33µs with respect to the others, each TDC may be effectively shared on a row basis. The maximum theoretical TOF is limited by the TDC range of 853ns, or equivalently 128 meters.
Fig. 6 Simplified event-driven readout of a single pixel row. Column decoder lines are not drawn in the picture.
Upon photon detection in one of the eight activated columns, the pixel pulls the corresponding line down by means of its output buffer, which, in turn, generates an inverted pulse along the row. On the right-hand end of the row, the eight lines are inverted, then fed onto an 8-input OR gate. At the output of the OR gate, i.e. TRG, digital pulses whose leading edge is accurately related to the photon arrival times are generated. This signal is utilized as a trigger for the time-to-digital conversion, as well as a latching signal to store the state of the address lines by means of eight flip-flops, as shown in Fig. 6. The delay through the OR gate ensures that the setup time of the flip-flops is not violated. Once the state of the eight lines is secured by the flip-flops, the row enters a dead time period of approximately 40ns, equivalent to the dead time of the triggered SPAD. Subsequent photon detections on the remaining pixels in the same row are simply missed during this dead time, not affecting therefore the previous measurement. It is however possible that photons impinging on two or more pixels trigger them simultaneously. In such situation, the state of the address lines will indicate that two or more pixels were involved in that TOF event. Consequently, the TOF computed by the TDC may be assigned to more than a single pixel. If, on the other hand, two pixel detections are delayed by more than the margin delay of the flip-flop setup time, approximately 200ps in our design, only the leading photon detection is taken into account. At the end of the pixel dead time, the address lines are pulled up again, thus enabling subsequent TOF detections.
3.3 Time-to-digital converter array
Figure 7 shows the flash-conversion TDC array architecture. At the top of the TDC array, fully-differential reference signals, namely ϕ0, ϕ1, ϕ2, and ϕ3, are generated by an 8-phase 600-MHz PLL. These 600MHz signals, illustrated in Fig. 8 , are uniformly delayed by 208ps and are shared vertically by all the TDCs.
Fig. 8 Example of waveforms of some TDC and PLL signals. In this example, two TOF evaluations are illustrated. Prior to the first TOF evaluation, the START signal is asserted once, thereby initiating the state of S[11:0]. Note that, when sampled and despite potential setup time violations, the phase state {ϕi}i≤3 always settles in the interval [0,7].
A coarse timing reference of 1.667ns for time-to-digital conversion is achieved by two 9-bit counters placed near the PLL that are synchronously incremented by ϕ0 and its inverted replica, respectively. The output of these counters, i.e. C0[8:0] and C1[8:0], are also shared by the remaining TDCs. A special TDC is utilized to register START signal assertions. Its function is to sample the state of {ϕi}i≤3, C0[8:0], and C1[8:0] and to encode this timing information onto a 12-bit digital signal, i.e. S[11:0], which, in turn, is shared by the remaining 64 STOP TDCs. The encoding logic is implemented as follows: The state of {ϕi}i≤3 is encoded into the 3 least significant bits of S, i.e. S[2:0], according to the sampled phase delay with respect to ϕ0 within a single clock period. The most significant bits of S, i.e. S[11:3], originate from either C0 or C1, depending on the 3-bit encoded sampled phase. As can be seen in Fig. 8, it is possible that the sampled value of C0 or C1 be corrupted if the sampling occurs during the state transition of either counter. However, when the state of one of the two counters is in transition, the other counter state is safely stable. As a result, depending on the decoded value of S[2:0], either C0 or C1 is safely utilized as S[11:3]. Upon assertion of its trigger signal, each stop TDC synchronously registers the state of {ϕi}i≤3, C0[8:0], C1[8:0], and S[11:0]. According to Fig. 7, the time difference between the STOP and START is finally computed by first generating the intermediate 12-bit value S’, based on {ϕi}i≤3, C0, and C1, in a similar way to which S is generated in the start TDC, followed by a subtraction operation between S’ and the sampled S value. The signal VLD is asserted so as to schedule the readout of the valid TOF result into a synchronization FIFO. Figure 8 illustrates the internal waveforms for a TDC involved in two hypothetical time-to-digital conversions. As can be seen in the Fig. 8, several STOP assertions may occur on a single TDC between two consecutive START assertions. Furthermore, although the proposed architecture is capable of performing flash conversion, the TDC undergoes a dead time cycle of approximately 40ns due to the pixel event-driven readout mechanism.
3.4 Phase-locked loop
The PLL, shown in Fig. 9 , is a type-II based on a commonly used zero dead-zone phase-frequency detector and charge pump. Based on an external reference signal of 75MHz and a fully-differential voltage controlled oscillator (VCO), the PLL synthesizes the four uniformly distributed 8-phase signals at 600MHz. The 75MHz reference signal is provided by a crystal oscillator that is assumed to be uncorrelated to the laser repetition rate, thus preventing systematic interference between the start TDC and the PLL’s VCO. A passive second-order filter is placed off-chip near the PCB-bonded image sensor.
Fig. 9 Schematic of the third-order PLL. A passive second-order loop-filter LP(ω) is placed off-chip.
3.5 Sensor readout circuit
During the image acquisition time, the image sensor constantly outputs TDC data together with binary words that identify the pixel that originated each TOF evaluation, independently of the current column decoder selection state. The pipelined readout circuit is operated synchronously with a clock frequency of 100MHz, thus achieving a maximum throughput of 16Gbit/s, or equivalently, 800Msamples/s. As can be seen in Fig. 2, the 64 TDCs are combined in eight time-multiplexed readout groups of eight TDCs each. A 3-bit signal, IDX[2:0], synchronously indicates the currently selected TDC in each group, thus allowing the companion FPGA device to identify the TDC, or equivalently the pixel row, the data should be assigned to. A readout word associated with each TDC comprises 21 bits of data, i.e. the 12-bit TOF word, the 8-bit COL binary pattern registered by the event-driven readout according to section 3.2, and a valid bit. The latter is also used by the readout circuit to prevent chip output pads from toggling when no valid data are available for that particular clock cycle. Since the COL pattern only indicates the pixel involved in the TOF evaluation from among the eight selected columns, the coordinates of this contiguously selected column block is read out on a separate bus, namely COLADD[7:0], see Fig. 2. Furthermore, since the readout circuit takes some clock cycles from the moment a photon is detected until its TOF data word is read out off-chip, it is important to register the column decoder coordinate data word and to delay its readout by the same number of cycles. This is achieved by pipelining the COLADD bus, thus ensuring that the companion FPGA is presented with a valid COLADD word that matches the currently read out TOF data.
4. Experimental results
A photomicrograph of the single-photon image sensor is shown in Fig. 10(a) . The chip was implemented in a high-voltage 0.18µm CMOS technology. A full sensor characterization was performed and the sensor chip was confirmed to be fully functional at its nominal clock frequencies. As can be seen in Fig. 10(a), the effective active area of the image sensor occupies approximately 1/3 of the total chip area, while the PLL and TDC array only occupies a relatively small area. The sensor chip measures 8x5 mm2, mostly due to the 256-pad ring limited design. While the chip could be readily mounted on a standard chip package, we have opted for bonding the chip directly onto a printed circuit board.
Fig. 10 (a) Photomicrograph of the CMOS single-photon image sensor. The chip measures 8x5mm2. (b) Experimental setup of the overall LIDAR system.
The overall LIDAR system, which at the current form factor still benefits from a large margin for miniaturization, is depicted in Fig. 10(b). In Fig. 10(b), the triple-LD driver and its collimating optics can be seen with illustrations on how the laser beams are deflected by the MEMS mirror. A standard imaging lens coupled with a NIR optical filter was utilized. The system is capable of acquiring full 256x64-pixel images at 10 frames/s.
4.1Dark count rate
As is commonly the case with SPAD-based image sensors [6], the proposed imager benefits from digital pixels whose signals are virtually free from readout noise. As a result, with the exception of photon-shot noise, the imaging noise performance is typically dominated by intrinsic SPAD noise, which, in turn, is mainly characterized in terms of the dark count rate (DCR). Thermally or tunneling generated carriers within the p-n junction, which typically generate dark current in linear mode photodiodes, cause unwanted trigger pulses in SPADs that may not be directly discerned from regular photon triggered events. The rate of these spurious Geiger events, i.e. DCR, was measured by acquiring an image in dark conditions for a fixed acquisition time. Figure 11 shows the DCR image of a sample chip sensor at 30°C with an excess bias (VE) of 3.3V above the SPAD breakdown voltage of 22.3V. As can be seen in the figure, a low number of pixels present very high levels of DCR, which typically appear in the intensity image as “salt and pepper” noise, without the “pepper” component. DCR impact on the image may be conveniently understood by sorting the DCR values of all the pixels and plotting them as a function of accumulated pixel count, or alternatively as shown in Fig. 12 , as the percentile of imaging pixels.
Fig. 12 DCR distribution at 30°C. The measured DCR from all 16384 pixels were sorted and plotted as a function of pixel percentile. On right-hand side, the same plot is shown with only the 2% noisiest pixels.
As can be verified in Fig. 12, the median DCR at room temperature was 750 counts/s. On the right-hand side of the figure, the noisiest two percentiles of the DCR are shown. A relatively well defined curve “knee” can be seen approximately at 99%, at which the DCR is 33.2kcount/s. In our application, our sensor needs to cope with considerable high levels of background light intensities, thus rendering DCR tolerable even for relatively noisy pixels. In applications where the DCR performance is a more critical parameter, the top 1% of pixels could be permanently disregarded at the image readout level. Since DCR is a strong function of the SPAD area, it is convenient to report the DCR density, especially when a comparison to other devices is of interest. In this work, the median DCR density was 8.66 counts/µm2, whereas the average DCR density was 45 counts/µm2, mostly due to a low number of outlier pixels. The median DCR density figure is slightly worse than the 6 counts/µm2 figure recently reported at the same temperature for a SPAD array of comparable size and optimized for low-light applications [29].
4.2 Time-to-digital converter array and overall timing evaluation performance
Timing performance evaluation was carried out after verification of the functionality of the on-chip PLL. The TDC array was carefully characterized with respect to its typical performance figures, namely differential non-linearity (DNL) and integral non-linearity (INL). Since the array of 64 TDCs is utilized in an imaging configuration, rather than mean performance, the worst-case TDC performance is essential. DNL was characterized at room temperature for all the TDCs simultaneously. While a single column of pixels was selected by the column decoder, the remaining chip functions were activated at their nominal clock frequencies during the characterization. In particular, the sensor readout circuit was expected to generate switching noise on-chip, which typically couples to critical reference signals through substrate and/or power supply lines.
In Fig. 13(a) , the DNL measured in picoseconds is plotted for all the 64 TDCs over the full measurable time range. The worst-case DNL was considerably smaller than the TDC resolution of 208ps, thus ensuring monotonic conversion performance. Figure 13(b) shows INL curves measured in picoseconds as a function of measurable time range. The worst-case INL among all the 64 TDCs and for all the TDC codes are also plotted as envelope curves in the figure. As can be seen, the worst-case non-linearity error was −0.56LSB, or equivalently −116ps. When converting this timing deviation to distance, it corresponds to a non-linearity error of 1.75cm, throughout the theoretical maximum distance range of 128 meters.
Fig. 13 (a) Measured DNL of individual TDCs in the array versus measurable time. (b) Measured INL of individual TDCs as well as envelope curves of the maximum deviations. Note that the worst INL among the 64 TDCs was −0.56LSB (−116ps).
In order to measure the timing resolution, also commonly referred to as the instrument response function (IRF), of the overall detection circuit including the SPAD and TDC contributions, an external 80ps full width at half maximum (FWHM) laser source was utilized. Due to the unavailability of picosecond laser sources at 870nm in our laboratory, the image sensor was directly exposed to a 635nm laser light at a repetition rate of 40MHz. The laser beam intensity was attenuated by means of neutral-density filters and made spatially uniform by means of a diffuser. Since the laser repetition rate was 40MHz, in order to evaluate the timing performance over the whole TDC range, the digital trigger output signal of the laser source was divided by 34 on the FPGA device, and then connected to the START signal of the TDC array. Based on this setup, data from all the 64 TDCs were acquired simultaneously for approximately one second. On the FPGA, 64 histograms were built as a function of the measured TOF. These histograms are plotted in Fig. 14 after normalization. As expected, the histograms comprise 34 peaks, corresponding to the 34 laser pulses contained in the time interval between two consecutive START trigger events. Note that 64 histograms are superimposed in the figure.
Fig. 14 Superimposed plot of 64 normalized histograms on photon arrival times. The histograms were acquired while exposing a complete pixel column to a 40ps pulsed laser source repeated at 40MHz. The laser trigger signal was divided by 34 inside a FPGA device before being connected to the TDC start signal.
In order to measure the actual timing IRF, a detailed view of the histogram peak plotted on a linear scale is shown in Fig. 15 , while the same data are plotted on a semi-log scale in the inset. The mean timing IRF among these curves was 519ps FWHM. The obtained timing IRF was considerably larger than the typical IRF of a SPAD, on the order of 100ps [6,10,19,26,29]. Many factors are believed to have contributed to the overall IRF, major contributors being the jitter sensitivity of the TDCs to power supply noise as well as crosstalk between channels. Note that the IRF was measured under strong – nonetheless realistic – photon detection activity. Although the IRF measurement was carried out with a 635nm laser, as opposed to the 870nm LDs utilized in our LIDAR setup, we do not expect a significant impact on the IRF figure. This assumption, which was valid for a similar SPAD in CMOS [30], is also supported by the fact that our IRF figure is dominated by circuit jitter, which does not depend on the signal wavelength. In addition, since the measured IRF is significantly shorter than the laser pulse duration of 4ns utilized in our LIDAR setup, the overall IRF is expected to have a negligible impact on LIDAR performance.
Fig. 15 Detailed view of the detected pulses in the set of 64 histograms. The average IRF at FWHM was 519ps. Inset: semi-log plot of the same data showing the slightly asymmetric IRF.
Furthermore, since the data were acquired for all the TDCs simultaneously, Fig. 15 already reveals the timing delays between the TDC channels, which are considerably shorter than the IRF. In the inset of Fig. 15, the typical asymmetric IRF of SPADs is also revealed, albeit considerably skewed by a large Gaussian jitter component.
4.3 Ranging performance and time-of-flight imaging
Quantitative evaluation of the proposed LIDAR sensor was carried out under 2klux of natural background light within a distance range of 20 meters. A non-cooperative target was built by wrapping a wide board in a white cotton fabric. It was placed in front of the sensor and moved from 2 meters to 20 meters in steps of 1 meter. The actual target position was controlled with a tape measure and the ambient light condition on the target surface was measured with a digital luxmeter. At each distance, a set of 100 depth image samples were acquired at 10fps.
The depth images were computed in real-time on the FPGA by performing histogram processing for each pixel. In Fig. 16(a) , the mean value of 100 distance samples from one pixel in the center of the target is plotted as a function of the actual distance. Based on the same samples, the mean error with respect to the ground truth and the standard deviation were computed. These figures, which correspond to the non-linearity error and the 1σ repeatability error of our depth image sensor, respectively, are plotted in Fig. 16(b). The worst-case repeatability error was 13.5cm throughout the range of 20 meters, thus leading to a relative precision of 0.68%. In the same range, the non-linearity error was better than 10cm.
Fig. 16 (a) Mean measured distance under 2klux of background light as a function of the actual distance for a pixel on the center of the non-cooperative target. (b) Non-linearity error (square marker, blue curve) and standard deviation (round marker, red curve) as a function of the actual distance.
Table 1 shows a performance comparison of this work to the recently published state-of-the-art in CMOS technology. With the exception of this work, the repeatability errors of the remaining sensors were characterized under low or undisclosed ambient light conditions. To the best of our knowledge, our sensor compares favorably to the state-of-the-art in terms of relative precision and range, in particular when taking into account the number of pixels, signal power, FOV, and frame-rate.
Table 1. Comparison of Performance to the Recently Published State-of-the-art in CMOS
The overall functionality of the proposed LIDAR system was also qualitatively evaluated by acquiring depth images of common targets. An example of such images is shown in Fig. 17 . Distance information is color-coded with near objects appearing in red while far objects are shown in blue. In the figure, the imaged person was standing at approximately 3 meters in front of the sensor, while the back of the room was situated at approximately 7 meters. Note that pixels that exhibit high DCR were fully functional in the TOF evaluation. These noisier pixels had a rather negligible impact on the resulting depth image. Column fixed-pattern noise in the ranging image, however, may be noticeable in Fig. 17. This effect occurred due to the simplified implementation of the interface circuit on the companion FPGA. Rather than horizontally scanning the eight activated columns in sync with the MEMS mirror deflection in steps of one column, see section 3.2, the selection window was moved in steps of eight columns at a time. However, since the same effect cannot be perceived when the sensor operates as a conventional image sensor, i.e. when it is not exposed to the LD signal, we believe the column fixed-pattern noise may be mitigated when the column selection window is scanned in smaller steps [23].
Fig. 17 Illustration of depth data acquired using the proposed sensor. The measured distance in meters is color coded, red being close to the sensor and blue being farther from it.
5. Conclusion
We proposed a new concept for MEMS-scanned LIDAR that benefits from a high optical signal-to-background ratio capability while employing a small area MEMS mirror. Our system concept takes advantage of an image sensor whose pixels are actively gated in synchronization with the laser scanning pattern. Owing to this synchronized gating approach, each pixel of the image sensor does not integrate background light when its corresponding image point on the target surface is not illuminated by the laser beam. Furthermore, since the imaging lens may be chosen independently for the receiving optics, large aperture and small size lenses may be effectively employed. These benefits are enabled by the introduction of a novel image sensor architecture based on an event-driven resource sharing principle, which, in turn, is nearly exclusively made possible by SPAD-based pixels.
A proof-of-concept prototype capable of operating in real-time was implemented. At the core of the system, a single-photon image sensor comprising 256x64 pixels was designed and characterized with respect to its performance metrics. The image sensor comprises fast column synchronization decoders, an event-driven pixel readout circuit, an array of 64 row-level high-throughput time-to-digital converters, and a 16Gbit/s global readout circuit. The image sensor was fabricated in a high-voltage 0.18µm CMOS technology and was confirmed to be fully functional at its nominal clock frequencies. The median dark count rate, i.e. the dominant intrinsic noise source in single-photon imagers, was 750 counts/s for circular SPADs of 10.5µm in diameter. While higher to some extent than the state-of-the-art for low-light imaging, this figure is assumed to be reasonably acceptable in LIDAR systems intended to operate under strong background light conditions. Full characterization of the TDC array was performed with respect to differential and integral non-linearities. The worst-case differential non-linearity was approximately −35ps, much smaller than the TDC resolution of 208ps. The worst-case integral non-linearity at room temperature, taking into account all the 64 TDCs, was −116ps, equivalent to −0.56LSB. This worst-case systematic time deviation corresponds to 1.75cm in distance error, throughout the theoretical maximum range of 128 meters. An 80ps laser source was utilized to measure the timing IRF of the overall detection channels, including the jitter contributions from the SPADs and TDCs. The measured IRF was 519ps under a realistic activity load, mostly due to the jitter sensitivity of the TDCs to power supply noise. This value was considerably shorter than the duration of the laser pulses utilized in the proposed LIDAR setup, i.e. 4ns, thus having a negligible impact on the ranging performance.
Quantitative evaluation of the TOF sensor under 2klux of natural ambient light revealed a repeatability error of 13.5cm throughout the distance range of 20 meters, thus leading to a relative precision of 0.68%. In the same condition, the non-linearity error was better than 10cm. To the best of our knowledge, these results compare favorably to the state-of-the-art, in particular when taking into account specifications that have a direct and strong impact on precision. Examples of these specifications are the number of pixels, signal power, sensor field-of-view, and frame-rate. The proposed setup was also qualitatively evaluated in this work by acquiring depth images in ranges of several meters. The resulting images demonstrated the full functionality of the sensor.
Acknowledgments
The authors would like to thank Mitsutoshi Maeda for helping in the design and assembly of the optical system, Marc Lany and R. S. Popovic of EPFL as well as Toshiki Kindo of Toyota Motor Corporation for their technical support on the design of the SPAD device.
References and links
1. H. K. Lim, “The 2nd wave of the digital consumer revolution: Challenges and opportunities!” in Proceedings of IEEE International Solid-State Circuits Conference, Digest of Technical Papers (Institute of Electrical and Electronics Engineers, New York, 2008), 18–23.
2. J. Dibbel, “Gestural Interfaces: Controlling computers with our bodies,” in 10 Emerging Technologies Technology Review issue May/June 2011 (MIT, Boston, 2011).
3. R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37(3), 390–397 (2001). [CrossRef]
4. S. B. Gokturk, H. Yalcin, and C. Bamji, “A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (2004).
5. T. Oggier, B. Büttgen, F. Lustenberger, R. Becker, B. Rüegg, and A. Hodac, “SwissRanger SR3000 and first experiences based on miniaturized 3D-TOF cameras,” in Proceedings of 1st Range Imaging Research Day, (ETH Zurich, Switzerland, 2005), 97–108.
6. C. Niclass, A. Rochas, P.-A. Besse, and E. Charbon, “Design and characterization of a CMOS 3-D image sensor based on single photon avalanche diodes,” IEEE J. Solid-state Circuits 40(9), 1847–1854 (2005). [CrossRef]
7. T. Ushinaga, I. A. Halin, T. Sawada, S. Kawahito, M. Homma, and Y. Maeda, “A QVGA-size CMOS time-of-flight range image sensor with background light charge draining structure,” Proc. SPIE 6056, 605604, (2006). [CrossRef]
8. G. Yahav, G. J. Iddan, and D. Mandelboum, “3D imaging camera for gaming application,” in Proceedings of IEEE International Conference on Consumer Electronics (Institute of Electrical and Electronics Engineers, New York, 2007), 1–2.
9. T. Ringbeck, “A 3D time of flight camera for object detection,” in Proceedings of the 8th Conf. on Optical 3D Measurement Techniques (ETH, Zurich, Switzerland, 2007).
10. C. Niclass, C. Favi, T. Kluter, F. Monnier, and E. Charbon, “Single-photon synchronous detection,” IEEE J. Solid-state Circuits 44(7), 1977–1989 (2009). [CrossRef]
11. G. Zach, M. Davidovic, and H. Zimmermann, “A 16x16 pixel distance sensor with in-pixel circuitry that tolerates 150klx of ambient light,” IEEE J. Solid-State Circuits 45(7), 1345–1353 (2010). [CrossRef]
12. R. J. Walker, J. A. Richardson, and R. K. Henderson, “A 128x96 pixel event-driven phase-domain ΔΣ-based fully digital 3D camera in 0.13μm CMOS imaging technology,” in Proceedings of IEEE International Solid-State Circuits Conference, Digest of Technical Papers (Institute of Electrical and Electronics Engineers, New York, 2011), 410–412.
13. D. Stoppa, N. Massari, L. Pancheri, M. Malfatti, M. Perenzoni, and L. Gonzo, “A range image sensor based on 10µm lock-in pixels in 0.18µm CMOS imaging technology,” IEEE J. Solid-state Circuits 46(1), 248–258 (2011). [CrossRef]
14. S.-J. Kim, J. D. K. Kim, S.-W. Han, B. Kang, K. Lee, and C.-Y. Kim, “A 640x480 image sensor with unified pixel architecture for 2D/3D imaging in 0.11μm CMOS,” in Proceedings of IEEE Symposium on VLSI Circuits (Institute of Electrical and Electronics Engineers, New York, 2011), 92–93.
15. B. Schwarz, “LIDAR: Mapping the world in 3D,” Nat. Photonics 4(7), 429–430 (2010). [CrossRef]
16. J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards Fully Autonomous Driving: Systems and Algorithms”, in Proceedings of IEEE Intelligent Vehicles Symposium (IV), (Institute of Electrical and Electronics Engineers, New York, 2011), 163–168.
17. K. Kidono, T. Miyasaka, A. Watanabe, T. Naito, and J. Miura, “Pedestrian recognition using high-definition LIDAR” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), (Institute of Electrical and Electronics Engineers, New York, 2011), 405–410.
18. C. Niclass, M. Soga, H. Matsubara, and S. Kato, “A 100m-range 10-frame/s 340x96-pixel time-of-flight depth sensor in 0.18μm CMOS,” in Proceedings of IEEE European Solid-State Circuits Conference (Institute of Electrical and Electronics Engineers, New York, 2011), 107–110.
19. S. Cova, A. Longoni, and A. Andreoni, “Towards picosecond resolution with single photon avalanche diodes,” Rev. Sci. Instrum. 52(3), 408–412 (1981). [CrossRef]
20. B. L. Stann, J. F. Dammann, M. M. Giza, W. B. Lawler, H. M. Nguyen, and L. C. Sadler, “MEMS-scanned ladar sensor for small ground robots,” Proc. SPIE 7684, 76841E (2010). [CrossRef]
21. T. Sandner, M. Wildenhain, T. Klose, H. Schenk, S. Schwarzer, V. Hinkov, H. Höfler, and H. Wölfelschneider, “3D imaging using resonant large-aperture MEMS mirror arrays and laser distance measurement,” Proceedings of IEEE/LEOS Optical MEMS and Nanophotonics (Institute of Electrical and Electronics Engineers, New York, 2008), 78–79.
22. T. Sandner, T. Grasshoff, M. Wildenhain, and H. Schenk, “Synchronized microscanner array for large aperture receiver optics of LIDAR systems,” Proc. SPIE 7594, 75940C (2010). [CrossRef]
23. K. Ito, C. Niclass, I. Aoyagi, H. Matsubara, M. Soga, S. Kato, M. Maeda, and M. Kagami are preparing a manuscript to be called “System design and performance characterization of a MEMS-based laser scanning time-of-flight sensor based on a 256x64-pixel single-photon imager”.
24. I. Aoyagi, K. Shimaoka, S. Kato, W. Makishi, Y. Kawai, S. Tanaka, T. Ono, M. Esashi, and K. Hane, “2-axis MEMS scanner for a laser range finder,” in Proceedings of IEEE International Conference on Optical MEMS and Nanophotonics (Institute of Electrical and Electronics Engineers, New York, 2011), 39–40.
25. C. Niclass, M. Sergio, and E. Charbon, “A CMOS 64x48 single photon avalanche diode array with event-driven readout,” in Proceedings of IEEE European Solid-State Circuits Conference (Institute of Electrical and Electronics Engineers, New York, 2006), 556–559.
26. C. Niclass, C. Favi, T. Kluter, M. Gersbach, and E. Charbon, “A 128x128 single-photon imager with on-chip column level 10b time-to-digital converter array,” IEEE J. Solid-state Circuits 43(12), 2977–2989 (2008). [CrossRef]
27. C. Niclass and M. Soga, “A miniature actively recharged single-photon detector free of afterpulsing effects with 6ns dead time in a 0.18µm CMOS technology,” in Proceedings of IEEE International Electron Devices Meeting (Institute of Electrical and Electronics Engineers, New York, 2010), 14.3.1–14.3.4.
28. L. Pancheri, N. Massari, F. Borguetti, and D. Stoppa, “A 32x32 SPAD pixel array with nanosecond gating and analog readout,” Proceedings of the International Image Sensor Workshop (IISS, 2011), R40.
29. C. Veerappan, J. Richardson, R. Walker, D. U. Li, M. W. Fishburn, D. Stoppa, F. Borghetti, Y. Maruyama, M. Gersbach, R. K. Henderson, C. Bruschini, and E. Charbon, “Characterization of Large-Scale Non-Uniformities in a 20k TDC/SPAD Array Integrated in a 130nm CMOS Process,” in Proceedings of IEEE European Solid-State Device Research Conference (Institute of Electrical and Electronics Engineers, New York, 2011), 331–334.
30. J. A. Richardson, L. A. Grant, and R. K. Henderson, “Low dark count single-photon avalanche diode structure compatible with standard nanometer scale CMOS technology,” IEEE Photon. Technol. Lett. 21(14), 1020–1022 (2009). [CrossRef]
