1. Introduction

The real world is rich in color and irradiance distributions creating a massive data set of extreme contrast irradiance values spread across various optical spectral bands. Smartly mining for a full irradiance range scene image data set with desired color information is a challenge for today’s dominant multi-pixel camera technology [1] where the limited instantaneous dynamic range (e.g., < 80 dB) and spectral sensitivity of the camera can fail to capture the true scene data that may be vital to the application. Industrial machine vision [2], food processing [3], archaeology and art conservation [4] along with biology, military, undersea environmental monitoring, material (e.g., soil) composition inspection, mining and mineralogy, and astronomy, are example imaging scenarios where data sets can exist with extreme optical contrast (e.g., 10⁷:1) bright targets across multiple simultaneous broad spectral bands, e.g., Ultra-Violet (UV) + Visible or UV + Near Infrared (NIR), or Visible + Near IR or visible and Short Wave IR (SWIR). For example, a scene can exist with two simultaneous and spatially adjacent targets, one being a very bright visible-NIR target and the other being a very low brightness visible-NIR target. Under these dual-band extreme contrast conditions, target recovery becomes a challenge for prior-art multispectral imaging instruments that deploy wavelength sensitive optics, e.g., gratings and mechanically or electronically tunable filters combined with traditional Photo-Detector Arrays (PDAs) such as a CCD, CMOS, and Focal Plane Array (FPA) multi-pixel devices [5–10]. Specifically, it is mainly the PDA in the spectral imager that limits its dynamic range performance. This is because a bright target will saturate the pixel in the multi-pixel detector as each pixel in the PDA has a limited quantum well capacity. In addition due to spillover effects, the bright target creates crosstalk noise in the neighboring pixels of the PDA. As an example, one commercial multispectral imager offers a 60 dB dynamic range which is inadequate for extreme contrast spectral imaging [11]. To counter the pixel saturation problem and extract a higher dynamic range from the limited dynamic range PDA, a High Dynamic Range (HDR) processing technique has been applied to multispectral imaging where gradual optical attenuation is applied to the scene to acquire multiple images and electronic imaging processing is used to reach an unsaturated final image [12,13]. Although this HDR approach and other methods (e.g., increasing pixel size, controlling pixel integration time, resetting pixel output voltage or using non-linear amplification at each pixel site) can improve the dynamic range of a camera under overall bright light scene conditions [14], a fundamental problem can occur when further attenuation of the weak light target in the scene will send the PDA pixel irradiance information into the noise floor of the limited instantaneous dynamic range PDA. Thus, a spectral camera that avoids or limits the use scene light attenuation and also avoids calibration and control of the many individual pixels in a PDA is highly desirable for obtaining true high (e.g., > 130 dB) contrast images. Furthermore, it would also be highly beneficial that this extreme (e.g., > 130 dB) instantaneous dynamic range imager has simultaneous dual-broadband channel detection with all-optical or electronic domain fine tuning capabilities within each band and additionally provides a low (e.g., < ̶ 50 dB) inter-pixel isolation and inter-band isolation levels. Such an imager can be envisioned to work in unison with traditional prior-art PDA-based multispectral and/or hyperspectral imagers to extract the desired application specific pixel data for smart spectral imaging.

Therefore to put things in context, earlier we proposed and demonstrated an agile pixel imager (called by some as a single pixel camera/imager) using a Texas Instruments (TI) Digital Micromirror Device (DMD) that deploys a point Photo-Detector (PD) and agile pixel programming to spatially map an incident irradiance pattern [15]. In effect, the large micro-mirror count (e.g., 1 million micro-mirrors) DMD when combined with the large quantum well capacity point PD forms a PDA. Furthermore, an agile pixel programmed as a desirable shaped and positioned light sampler on the DMD extracts the incident irradiance on the DMD and transfers it for capture by the point PD, thus forming the agile pixel imager. To add operational robustness, a two point PD design has also been proposed and demonstrated for this agile pixel imager [16]. This starring-mode imager has been demonstrated for various spectral bands including UV, visible, and NIR [17], plus a design has been demonstrated that enables imaging plus optical spectrum analysis [18]. Recently, we have also shown how our original DMD imager can deploy time-frequency agile pixel processing to realize a much improved near 80 dB dynamic range imager called the Coded Access Optical Sensor (CAOS) camera [19–23]. Specifically, the CAOS imager assigns independent time-frequency codes to selected agile pixels in the image and simultaneously detects all the light and uses Radio Frequency (RF) wireless type time-frequency domain signal processing in-parallel to decode all agile pixel irradiances of the selected image pixels. The CAOS imaging mode is similar to operations of a mobile phone wireless network where specific telephone numbers are assigned to mobile phone users within a network of many simultaneous users where extreme sensitivity electronic signal processing is used to recover phone data from the very weak RF signal picked up by the phone’s electronic receiver. CAOS exploits the same extreme sensitivity and detection principles used in RF wireless networks to greatly enhance the dynamic range of our original agile pixel imager. Furthermore, simultaneous agile pixels detection via CAOS-mode can improve imaging speed. Do note that the basic starring mode DMD imager design has also been used for compressive sensing-based imaging [24] where image recovery is based on image projection measurements and iterative algorithm image processing. In contrast, the imaging works described by us [15–23] and demonstrated in this paper directly sample and preserve the true optical irradiances of the incident image and the imaging procedures are not based on compressive sensing techniques. One must also appreciate that both coding of light (e.g., via rotating patterned gratings for spectroscopy [25] and a moving reticle for television pictures [26]) and single PD irradiance map detection date back to over 50 years [27,28] and advances in devices and techniques within the modern wired and wireless optical and electronic communications and display infrastructure are seeding the CAOS imaging platform. In addition, DMDs have been used with CCD/CMOS imagers for spectrometry [29] plus control of camera parameters such as blooming [30], Field of View (FOV) and pixel level irradiance integration times [31].

Building on our prior works in two point PD DMD-based imaging and the CAOS camera, this paper for the first time provides the workings of the desired extreme contrast bright multi-spectral target scenario dual optical band CAOS imager design that features extreme (> 130 dB) instantaneous dynamic range as well as low (e.g., < ̶ 50 dB) inter-pixel and inter-channel crosstalk. A proof-of-concept dual-band imager is experimentally demonstrated using visible and near infrared LED targets in a viewed scene. All-optical narrowband filtering as well as CAOS-mode RF domain filtering is engaged for the first time with the CAOS-mode encoded image pixel data to extract finer wavelength content from the imaged agile pixel. Compared to our previous 80 dB CAOS mode dynamic range performance [19,20], it is shown for the first time how using a higher bits ADC with both amplitude range and sampling duration control along with a larger data set DSP implementation extends the imager dynamic range to a desired 136 dB level. The paper provides the detailed design and experimental analysis of the proposed simultaneous dual band CAOS camera.

2. Simultaneous dual optical band CAOS camera design

Shown in Fig. 1 is the proposed simultaneous dual optical band CAOS camera design. Multi-wavelength light passes through imaging lens L and a Smart Module (SM) optic to be incident on the DMD1 plane. The SM optic can contain light conditioning optics such as variable focal lenses, variable apertures, variable attenuators, and polarizers. The user decides the components make-up of the SM optics as well as on the two different spectral bands for camera operations called spectral band 1 and spectral band 2. Given the spectral band 1 and 2 wavelengths, the user selects the appropriate point PD unit detector technology deployed in the camera.

 

Fig. 1 Proposed simultaneous dual optical band CAOS camera design.

Download Full Size | PDF

The PD unit as shown can also use a Variable Gain Amplifier (VGA) module or engage optical amplification before photo-detection. Selected agile pixels on the DMD1 plane are time-frequency modulated with specific CAOS-mode codes enabling the image irradiance to acquire different RF codes, much like different telephone numbers assigned to different pixels in the optical image. The TI DMD1 micromirror has two tilt states causing the optical image time-frequency modulated light signals to simultaneously show up in both PD channels of the CAOS camera. As the DMD1 is a digital-only spatial light modulator, each agile pixel time-frequency modulation produces its CAOS-mode specific square wave electrical signal at the point PD outputs. L1 and L2 act as imaging lenses between the DMD1 and PD1/PD2 planes. For multispectral imaging operations, F1 and F2 are fixed broadband spectral filters matched to spectral bands 1 and 2, respectively. For hyperspectral imaging operations, TF1 and TF2 are tunable spectral filters matched to spectral bands 1 and 2, respectively. For example, F1, TF1 and PD1 in channel 1 of the camera can deploy visible channel devices while F2, TF2 and PD2 in channel 2 arm can use NIR wavelength selection components. Because the TI DMD1 operates between approximately 320 nm to 2500 nm wavelengths [32], a variety of dual bands can be realized for the proposed camera. There are various options for the required TF1/TF2 modules such as designs using bulk Acousto-Optical Tunable Filters (AOTFs), DMD, volume Bragg gratings, and liquid crystals [33–36]. PD1 and PD2 units provide RF signals whose RF spectra contain CAOS coded agile pixel irradiance data for the optical filtered light from the two different spectral bands of the camera. Specifically, PD1 and PD2 provide spectral image data for spectral bands 1 and 2, respectively. Decoding this detected RF spectra using time-frequency electronic signal processing recovers the agile pixel irradiance values for the selected wavelengths. RF signals from both PD1 and PD2 are recorded simultaneously and subjected to high speed electronic processing.

There are a number of innovative design features of the Fig. 1 imager design that create enhanced imaging performance in terms of dynamic range, inter-pixel crosstalk, inter-band crosstalk, and within broadband channel spectral channel crosstalk. Note the DMD time-frequency modulation CAOS-mode operation instantaneously sends selected agile pixels input image light to both optical detection channels of the imager. Because the two detection channels are physically separated, fixed broadband optical filters (F1/F2) matched to the point PD (PD1/PD2) broad wavelength band can be deployed to greatly reject the unwanted band light, implementing spatially powered optical spectral filtering. The CAOS-mode creates agile pixel light signals with RF modulation, each pixel having its unique electronic RF spectral signature that is decoded after photo-detection using RF wireless-style processing. Furthermore, note that TF1/TF2 can be programmed in an additional CAOS-mode to time-frequency modulate selected hyperspectral wavelength channels within agile pixels, allowing the use of RF spectral processing to detect specific narrow spectral band image data. Because RF domain signal processing can operate with extreme dynamic ranges, electronic filtering can be used to further reject optical spectral crosstalk within a broadband optical channel. Thus the proposed Fig. 1 imager can engage both optical and electronic filtering to reject unwanted optical wavelengths. In the same spirit, spatial crosstalk between agile pixels in the image can also be reduced using RF domain electronic processing.

Another innovative feature that is empowered by the CAOS-mode is the use of controlled ADC and DSP operations to extend the instantaneous dynamic range of the imager and in-turn improve image spatial and spectral crosstalk performance. Specifically, proposed and shown experimentally for the first time in this paper is how using a higher bits ADC with both amplitude range and sampling duration control along with a higher data set DSP implementation extends the imager dynamic range by creating significant signal processing gain and noise floor suppression. This electronic digital domain feature is possible because of the CAOS mode agile pixel operations via point PD detection that naturally produces an RF spectrum rich data set that can engage the power of wireless-style DSP processing. For example, consider a N-bit ADC using a sampling rate of f_s Kilo-samples/second with a agile pixel sampling duration time of t_s seconds. The voltage range of such an ADC can also be programmable and can be set between various voltage ranges to create a zoomed in extreme resolution detection of a weak signal. The output of the N-bit ADC then undergoes DSP, for example, spectrum analysis via DSP Discrete Fourier Transform (DFT) operation computed by the FFT algorithm implemented in a dedicated processor. The DFT produces processing gain resulting in noise suppression. Specifically, FFT-based processing gain in dB is equal to 10 × log(M/2) where M is the number of points in the FFT [37,38]. Given that the output of a point PD has a designed Dynamic Range (DR) of A dB, subjecting the point PD signal to FFT processing leads to a final output enhanced camera DR of A + 10 × log(M/2) dB. For example, a two million point FFT can produce an additional 60 dB processing gain for the imager, enabling exceptional agile pixel image and spectral map capture.

As described in our prior CAOS related work [20], the speed of operation of a CAOS platform-based imager depends on a number of factors associated with using the CAOS mode. These factors include the frame rate of the DMD1, response time of the point PDs, the number of activated simultaneous agile pixels on the DMD1, the time-frequency modulation code temporal parameters (e.g., code duration and frequency), ADC sampling rate and total sampling duration of an activated agile pixel, the speed of DSP to compute agile pixel irradiance values, number of hyperspectral channels, and speed of operations of the hyperspectral channel selection optical devices. Thus, one must optimize the CAOS camera parameters to meet the application specific imaging requirements. Also note that the spectral image pixel maximum dynamic range obtained via our imager depends not only on the mentioned optical control and electronic processing parameters, but inherently depends on the sampled scene spectral content and its spatial irradiance map. Hence the dynamic range and speed of the imaging operation should be optimized using initial scans by the proposed dual-band CAOS imager working in unison with image spatial and spectral intelligence gathered via prior-art spectral sensor systems and CAOS camera hardware-based other computational imaging methods.

3. Visible-NIR simultaneous dual optical band CAOS camera experiment

The proposed CAOS camera design for simultaneous dual-band imaging is implemented in the laboratory for the visible and NIR bands. The camera optical assembly uses a low cost DLP3000 TI DMD chip having a micromirror pitch of 7.637 μm, a 608 × 684 micromirror array arranged in a diamond configuration and a micromirror tilt angle θ = ± 12° with respect to the DMD normal. This DMD1 with its controls has a frame update rate up to 4000 Hz.

Lenses L, L1, L2 have focal lengths of 5 cm, 2.54 cm, and 2.54 cm, respectively. Distances for experiment are L to DMD1 = 5.3 cm, DMD1 to L1 = 8.4 cm, DMD1 to L2 = 8.4 cm, L1 to PD1 = 3.7 cm, and L2 to PD2 = 3.7 cm. The camera visible band Channel 1 (CH1) uses a PD1 that is a Thorlabs switchable gain silicon detector model PDA36A with an active area of 3.6 mm x 3.6 mm. The NIR Channel 2 (CH2) of the camera uses PD2 that is a Thorlabs switchable gain germanium detector model PDA50B with an active area having a 5 mm diameter. Fundamentally, PD1 silicon PD technology covers the spectral range from 350 nm to 1100 nm whereas PD2 based on germanium-based photo-detection covers wavelengths from 800 nm to 1800 nm. The visible channel in the camera deploys a Thorlabs bandpass color filter F1 model FGS900M having 315 nm to 710 nm transmission wavelength range. The NIR channel in the camera uses a Thorlabs high pass filter F2 model FEL0800 having 800 nm cut-off lower wavelength and a recommended upper wavelength of 2200 nm. To conduct the first basic narrowband imaging tests in the visible band, narrow bandwidth Thorlabs filters representing TF1 are mechanically deployed in the visible channel of the camera. Specifically FB450-40, FB550-40, FB620-10 filters have wavelengths centered at 450 nm, 550 nm, and 620 nm and Full Width Half Max (FWHM) bandwidths equal to 40 nm, 40 nm and 10 nm, respectively. The CAOS visible-NIR simultaneous dual optical band imager is first set-up using 50 ohm PD signal terminations, a STMicroelectronics STM32F4 microcontroller board using 12-bit ADCs and a Personal Computer (PC) that implements the FFT DSP. The control and synchronization of these components along with the DMD1 is done through the developed in-house Graphic User Interface (GUI) software.

The CAOS visible-NIR simultaneous dual optical band imager is deployed to image the scene shown in Fig. 2 made up by four bright LEDs having central wavelengths of 450 nm, 525 nm, 625 nm and 1450 nm with FWHM values of 20 nm, 25 nm, 14 nm, and 105 nm, respectively. These blue, green, red, and NIR LEDs have optical powers of 7 mW, 4 mW, 12 mW, and 5 mW, respectively. Both PD1 and PD2 units have VGAs with variable gain settings from 0 dB to 70 dB. To register strong photo-detected signals, these gains are set to 70 dB. To acquire viewed images by the camera for the visible band and NIR band, the camera is programmed in CAOS-mode so that two simultaneously agile pixels on the DMD1 are used to time-frequency modulate the dual-band optical image irradiance falling in the DMD1 chip. Each agile pixel is square shaped and programmed to have a 152.74 µm (20 micromirrors) agile pixel side. These two simultaneously agile pixels are programmed in the Frequency Division Multiple Access (FDMA) CAOS-mode [19] to oscillate at the RF frequencies of f₁ = 133.4 Hz and f₂ = 200.2 Hz. Both PD1 and PD2 generate these coded RF signals that undergo analog-to-digital conversion using the 12-bit ADCs operating at a sampling rate of 2.4 Kilo-samples/second and a total sampling time duration of 6 seconds. Square wave signal frequencies f₁ and f₂ are selected such that their harmonics do not overlap in the frequency space to create inter-pixel crosstalk. The 6 seconds PD sampling time is chosen to produce a sampled signal with a time-bandwidth product that closely matches M, the number of points in the implemented FFT. With the deployed point PD modules, the generated photodetector photocurrent i_PD(t) is given by $i_{PD}(t)={\displaystyle {\sum }_{n=1}^N{i}_n}(t)$, where i_n(t) is the photo-current produced by the CAOS-mode n^th agile pixel amongst the set of N agile pixels simultaneously falling on the point PDs. The photo-current i_n(t) is given by i_n(t) = c_n(t) × I_n(x,y) × A_n × R(λ) × D(λ), where I_n(x,y) is the spatially averaged irradiance in W/m² at the n^th agile pixel location centered at the x_n,y_n coordinates of the image map incident on the DMD1. Irradiance averaging is done over A_n, the area of the n^th agile pixel on the DMD1. c_n(t) is the time-frequency CAOS code used for the n^th agile pixel. R(λ) is the point PD responsivity value in units of A/W for a given wavelength λ. D(λ) ranges between 0 and 1 (1 is for 100% micro-mirror reflectivity) and is the optical attenuation factor due to the CAOS-mode DMD1 (or DMDs when TF1/TF2 contains a DMD) for a given wavelength λ. Recall that the DMD has a broadband spectral response from 320 nm to 2500 nm [32]. However the reflectance of the DMD window material is not uniform in this range. In [32] TI demonstrated that for the visible region range of 420 nm to 700 nm, the typical transmittance observed for the deployed DMD is approximately 97% indicating that D(λ)~1 for the 3 LED visible colours used in our experiment. Regarding the NIR wavelength region, the transmittance varies according to the specific NIR wavelength. For the test target LED having a 1450 nm wavelength, the transmittance is approximately 72% giving a D(λ)~0.72. To compute the true DMD1 plane incident spectrally distributed image irradiance, D(λ) is considered in the post-processing for image generation.

 

Fig. 2 Shown is the Visible-NIR bands target scene created using 3 visible LEDs and 1 NIR LED. Photos show LEDs off (left) and LEDs on (right). These photos are taken with a Nikon D3300 DLSr visible camera.

Download Full Size | PDF

The VGA voltage output v_n(t) at the ADC input due to the n^th agile pixel current contribution i_n(t) is given by v_n(t)=i_n(t)×G_A, where G_A is the trans-impedance gain of the VGA, thus v_n(t)= i_n(t)×G_A = c_n(t)×G_A×I_n(x,y)×A_n×R(λ)×D(λ). The FFT DSP is implemented on the v_n(t) signal which is a scaled version of the agile pixel optical irradiance as one can write v_n(t)= K×I_n(x,y), where the scaling factor K = c_n(t)×G_A×A_n×R(λ)×D(λ). The FFT DSP value for the n^th agile pixel is given as |S_n(f)|=FFT[v_n(t)] with frequency f in Hz. The digital data from the ADC representing v_n(t) is fed to the PC that implements 16384 point FFT-based agile pixel decoding to recover the visible band and NIR band irradiance values at the selected agile pixels.

To cover the entire scene being projected on the DMD1 chip plane, a total of 442 individual agile pixels are programmed and scanned in time sequence with N = 2 agile pixels per PD1/PD2 detection time slot. For the CAOS-mode FDMA codes, c₁(t) and c₂(t) are unity amplitude square wave signals with time periods of 1/f₁ and 1/f₂, respectively. Line-by-Line scanning is used giving a total agile pixel line scan length of 442 (number of pixels) x 216 µm (pixel diagonal) = 95.48 mm. Presented next in Fig. 3 is the scaled irradiance data acquired via FFT processing as cameras typically provide scaled irradiance images.

 

Fig. 3 (a) Visible scaled irradiance map obtained using the CAOS visible-NIR simultaneous dual optical band imager visible channel. (b) NIR scaled irradiance map obtained using the CAOS camera infrared channel.

Download Full Size | PDF

Figure 3(a) shows the visible scaled irradiance map obtained using the CAOS camera visible channel while Fig. 3(b) shows the NIR scaled irradiance map obtained using the CAOS camera infrared channel. In both cases shown in Fig. 3, the base 10 logarithm of the experimentally obtained scaled irradiance map I_s(x,y) = K I(x,y) is plotted along the scene x-y axes. Figure 3 shows that indeed the CAOS dual optical band imager has correctly located the positions of the 3 visible targets and 1 NIR target in the viewed scene.

Note that at this stage of camera operations, one has not conducted narrow spectral channel imaging within desired spectral channels for the detected dual-band images. To do so, the camera switches to its narrower band operations spectral camera mode by engaging TF1 and TF2 fine channel wavelength tuning in the two independent broadband channels in the camera. Specifically, since we are using 3 different color visible targets, the visible channel is tested for hyperspectral imaging by mechanically inserting a specific color wavelength filter (specifications of filters defined earlier) within the visible light channel of the camera. Figure 4 shows the experimentally obtained CAOS narrow spectral band irradiance maps at (a) λ = 450 nm with FWHM of 40 nm, (b) λ = 550 nm with FWHM of 40 nm and (c) λ = 620 nm with FWHM of 10 nm. The electronic FFT processed signal shown in Fig. 5 shows the detected light location (on the agile pixel scan line) and its normalized irradiance. Specifically, Fig. 5(a) plot is generated from PD2 while Fig. 5(b) plot is generated from PD1. This Fig. 5 data shows that the unwanted wavelength band falls in the noise floor of the FFT trace giving a measured inter-spectral band optical isolation of 10 log [1/(2.88 x 10⁻⁴)] = ̶ 35.3 dB (or an electrical isolation of ̶ 70.6 dB) between the visible and NIR bands for PD1. Regarding the measured inter-spectral band optical isolation for CH2, its value is computed as 10 log [1/(9.97 x 10⁻⁴)] = ̶ 30 dB (or an electrical isolation of ̶ 60 dB). Note that the inter-spectral band camera isolation can be further improved if cascading of multiple F1 and F2 filters is used in the CH1 and CH2 paths, respectively.

 

Fig. 4 Narrower spectral channel CAOS visible-NIR simultaneous dual optical band imager scaled irradiance maps at (a) λ = 450 nm with FWHM of 40 nm, (b) λ = 550 nm with FWHM of 40 nm and (c) λ = 620 nm with FWHM of 10 nm.

Download Full Size | PDF

 

Fig. 5 Shown is the normalized optical irradiance plotted versus agile pixel scan distance as the agile pixel conducts a line-by-line scan over the DMD1 plane making 442 agile pixels. Plot in (a) is generated from PD2 (NIR channel) while plot in (b) is generated from PD1 (Visible channel).

Download Full Size | PDF

Note that one can compute the optical (and electronic) contrast of the detected narrow spectral channel images by using the experimentally measured agile pixel irradiance values via the FFT spectral trace |S(f)| data. One can define the image electrical contrast in dB as being equal to 20 × log(|S(f)|_max/ |S(f)|_min). Here |S(f)|_max and |S(f)|_min are the maximum and minimum values of the electronically processed FFT signal for a given narrow spectral band image map. For the experimental narrow spectral band images at 450 nm, 550 nm, 625 nm, and 1450 nm, |S(f)|_max values are 510.5, 110.4, 788.9, and 58.15, respectively. At 450 nm, 550 nm, 625 nm, and 1450 nm, the |S(f)|_min values are 0.1473, 0.1587, 0.1315, and 0.1578, respectively. Given these values, the narrow spectral band images at 450 nm, 550 nm, 625 nm, and 1450 nm have measured electrical contrast ratios of 70.78 dB, 57.125 dB, 75.56 dB, and 51.32 dB, respectively. Note that the measured narrow spectral band image contrast value is strongly dependent on the various camera parameters such as optical filter and PD spectral responses.

As described in the design section, the proposed dual band CAOS imager can also engage the CAOS-mode in a suitable TF1/TF2 module to bring the power of RF domain signal processing and filtering into effect for in-band wavelength analysis for the selected agile pixels in the imaged scene. To demonstrate this imager feature, a TF1 filter based on the DMD spectral filter design, described in [33], is placed within the CAOS imager implemented in the laboratory. This TF1 design is shown in Fig. 6(a) and uses a 500 lines/mm visible band transmission amplitude grating G with another DMD called DMD2 (same technical specifications as DMD1) and an lens L3 of focal length of 3 cm that images the DMD2 plane onto the PD1 plane. The distance between L3 and PD1 is 4 cm. G is placed just after the lens L1 in the Channel 1 assembly to produce a spatially distributed white light spectrum (showing the main red, green, blue wavelength bands across ~400 – 700 nm) along the DMD2 horizontal axis (see Fig. 6(b)) covering a length of ~4.5 mm (the chip length is 6.57 mm). To create a suitable white light agile pixel on the imager DMD1 plane, a 5 W LED white light torch having a luminosity of 180 lumens placed 60 cm from L with a focal length of 7.5 cm is imaged on to the imager DMD1 plane where a 1.527 mm x 1.527 mm CAOS mode agile pixel is formed at a f₀ = 400 Hz test code frequency. This pixel’s entire white light irradiance shows up as spatially separated wavelength channels on DMD2 plane, all carrying a carrier code of 400 Hz that are summed up at PD1. The white light pixel generated current undergoes 2,097,152 points FFT signal processing to produce a white light agile pixel scaled irradiance data signal shown in Fig. 7(a). The data acquisition parameters using a higher performance ADC are: 16 bit resolution ADC, sampling rate of 20 Kilo-samples/sec and sampling duration of 60 seconds.

 

Fig. 6 (a) Shown is the design of the DMD2-based TF1 module used in Channel 1 of the CAOS camera that implements CAOS-mode narrower wavelength optical spectral detection. (b) Photograph of the DMD2 plane spatially distributed visible spectrum produced by light coming from the DMD1 programmed CAOS-mode white light agile pixel.

Download Full Size | PDF

 

Fig. 7 Shown are the detected RF spectrum peaks |S(f)| for different visible band spectral selections programmed by the CAOS-mode of DMD2 in TF1. (a) |S(f = f₀)| = 49.68 × 10⁻⁴ obtained when the entire DMD2 is programmed to direct all spectral components of the white light agile pixel at f = f₀ = 400 Hz CAOS-mode to PD1. (b) |S(f₀ + f₃)| = 0.4581 × 10⁻⁴ obtained when DMD2 is programmed to direct only the red color spectral band light to PD1 using a f₃ = 50 Hz CAOS-mode. (c) |S(f₀ + f₂)| = 1.091 × 10⁻⁴ obtained when DMD2 is programmed to direct only the green color spectral band light to PD1 using a f₂ = 62.5 Hz CAOS-mode. (d) |S(f₀ + f₃)| = 0.4597 × 10⁻⁴ and |S(f₀ + f₁)| = 1.074 × 10⁻⁴ obtained when DMD2 is programmed to simultaneously direct the red and blue color spectral band light components to PD1 using f₃ = 55.5 Hz (red) and f₁ = 83.3 Hz (blue) CAOS-mode.

Download Full Size | PDF

To independently monitor the color content of this white light agile pixel, next different CAOS codes are assigned to the different color/spatial zones defined as a second set of time-frequency CAOS mode agile pixels with frequencies f₁, f₂, f₃ for blue (λ1), green (λ2), and red (λ3) spectral bands, respectively. The spatial size of the agile pixel on DMD2 determines the spectral bandwidth of the spectral selection. DMD2 is programmed with a square agile pixel with a side length of 382 microns and a diagonal length (along the wavelength spread axis) of 540 microns. Given the 500 lines/mm grating period and a distance of 4.8 cm between G and DMD2, each agile pixel of a 540 microns width on DMD2 makes a spectral filter bandpass of (0.54 mm/4.5 mm) × 300 nm = 36 nm. In contrast to the CAOS coding of the agile pixels in the image space on DMD1, the DMD2 agile pixels simultaneously implement wavelength coding for the multi-color light for all agile pixels on DMD1. Figure 7 highlights the detected peaks |S(f)| for different visible band spectral selections programmed by the CAOS-mode of DMD2 in TF1. Figure 7(a) gives |S(f = f₀)| = 49.68 × 10⁻⁴ obtained when the entire DMD2 is programmed to direct all spectral components of the white light agile pixel at f = f₀ = 400 Hz CAOS-mode to PD1. In this case, DMD2 does not time-frequency modulate the incident light, creating a strong carrier signal at 400 Hz observed in the FFT spectrum. Figure 7(b) gives |S(f₀ + f₃)| = 0.4581 × 10⁻⁴ obtained when DMD2 is programmed to direct only the red color spectral band light to PD1 using a f₃ = 50 Hz CAOS-mode. In this case, the double CAOS-mode implementation using DMD1 and DMD2 produces double sideband time-frequency optical modulation, creating at f₀ + f₃ and f₀ ̶ f₃ the FFT spectrum sidebands around the carrier f₀. It is adequate to use either sideband spectral strength to determine the relative spectral strength of the incident multispectral light. Similarly, Fig. 7(c) gives |S(f₀ + f₂)| = 1.091 × 10⁻⁴ obtained when DMD2 is programmed to direct only the green color spectral band light to PD1 using a f₂ = 62.5 Hz CAOS-mode and Fig. 7(d) gives |S(f₀ + f₁)| = 1.074 × 10⁻⁴ and |S(f₀ + f₃)| = 0.4581 × 10⁻⁴ obtained when DMD2 is programmed to simultaneously direct only the blue and red color spectral band light components to PD1 using f₁ = 83.3 Hz (blue) and f₃ = 55.5 Hz (red) CAOS-mode. This white light agile pixel blue (λ1,f₁), green (λ2,f₂), red (λ3,f₃) color data indicates that compared to the red light content, blue and green light spectral contents are a factor of ~2.3 larger, a result also confirmed using optical power and filter-based direct spectral measurements of the light exiting the white light LED.

Thus the deployment of a programmable CAOS-mode TF1 and TF2 modules within the proposed dual band CAOS imager empowers spectral imaging, allowing the user to sift out desired simultaneous spectral features via electronic domain signal processing for high dynamic range spectral monitoring.

4. CAOS camera extreme dynamic range analysis and experiment

The camera dynamic range in dB is defined as DR (dB) = 20 × log(P_max/P_min) where P_min is the minimum detectable optical power by the PD and P_max is the maximum detectable optical power by the PD. To determinate the DR of both PD1 and PD2 modules in the optical design shown in Fig. 1, one must note that each point PD-VGA module has its own electrical characteristics such as the PD G_A, PD bandwidth (BW), and PD Noise Equivalent Power (NEP). These PD module parameters are dependent on the VGA switchable gain setting used during the optical image acquisition operation via the CAOS-mode agile pixels sampling. Considering an ADC having a maximum input voltage limit V_max, the PD’s maximum current output i_max can be found as i_max = V_max/G_A. G_A varies according to the impedance termination used which can be either a high impedance (Hi-Z) such as used in the CAOS camera extreme dynamic range test to be described next or the 50 ohms impedance termination such as used previously in the microcontroller-based imaging system test setup in the dual band imager experiment. The maximum detectable optical power P_max at a point PD can be found as P_max = i_max/R(λ). To compute the DR for the point PD, it is necessary to first compute the minimum detectable optical power P_min at the point PD. In order to do so, the NEP value of the PD needs to be considered. The NEP, which is the sensitivity of the PD device, represents the minimum optical signal that can be optically detected per square root bandwidth with units of W/√Hz. The NEP value is wavelength dependent and to compute its value the generalized formula NEP(λ) = NEP_min × [R_max/R(λ)] is used, where NEP_min corresponds to the NEP value at the PD’s peak wavelength λ_p and R_max is the responsivity value at the peak wavelength. By knowing the NEP value at the detected wavelength it is possible to compute the minimum detectable optical power P_min as P_min = NEP(λ) × √BW, where BW is the measurement electrical bandwidth of the PD unit. With both P_max and P_min values determined, the point PD DR and hence camera raw DR is computed.

To obtain the designed DR values for the wavelengths used in the dual band CAOS imager experiment, the following considerations are taken into account. The maximum voltage value V_max that the 12-bit ADC is able to sense is equal to 3 Volts. This 3 V value represents the PD1/PD2 optical saturation limitation for the dual band CAOS imager. The demonstrated imager PD1/PD2 units have VGA gains set to 70 dB with PD1 data sheet specified bandwidth BW₁ = 5 × 10³ Hz and PD2 bandwidth BW₂ = 210 Hz. Both PDs are designed to have 50 Ω termination impedances giving data sheet specified gains G_A1 = G_A2 = 2.38 × 10⁶ V/A. Via datasheets, λ_p = 970 nm and R_max = 0.65 A/W for PD1 and λ_p = 1550 nm and R_max = 0.85 A/W is for PD2. NEP_min corresponds to the NEP value at the PD’s peak wavelength λ_p. For PD1, NEP_min is equal to 2.10 × 10⁻¹² [W/√Hz] whereas for PD2, NEP_min is equal to 4.63 × 10⁻¹² [W/√Hz]. The hyperspectral imager experimental test wavelengths λ₁ = 450 nm, λ₂ = 550 nm, λ₃ = 625 nm, and λ₄ = 1450 nm have computed responsivity values R(λ₁) = 0.114 A/W, R(λ₂) = 0.22 A/W, R(λ₃) = 0.299 A/W, and R(λ₄) = 0.8175 A/W, respectively. The computed maximum detectable optical power at the specific wavelengths is P_max1 = 1.10 × 10⁻⁵ W, P_max2 = 5.73 × 10⁻⁶ W, P_max3 = 4.21 × 10⁻⁶ W and P_max4 = 1.54 × 10⁻⁶ W. Using the computed NEP(λ) values for the 4 wavelengths, the computed minimum detectable optical power by PD1 at the specific wavelengths is P_min (λ₁) = 8.46 × 10⁻¹⁰ W, P_min (λ₂) = 4.39 × 10⁻¹⁰ W, P_min (λ₃) = 3.23 × 10⁻¹⁰ W, and P_min (λ₄) = 6.97 × 10⁻¹¹ W. Using the calculated P_max and P_min values, the designed PD DR values in dB for the test wavelengths gives DR(λ₁) = DR(λ₂) = DR(λ₃) = 82.3 dB and DR(λ₄) = 86.88 dB. Indeed, these designed DR values are more than adequate to capture the deployed hyperspectral image targets that register at best an experimentally measured electrical contrast of 75.56 dB for the red LED target.

In order to test the extreme DR range (and image contrast) performance of the presented CAOS camera, a super bright target in the form of a 15 mW 633 nm Melles Griot He-Ne laser is deployed as a target in the viewed scene. In this experimental setup, Thorlabs Neutral Density (ND) filter-based variable optical attenuators are deployed to attenuate the laser light to simulate a high contrast target that reaches an extreme optical contrast ratio of 10⁷:1 (or an equivalent camera electrical DR of 140 dB). By deploying x-y slit scanning on the DMD1 plane, the spatial location of the deployed CAOS agile pixel is positioned to sample the high brightness laser target so that PD1 generates the maximum signal. A CAOS-mode modulation frequency of f = f₁ = 400 Hz is used in the experiment with N = 1 as there is one agile pixel. Given the red wavelength, only the PD1 channel of the camera is engaged. Specifically, the PD1 output is connected to a high termination impedance Data Acquisition (DAQ) card model 6211 from National Instrument that has programmable high speed 16-bit ADCs. PD1 can handle extreme optical power levels to produce maximum 10 V output signals and the DAQ card has a specified V_max = 10 V. Given the presence of a strong bright source (a laser beacon) in the target scene, the PD1 VGA gain is set to 0 dB which in-turn per datasheet sets the PD1 unit BW = 10 MHz and PD gain to G_A = 1.51 × 10³ V/A for the high termination impedance setting. The computed R(λ = 633nm) = 0.312 A/W and the computed maximum detectable optical power using V_max = 10 V is P_max = 21 × 10⁻³ W. At 0 dB setting, NEP_min = 2.91 × 10⁻¹¹ [W/√Hz] and NEP(λ = 633nm) = 6.06 × 10⁻¹¹ [W/√Hz] giving a computed minimum detectable optical power is P_min = 1.92 × 10⁻⁷ W. Using PD1, the designed raw DR capability of the CAOS camera when using the maximum acceptable DAQ voltage range of 10 Volts is computed to be 100.88 dB. However, using the deployed laser with the agile pixel size matched to the multispectral imaging experiments agile pixel size of 20 × 20 micromirrors, (152.74 µm × 152.74 µm), an approximately 2 V signal is produced from PD1, i.e., a factor of 5 below the PD1 total light detection capacity. In effect, the designed PD1 DR for the CAOS camera extreme DR test is 86.9 dB. It is well known that the Signal-to-Noise Ratio (SNR) in dB for a P-bit ADC is given by SNR(dB) = 6.02 × P + 1.76 [37]. With P = 16 bits, the deployed DAC has an SNR of 98.08 dB that is adequate for capturing the direct output of PD1 with a designed DR of 86.9 dB. Another point to note is that about 1/5 of the full well capacity of PD1 is used in our experiment implying that one could simultaneously detect 5 such extremely bright agile pixels in the scene by using up the full 10 V range of the PD1 output. Of course, one can decrease the agile pixel sampling area, thus reducing the total optical power collected by PD1 which leads to simultaneously registering a larger number of agile pixels in the image space. It is important to point out that the CAOS camera inherently makes the best and most efficient use of the relatively large full quantum well capacity of the point detectors PD1 and PD2. Such is not the case in prior-art PDA-based cameras where an incident bright extreme contrast image causes the many high spatial resolution smaller capacity quantum wells to be partially filled while many other quantum wells in the PDA are over filled and create spill over to nearby wells, thus causing pixel saturation and inter-pixel crosstalk noise. In short, CAOS is also quantum well capacity efficient.

The 16-bit DAC uses a sampling rate f_s of 20 Kilo-samples/second with a DAQ card sampling duration time t_s set to 5 seconds. The voltage range of the DAQ is programmable and set to the −10 to + 10 Volts range. The output of the 16-bit DAC undergoes RF spectrum analysis via DSP DFT operation computed by the FFT algorithm implemented in the PC. As mentioned earlier, the DFT produces processing gain resulting in noise suppression and the CAOS camera engages this special DFT capability to pull out the desired agile pixel data signal from the noise background created under extreme image contrast conditions. Specifically, FFT-based processing gain in dB is equal to 10 × log(M/2) where M is the number of points in the FFT [37,38]. M is first calculated as M = f_s × t_s and subsequently increased to a power of 2 number for computationally efficiency purposes so that the algorithm complexity decreases from O(n²) to O(n log n) where “O” stands for order of [39]. For the first stage of the high dynamic range experiment, f_s × t_s = 100,000. With M = 2¹⁷ = 131,072 satisfying the algorithm efficiency condition for the FFT implemented in MATLAB gives a designed processing gain G_FFT = 10 log (M/2) = 48.16 dB. Given that the output of PD1 has a designed DR of 86.9 dB, subjecting the PD1 signal to FFT processing leads to a final output designed camera DR of 86.9 + 48.16 = 135.06 dB.

To initiate the extreme dynamic range test to reach this 135.06 dB camera DR, optical attenuation of the laser beam is gradually increased using the optical ND filters starting with Optical Density (OD) values of 1, 3, 4, 5, 5.3 and 5.5 giving photo-detected output electrical signal attenuations of 20 dB, 60 dB, 80 dB, 100 dB, 106 dB and 110 dB, respectively. Here electrical attenuation in dB is equal to 20 × log10^OD. The signal FFT spectral traces for laser light optical attenuations of 0 dB, 50 dB, and 55 dB are shown in Figs. 8(a), 8(b), and 8(c), respectively. Note that when optical attenuation reaches 55 dB, the fundamental RF f = f₁ = 400 Hz FFT spectral peak due to the photo-detected square wave signal disappears into the spectrum trace noise floor. At this stage, the CAOS camera has demonstrated a camera dynamic range near 100 dB (50 dB x 2) where camera DR is 20 × log(|S(f₁)|_max/|S(f₁)|_min) with |S(f₁)|_max determined at the zero optical attenuation setting and |S(f₁)|_min determined when the RF spectral peak almost disappears in the spectral trace noise floor. Specifically, the Fig. 8(b) data gives an experimental DR = 98.6 dB, nearly matching the expected DR of 100 dB using the 50 dB optical attenuator. At the 55 dB optical attenuation of the target light, the camera produces the Fig. 8(c) FFT spectral signal where no clear f₁ peak is seen. In this instance, the DAQ is switched to its – 200 mV to + 200 mV super-resolution 50X DAC range compression mode. Doing so produces the Fig. 8(d) FFT spectral trace result where the peak is again clearly visible indicating a recovered camera DR of 110 dB. This recovery is possible because the DAC signal sampling resolution is improved by a factor of 50, changing from 30 mV to 6 µV allowing finer signal quantization, thus improving the digitization quality of the very weak RF signal produced by PD1.

 

Fig. 8 Highlighted are the detected peak |S(f₁)| for different optical attenuation, DAQ voltage range configuration, and sampling duration time. (a) |S(f₁)| = 1.228 obtained for a 10 volts DAQ range setting when no optical attenuation is used and t_s = 5 s. (b) |S(f₁)| = 1.43 × 10⁻⁵ obtained for a 10 volts DAQ range setting when ND filter is equal to OD = 5 and t_s = 5 s. (c) Peak at f₁ buried in system noise floor for a 10 volts DAQ range setting when ND filter is equal to OD = 5.5 and t_s = 5 s. (d) |S(f₁)| = 3.56 × 10⁻⁶ obtained for a 200 mV DAQ range setting when ND filter is equal to OD = 5.5 and t_s = 5 s.

Download Full Size | PDF

The Fig. 9(a) FFT trace giving a measured DR of 116.2 dB is obtained when a PD signal electrical attenuation of 120 dB is implemented using a ND filter with an OD of 6. An increase of the target light attenuation using a ND filter with a 6.3 OD (which is equal to 126 dB electrical attenuation) results in the FFT trace shown in Fig. 9(b) where once more the FFT spectral peak carrying the irradiance information of the CAOS-mode agile pixel signal is buried in the spectrum noise. In order to extract the CAOS pixel irradiance information from the FFT spectral peak, the sampling duration acquisition time parameter of the DAC is increased by a factor of 12, i.e., from 5 seconds to 60 seconds. For the MATLAB FFT, this increase in integration time increases the M value to 2²¹ = 2,097,152 giving a FFT processing gain of G_FFT = 60.2 dB. As it is shown in Fig. 9(c), using this increased processing gain recovers the FFT peak which is well visible with respect to the noise floor giving an experimental CAOS camera DR of 125.7 dB. With the increased G_FFT, the designed camera DR = 86.9 + 60.2 = 147.1 dB. Next the attenuation is further increased to a value of 6.6 OD which corresponds to a PD signal electrical attenuation of 132 dB. For this attenuation, Fig. 9(d) gives an experimental CAOS camera DR value of 130.5 dB. As shown in Fig. 9(e), by further increasing the applied optical attenuation to a value of 6.9 OD which is equivalent to 138 dB PD signal attenuation, one can no longer recover the super weak CAOS pixel irradiance signal since it is hidden in the FFT spectrum noise floor. From Fig. 9(e) a |S(f₁)|_min = 2 × 10⁻⁷ just above the noise floor gives an experimental CAOS camera DR of 136 dB. This extreme DR number of 136 dB showcases the power of the proposed wavelength sensitive CAOS camera technology.

 

Fig. 9 (a) |S(f₁)| = 1.902 × 10⁻⁶ obtained for a 200 mV DAQ range setting when ND filter is equal to OD = 6 and t_s = 5 s. (b) Peak at f₁ buried in system noise floor for a 200mV volts DAQ range setting when ND filter is equal to OD = 6.3 and t_s = 5 s. (c) S(f₁)| = 6.371 × 10⁻⁷ obtained for a 200 mV DAQ range setting when ND filter is equal to OD = 6.3 and t_s = 60 s. (d) S(f₁)| = 3.662 × 10⁻⁷ obtained for a 200 mV DAQ range setting when ND filter is equal to OD = 6.6 and t_s = 60 s. (e) Peak at f₁ buried in system noise floor for a 200mV volts DAQ range setting when ND filter is equal to OD = 6.9 and t_s = 60 s.

Download Full Size | PDF

Do note that the approximately 11 dB difference in experimental versus designed camera DR is due to several factors that includes error tolerances of optical and electronic components specifications from the datasheets, measurement errors due to deployed test instruments, and drift from the perfect linearity assumed for the point PD responses as well as their built-in trans-impedance amplifiers. Specifically, the deployed point PDs are designed for an ideal linear response in regions sufficiently far from the PD saturation point and the PD noise floor [40]. In an optimized CAOS camera design for application-specific field deployment, the experimental camera can be appropriately calibrated to produce true image irradiance data.

5. Conclusion

For the first time, proposed and demonstrated is a novel spectral CAOS camera design that has the inherent capability to provide very high dynamic range with simultaneous dual-band spectral imaging. The imager features low inter-pixel and spectral channel crosstalk via application dependent agile-pixels CAOS-mode programmability that allows electronic signal processing-based optical irradiance monitoring and filtering both in the image spatial domain as well as the optical spectral domain. When used with classic multispectral and hyperspectral imagers, this camera can extract useful spectrum specific irradiance image pixel data required by the end user. In effect, the proposed spectral CAOS camera can be programmed as a precision spatial and spectral irradiance sifter that pulls out otherwise unseen image data. Because the CAOS camera hardware allows the placement of programmable spatial masks on the DMDs, for certain sparse data set scenarios, the compressive sensing algorithm implementation is possible by first using the proposed camera’s non-CAOS staring mode [15] to record the point PD voltage value for each spatial mask in a sequence of P masks and then combining this point PD data set with the spatial mask data set via numerical optimization based post-processing to produce an estimate of the imaged scene. Thus after iterative computation using higher P values and modified optimization criteria, one gets an improved estimated sparse image data set of a limited dynamic range that can also be used to assist the camera CAOS-mode extreme dynamic range region of interest imaging operations. In such a hybrid design case, the agile pixel can be thought of as a space-time-frequency representation that combines time-frequency encoding/decoding of CAOS with spatial-coding of classic transform and compressive sensing as well as ghost imaging [41]. Speed of CAOS camera sifting operations depends on various factors include agile pixel size, pixel count, detector integration time, DSP algorithm and PC speed, as well as application dependent conditions and the parameters of the assist imager. Experiments for the first time using the proposed spectral imager have been successfully conducted with a 4 LED spatially distributed multi-colour (3 visible LEDs and 1 IR LED) test target observed using the CAOS-mode to spatially sample the image space and mechanically placed narrowband spectral filters to determine the specific target spectral bands. Images of the targets for the visible and NIR bands show a better than ̶ 60 dB inter-band measured electrical isolations. Fully programmable CAOS-mode spectral filtering implemented with a DMD-based narrowband filter placed within the CAOS imager demonstrates simultaneous wavelength filtering and monitoring of red, green, and blue bands of a white light LED target. Therefore, the power of dual CAOS-mode operations is demonstrated here, one CAOS-mode for the image space and one CAOS-mode for the optical spectral space. To increase camera dynamic range, proposed is the use of a higher bits ADC with both range and sampling duration parameter control along with a larger data set electronic DSP to extract higher DSP gain and realize additional noise suppression. Specifically, by experimentally engaging programmable 16-bit ADC control and the 60 dB DSP gain of a near 2 million point FFT applied to the point PD signal, an extreme dynamic range of 136 dB has been demonstrated for the CAOS camera agile pixel using an optical variable attenuator irradiance controlled visible laser target. These fundamental experiments point to the power of the CAOS camera for diverse applications in imaging. Future work using a much higher speed (e.g., 32 KHz frame rate) DMD relates to the optimization and extensions of the proposed dual band CAOS camera design to application specific demonstrations in multispectral and hyperspectral imaging including hybrid (H) CAOS camera designs engaging prior-art spectral imagers as well as prior-art computational imaging methods within the CAOS hardware platform.