1. Introduction

There has been an increasing interest in systems that are able to visualize in real-time concealed objects under clothes, in bags, or through other light obstructing barriers. The well-known metal detector technology is inexpensive, but cannot be used remotely, does not provide imaging capabilities and can detect only iron-based materials. On the other hand, there are several new technologies that are able to image a broad range of materials. These technologies are based on active or passive millimeter waves, terahertz or back-scattering x-ray. However, these technologies are often relatively expensive, with a limited imaging range and most of them might not work in real-time at a reasonable emitted power.

Previous works [1–3] on all-ultrasonic systems showed that ultrasound waves could penetrate fabrics and could be used in order to detect various kinds of concealed objects with a very high sensitivity: an ultrasound wave is emitted towards a target, and the ultrasound echo is then recorded. In a way similar to medical echography, all the different layers below the clothes provide a weak echo, allowing the reconstruction of an image of the underlying object. However, efforts towards an ultrasonic imaging system for purposes similar to those outlined in this paper have not succeeded [3] in order to get a few centimeters imaging resolution, the ultrasound transmitter and receiver must be large (of the order of several meters), and must be installed near the target (a few meters away). Moreover, due to the ultrasound attenuation in the air, the range is limited, since the ultrasound wave needs to reach the target and to go back to the antenna which detects the scattered ultrasound wave. In fact, there is a link between the attenuation and the resolution. Typically ultrasound in air has a wavelength of ~1 cm for ultrasound frequencies above 30 kHz. For a large ultrasound dish of 1 m diameter and a target distance of a few meters, a resolution of approximately 1 cm is a physical limit caused by diffraction. Therefore, this technique has a limited range-resolution trade-off. Heyman [4] tried to overcome this limitation using non-linear acoustics, but the resulting signals from the target are inherently weak and the technique suffers from a non-practical geometry. The objectives of the technique presented here is to overcome these obstacles, at least partially. We suggest and demonstrate remote imaging of objects concealed under clothes by using a combination of ultrasound and coherent light.

Let us first describe briefly the way we use the coherent light and the ultrasound wave. Acoustic waves easily penetrate clothes and are being reflected differently by the human body compared to concealed objects, due to a difference in the acoustic impedance. When back-reflected, the acoustic wave induces a reflected vibration that contains information regarding the concealed object. This wave combines with the incident acoustic wave on the outer layer, in order to form a spatial interference pattern that reflects the concealed object shape. Therefore the acoustic wave transfers in-depth information onto the outer, diffusive surface of the cloth as an acoustic interference pattern.

On the other hand, coherent light impinging on a turbid medium generates a random speckle pattern with known statistical properties [5]. This pattern is extremely sensitive to minute changes in this turbid medium. Therefore it is in principle possible to learn from changes in the speckle pattern about changes in the turbid medium. In addition, since changes in the speckle pattern follow practically instantaneously changes in the medium properties, analyzing the speckle pattern is very attractive for studying dynamic behavior of the turbid medium [6–12]. In these recent years, optical imaging based on static or dynamic speckle sensing has been the object of increased interest in a wide range of scientific fields: quantitative analysis of blood flow [13], imaging through fog or smoke [14], imaging of vibrating objects obstructed by turbid media [15], 3D shape measurement [16], and speech extraction [17]. The technique we are describing here combines both properties: the dynamic acoustic pattern formed at the outer surface of the cloth is sensed, measured and analyzed through the speckle pattern generated by the coherent laser beam.

A fast camera images the speckle pattern and detects minute changes in in its structure. Mathematical algorithms are then used in order to reconstruct the reflected acoustic intensity, and therefore the shape of the concealed object. We demonstrate that this technology detects concealed objects of various shapes and of different materials in real-time at a working distance of 15m, and with a high spatial resolution (on the order of millimeters). This is made possible because the technique measures the near field ultrasound pattern optically, and therefore is not limited by the acoustic wave wavelength.

2. Concept description

The setup description is given in Fig. 1. An ultrasonic wave is sent towards a target, together with a laser beam. The ultrasonic wave can penetrate the barrier and is reflected back from each interface. The laser beam cannot penetrate the barrier but is reflected back by the most external layer. When the ultrasonic wave is reflected by the inner layers, it makes the outer layer vibrate at the same ultrasonic frequency with a certain delay corresponding to the time the ultrasound wave spends during its travel from the outer layer to the inner layers and back. The strength of the vibration essentially depends on the reflection coefficient of the different inner layers. Therefore, the reflected laser beam will be modulated, depending on the strength of the reflected ultrasonic wave.

 

Fig. 1 A schematic description of the detection concept. An ultrasonic wave (US) is sent towards a target, together with a laser beam. When the ultrasonic wave is reflected by the inner layers, it makes the outer layer vibrate at the ultrasonic frequency which is detected by the dynamics of the speckle pattern of the laser light. At the positions where there is a concealed object under the clothes, the ultrasound wave is reflected back stronger, and the object can be visualized.

Download Full Size | PDF

In order to illustrate the demodulation principle, we will consider what happens on one single optical detector. We will also consider only two main ultrasonic reflections: one from the outer layer and one from the object beneath the barrier. We will neglect all other reflections (such as on other barrier layers). The more general case of multiple reflections can be easily generalized.

The signal S on the detector can be written as:

An equivalent process that is easier to implement in this case is to open a gate for a short time, two or four times per period (defined as 2π/$\text{ω}$), accumulating the signal for each phase (therefore obtaining respectively two and four data points). In the later case, one obtains for the case of the four phases:

The value of (S2-Sl)² + (S4-S3)² = 4(a² + b² + 2ab.cos($\text{φ}$)) is independent of $\theta$. If b is large compared to a (for example in the case of reflection on metallic object) then the signal is approximately ~4b² and the spatial variations in the signal essentially reflect the spatial variations in b².

Since the information carried by the back-reflected laser is contained in its phase, in order to retrieve the modulated signal, light from the target must be mixed with light from the same laser on the detector (local oscillator). The mixing process has three major objectives:

Usually, this homodyning process is performed with a local oscillator that is derived from the laser itself, and mixed with the signal. This leads to additional hardware and is not optimal in this particular case. In fact, light that is reflected back by the most outer layer is only slightly modulated by the ultrasonic wave. Most of the light is not modulated at all. It can therefore serve as a local oscillator that amplifies the modulated signal (auto-homodyning). A major advantage of this process (compared with a local oscillator directly derived from the laser) is that the local oscillator and the signal propagate along the same optical path until the target. The mixing therefore removes atmospheric fluctuations that would have spoiled the signal to noise ratio. A second advantage of the auto-homodyning is that the coherence length of the laser can be relatively small. In the standard homodyning process, it must be at least twice the laser-target distance. After the homodyning process, the useful signal is located at the ultrasound frequency that acts now as a carrier in an amplitude modulation scheme. This demodulation process must be performed independently for each pixel. When the laser scans the entire target, an image can be obtained, that displays the ultrasonic reflection strength at each region of the target, and therefore provides an ultrasonic image of the target.

Additional considerations are worth mentioning: the choice of the acoustic frequency, the influence of the speckle size and the parameters influencing the signal to noise ratio. The choice of the acoustic frequency is driven by two main considerations. As we explained before, the dependence of the resolution on the acoustic frequency is weak since we are sensing the acoustic near-field, and therefore this is not the primary concern. The first consideration is the acoustic penetration within the fabric: the lower the frequency the better the penetration, and the lower the attenuation of the acoustic wave in the air. The second consideration is the system size: the emitter size grows typically as the square of the wavelength, so the higher the frequency the smaller the system. The compromise therefore depends on the exact application. In our system we chose the 19.6 kHz transducer with an active diameter of 20 cm and 7° @-3dB full angle beam opening. The speckle size has a major influence on the signal quality and measurement precision. The speckle size is determined by the degree of defocussing of the optics. Strong focusing generates very small speckle so that a large number of speckle zones are imaged onto one single pixel, therefore smearing the temporal speckle fluctuations and reducing the signal to noise ratio. On the other hand large defocusing leads to a loss of resolution and smearing of the spatial information. Although the precise level of focusing is largely dependent on the overall system, a speckle size to pixel size around 1 seems adequate. Finally let us consider the parameters that influence the signal to noise ratio. Since the technique is intended to be performed on moving people, the optical signal integration time is limited by the speckle decorrelation time, so in the best configuration, the experimental noise is ultimately the optical shot noise. This in turn determines the weakest acoustic power that must be emitted in order to get an optical signal above shot noise times the number of desired grey levels (dynamic range).

3. Target description

We have used two types of targets: mannequins and people-based. The target in general comprises an object attached to the upper part of the body, and is covered by one or several layers of fabrics (Fig. 2). In the different experiments, the size, materials of the objects as well as the kind of fabric and the number of layers were varied in order to assess the concept validity and detection limits.

 

Fig. 2 Example of target set-up: different objects are attached on the upper part of the body, and are covered by one or several fabric layers (clothes).

Download Full Size | PDF

In Table 1 we show the transmission of a direct 30 KHz ultrasound beam through different kinds of fabrics.

Table 1. 30KHz ultrasound transmitted in present throw difference clothes and clothes materials

View Table

4. Single detector detection scheme

In order to validate the technique, we initially performed a point by point measurement, by scanning a laser beam on the target and using a single detector. The setup is described in Fig. 3. A 780nm coherent laser beam (Power Technology Model PPMT25) with a few MHz linewidth is coupled to a collimator with an output beam diameter of 3 mm and a beam divergence of ~1 mrad. The beam is scanned on the target using a motorized scanning mirror. The mirror angularly moves in two orthogonal directions at a constant angular velocity, therefore providing a two-dimensional scan of the target. The typical laser spot velocity on the target is 30 mm/s. We simultaneously transmit a collimated continuous ultrasound wave at the frequency of 50 kHz towards the target.

 

Fig. 3 Experimental set-up: A signal generator (SG) generates a 50 KHz pure sine signal that is amplified and emitted through the ultrasound transmitter. On the other hand a diode laser beam is sent to the target through a motorized pan-tilt mirror. A photomultiplier detector (PMT) with limited aperture and dual amplifier stage detects the impinging optical signal which is then digitized using an analog to digital card (ADC). The signal is then processed in the computer. The detector is built around a linear array of 16 photomultipliers. One of the advantages of the photomultiplier tubes is their large dynamic range. Each element’s detecting area is 1x16 mm. The laser spot size on the target is chosen such that the speckle size fits the short dimension of the photomultipliers area.

Download Full Size | PDF

Because this setup collects surrounding light, it is very sensitive to the background light. We applied a collector lens with a large focal length (20 cm) in front of the detectors in order to narrow the detection field around the target and reduce the background. In addition to the spatial filtering, a narrow interference filter was set in front of the photomultiplier array in order to spectrally filter the surrounding light. For each of the 16 channels, the detector is AC coupled to a sensitive preamplifier and a secondary amplifier which amplifies the signal to the desired level. Between both amplifiers we set a high pass filter in order to avoid signal saturation. Amplified signal is sampled by a fast analog to digital converter (National Instruments) at 200 KS / s sampling rate (16 bits sampling). For each channel, traces of 4096 samples are then sliced and the power spectrum of each trace is calculated. The trace length was determined by maximizing the signal to noise ratio, as explained below. The power spectra from all the channels are then averaged. The modulation signal peak (at the ultrasound frequency) and the background around the peak are measured for each laser position. The background signal is removed from the peak signal. The resulting signal is plotted on an intensity scale on a grid representing the positions of the laser beams where measurements have been performed.

5. Results with the single detector detection scheme and set-up limitations

In Fig. 4(a) we use a mannequin with a concealed object made of cloth and filled with sugar powder. Figure 4(b) displays the intensity map obtained according to the procedure described above. As expected, we observe an increase in the optical signal when the laser beam scans the region when the object is concealed. Elementary image processing is performed on the map, such as threshold operation, in order to facilitate object recognition (Fig. 4(c)).

 

Fig. 4 Single detector scheme results: (a) Concealed object. (b) The modulated image at the ultrasonic frequency (scanned area) (c) The detected image obtained after a threshold operation on the data.

Download Full Size | PDF

We have performed experiments on a wide variety of clothes and objects. This set up is very sensitive and has enabled to check the concept with success. However the scanning process is very slow in order to obtain a high enough signal to noise ratio (SNR). During the scan the target must be still and experiments with real people led to poor results. In fact the maximum individual trace duration is limited by the decorrelation time of the speckle pattern. When the duration of the trace is longer, random phase jumps occur and the overall signal tends towards zero. The trace duration therefore must be smaller than the decorrelation time. The signal to noise grows linearly with the trace duration until the duration reaches the decorrelation time, and then grows like the square root of the number of traces that are integrated. Therefore small movements of real people decrease the decorrelation time and therefore strongly decrease the SNR. However, practical applications require a real-time system capable of scanning a large region of interest, and able to reach a few tens of frames per second. The way to do it is to replace the time integration by spatial integration, signal from several nearby detectors is averaged, and we therefore used a high speed camera.

6. Parallel detection using a high-speed camera

We replaced the single detector and scanning system by a high speed camera and a wide area laser illumination depicted in Fig. 5. We then output the data at a very high rate to a PC for further processing. We designed a specific digital signal processing tool in order to detect tiny intensity modulation at the ultrasound frequency.

 

Fig. 5 Characteristics of the detection set-up: The system is built around three main components: the ultrasonic transmitter (US), the 25W fiber laser and the fast SWIR camera. The system is controlled by a controller and process units that also performs real-time processing of the raw data and display it on the screen.

Download Full Size | PDF

Since the setup is designed for experiments with people, we chose to work at an eye-safe wavelength (1550nm) where the maximum permissible exposure (MPE) is more than 50 times higher than that of near infrared wavelength at the same conditions. Since we need to sample at least 2 points per ultrasound cycle, the camera's frame rate had to be very high (frame rate > 40,000 fps). This prevented us from working with the camera at a full field but only on a small Region Of Interest (ROI). In addition we used a 25W high power high coherence length fiber laser (IPG model ELR-25-1550-LP-SF).

The laser is expanded to about 1 meter diameter at a distance of 5 meters. The camera (XenICs model Cheetah 640CL) is equipped with a 50 mm focal length lens F/4 (Navitar SWIR lens f = 50mm). A continuous wave ultrasound transmitter with a central frequency of 19,600 Hz and an acceptance angle of 7 degrees is located in front of the optical system, and sonificates the target. We made sure that the three axes (ultrasound, laser and camera) cross at the working distance of 6-9 meters. An additional color camera Prosilical GC650C is installed above the high-speed camera, and takes a 100 fps color video recording of the whole scene. The high speed camera is connected to the computer system (8 cores PC platform, 16GB RAM) through a GigE communication port. The SWIR camera data is grabbed into the computer RAM memory through a Camera Link (full) 8 bit frame grabber (National Instrument NI PCI-1429). The system software application contains two parts: a C + + application for grabbing, processing, visualizing and saving the data in real-time, and a Matlab application that loads the acousto-optic processed data, post-processes the acousto-optic images, applies a detection algorithm and merges the binary detected image with the color image and finally concatenates the images as a movie. The system performance in this demonstration was:

In accordance with the Nyquist criterion, the high-speed camera must grab images at a frequency of at least twice the ultrasound frequency. the sampling rate of the whole image is equivalent to the sampling rate of a single detector, as described above. The pre-processing concept of the data is presented in the picture below:

Once the scene is grabbed, we reshape the data to isolate the trace of each pixel independently, as if it was a single detector, to enable the analysis each pixel as a function of time. Because we are not interested in the phase information of the ultrasound but only in its intensity, we perform a power spectrum operation on the signal and extract the signal at the ultrasound frequency in the power spectrum, in a way similar to what were presented for a single detector (Fig. 6). We repeat this process for all the pixels and get a 2D image of the power spectrum of each pixel at the ultrasound frequency. This is the acousto-optic image of the target. The integration time of each acousto-optic frame is the product of the number of the integration time per frame (inverse of the frame rate) and the number of frames. For the needed frame rate, we had to reduce the region of interest (ROI) of the high-speed camera to 128 columns by 64 lines. In order to increase the field of view of the system, we electronically shifted the ROI location of the camera sensor after each acousto-optic frame. We obtain a mosaic of acousto-optic images that are assembled together at the end of the cycle. The ROI position is controlled electronically via the camera control communication port.

 

Fig. 6 Scheme of the detection concept and processing. A series of frames is grabbed (a). Then for a given pixel, amplitude for each frame is reported as a function of the time (b). A power spectrum operation is then performed on this trace and the value at the ultrasound frequency is recorded.

Download Full Size | PDF

7. Results and discussion

Extensive experiments have been conducted. Variations due to the following parameters have been measured: indoor and outdoor conditions, target’s movement, response to different materials, shapes and locations on the subjects’ bodies. The prototype settings were fixed and the system recorded for 8 seconds. Operation range was set to be 6-7 meters. Different kinds of materials, shapes, clothes and locations on the body have been recorded.

Various objects have been tested (Figs. 7 and 8): A metallic gun, a plastic bag with different powders, a plastic bag with metallic screws, a plastic bottle with liquid, electric wires, modeling clay, a hammer and a plastic remote control device.

 

Fig. 7 (left) Plastic bags filled with organic powder (flour, sugar etc.) which resemble improvised explosives. (right) the AO image from the recorded movie. The bags are clearly visible.

Download Full Size | PDF

 

Fig. 8 (left) Plastic bottle containing liquid. (right) the AO image from the recorded movie. The bottle is clearly visible too.

Download Full Size | PDF

Good results were obtained from all objects and locations on the body. Especially good results were obtained from nylon bags, empty or filled and plastic based materials. It was found that a skilled operator could detect an object with a weaker response (for example when experimenting with thick clothes or low cross-sections).

We obtained movies of moving people at the distance of 7 meters, concealing different kinds of objects and wearing different kind of clothes. We also created a movie of a walking person from 15 m to 7 meter in front of the system.

6. Conclusion

A new technique based on the combination of ultrasound and laser beams for remote sensing and detection of concealed objects under clothes has been demonstrated. A proof of concept real-time system has been developed and conclusive results have been established: we have presented for the first time the capability of real-time remote imaging of concealed objects under clothes on moving and walking people, both inside and outside, and even under sunny conditions. The system provides mm range resolution and currently a working distance of 5 to 10 meters. We are in the process of developing a low-cost version of this system based on a “lock-in” camera.