Open Access
Add to BibSonomyAbstract
Imaging systems comparisons remains today a sensitive subject because of the difficulty to merge radiometric and spatial dimensions into a single, easy to use, parameter. By leaning explicitly on professional image users and their requirements we show how to build such a criterion, called Mission-Quality. A specific observation campaign is described and its results are used to calibrate and carry first proof of the criterion adequacy.
©2008 Optical Society of America
1. Introduction
The Mission-Quality criterion has been developed to solve trade-off conflicts in early stages of imaging system conceptions. It is, in essence, a measure of the quality of the imagery delivered by an imaging system for a given intelligence gathering mission. By requiring explicit descriptions of objects to be imaged, we were able to supersede previous criterions which either ignore imaged objects or credit them of ideal properties [1, 2] (as in Johnson’s bars test pattern). But such detailed information is impractical, thus we develop here an important simplification which founds the spatial characterization of observation missions on an independent synthesis carried by the photo-interpreters community which links observation missions to a minimal length to be observed. We show that this characterization translates into a measure of the deterioration of a disk image through the imaging system. Mission-Quality equations are then accordingly derived and we establish that a different model of image deterioration is needed to take fully into account the effect of noise on imaging systems.
2. The Mission-Quality criterion
Figure 1 illustrates the imaging process model we use in our developments. The discrete quantities are distinguished from the continuous by their indexes. Discrete quantities are indexed (i, j) and continuous ones are (x, y). Fourier Transforms of discrete and continuous signals, though, are both modeled as continuous frequencies.
Fig. 1. Model of image acquisition and reconstruction. III(x,y) being the two-dimensional Dirac comb and h(x,y) the point-spread function of the imaging system.
The numerical signal S is expressed as:
in which G is the steady-state gain of the global process, ls the sampling length, * the convolution operator, and h(x, y) the dimensionless point-spread function of the imaging system.
Since we want to evaluate the imaging potential of the electro-optical system, we assume that we don’t have to take into account display imperfections. Such defaults are easily offset by a skilled user with control over image presentation dynamic and magnification. There is now a continuous expression for the imaging process of a luminance field:
III(x,y) being the two-dimensional Dirac comb, h(x,y) the point-spread function of the imaging system, sinc(x)=sin(πx)/πx the cardinal sinus and NBL(x,y) is the band-limited noise arising in the discrete quantum process of photon measurement as well as in discrete electrical measurements. This noise is intrinsically limited by half the sampling frequency.
We assume that imaging system systems can only be compared through their final product, the image, and a requirement, in our case the fulfillment of the observation mission. In this respect we can build a metric based on a Fidelity Measure comparing the observed object and its image interpolated into the continuous object space (see Eq. 3)
Object being the continuous incoming luminance field of the observed scene and Image, the continuous luminance field reconstructed from sampled measures of the incoming field through the imaging system. Unfortunately this quantity diverges as soon as the noise enters the equation. It is thus necessary to limit the integration to a given spatial domain D (see Eq. 4), functions thus limited will be noted function|D. This is linked to the fact that a significant surrounding context is necessary to detect an object [3].
By assuming that the spatial domain of integration includes both object and image we can now drop the domain from the Object function. And, by using the Parseval relation we can write:
The Fourier Transform of these quantities is symbolized by a hat above. Unfortunately, this scheme implicitly compares signals at every scale. This means that, if we want to detect trees, the fact that the imaging system cannot render bark details is a definitive minus, whereas the mission might be achieved perfectly well without detecting bark. Thus, instead of considering extremely small scales, we propose using a scale range tailored to observation objectives. The limitation of comparison scales transposes itself directly in the Fourier domain into a simple frequency limitation truncation by a given maximum frequency νmax [1,2] (see Eq. 6) which will be detailed in the following paragraph.
Since frequencies above half the sampling frequency are irrevocably lost, the integration domain of the Quality measure is naturally subdivided by this frequency and the distinction can thus be made between the spectrum disturbed by the imaging process and the missing spectrum. Inside the useful spectrum, it is advantageous to further distinguish between the attenuated spectrum, which is characteristic of the continuous imaging process, and the aliased spectrum. Then we extract the noise and obtain the following expressions which characterize distinct physical disturbances [1,2,4]:
νe being the sampling frequency. The filtering term reflects the image disturbance caused by filtering the image spatial frequencies, the aliasing term represents the image disturbance caused by the folding of spatial frequencies, the loss term is the degradation caused by the absence of spatial frequencies falling outside the sampling passband, and the noise term is the degradation caused by the noise affecting the imaging system.
The aliasing term can be simplified by noting that image frequency separated by a full sampling frequency are uncorrelated thus, because of the integration of this quantity and the law of large number, the value of the second part of the aliasing term is null.
The same property shows that crossed terms in the development of the squared sum are also null, thus allowing a further simplification:
The aliasing term ends with a very simple formula:
Calculus confirmed this result with errors near the numerical precision. Having built a characterization of the quality of the transfer of an image through an imaging system we now show how to link it to the quality of the imaging system toward a given Observation Mission.
3. Observation mission translation into reference object(s) and maximum frequency
3.1 The Reference Object
Then, in fact, we need to exhibit some reference image whose transfer quality in the imaging system is representative of the fulfillment of the Observation Mission. We first proposed an approach consisting of characterizing any observation mission by the quality of the transfer of one or more reference shapes into the imaging system considered [1]. These geometrical shapes were to be extracted from the explicit list of all equipments photo-interpreters need to discriminate between. We established then how to transform this task of discrimination into tasks of detection of one or more isolated shapes, for which a Mission-Quality calculation is possible (with the knowledge of the shape radiometric contrast). A first experiment was then conducted with non-professional observers concluding at a good correlation between subjective assessment of the observation mission proposed and its Mission-Quality calculated value. But once this scheme had been proposed to professional photo-interpreters they disputed the adequacy of such a definition of the observation mission. They argued, in particular, that the actual list of equipment is intractable and that they use many undocumentable clues. Thus the only solution lies in an eventual correspondence between observation missions and reference image(s) made by photo-interpreters.
As a matter of fact, there is such a body of knowledge which is called the STANAG 3769 [5], a NATO standardization agreement. It has been developed by photo-interpreters and gives for any observation mission a single length value described as the “minimum resolved object sizes for imagery interpretation”, shortened as MROS. It states for example that “To determine shape, appearance, length-width ratio and the presence of major vehicular components a measurement of 0.6m is required”.
This length corresponds to the minimal spatial precision needed for the fulfillment of the observation mission. Since this value has been established before the advent of electro-optical imaging systems, it clearly doesn’t relate to the sampling frequency. Following such educated guidance we hypothesize that the spatial performance of an imaging system relative to a given observation mission lies into its capability to render a given size detail. Since no direction should be favored, it is natural to consider all orientations of our length of interest which give rise to the disk shape. So we establish a correspondence between the quality of the imaging of a disk shape on a uniform background and the observation performance of a STANAG 3769 class of equipment. Our claim is supported by previous works [6–8] in particular by A. van Meeteren [6], which presented experimental evidence that identifying a set of military targets embedded in nonuniform backgrounds is equivalent to detecting uniform disks on uniform backgrounds. The natural remaining step is to equate the disk diameter with the minimum resolved object sizes (MROS) given by the STANAG 3769 for the considered observation mission.
3.2 Maximum Frequency of interest for the Observation Mission
We now need to define the maximum frequency or, equivalently, the minimum resolution under which no more meaningful information can be collected for the observation mission. In practice object-related observation missions belong to one of four classes: Detection, Reconnaissance, Identification, and Technical Analysis. As we have seen in the previous paragraph, ‘standard’ lengths or ‘reference’ lengths can then be found on official scales of interpretation [5,9] for given kind of object (planes, vehicles, boats, buildings). This length cannot be used directly here to set the maximum frequency, since it would prevent our considering even slightly better resolutions, which might be needed. We simply propose using the ‘standard’ length corresponding to the next-higher category of observation mission because, if an imaging system can achieve qualitatively better missions, it is clearly over-specified. We consider that all the intermediate dimensions are potentially useful for the mission. It is the modified Fidelity measure that will decide between utility and drawback of using these dimensions.
We have noticed that the observable length needed for many military vehicles reconnaissance missions is nearly half the observable length (i.e. length/2) of the easier mission of detection [5]. Moreover the NIIRS [9] (National Imagery Interpretability Rating Scale) was also initially calibrated to a resolution relation which gives one-NIIRS difference as a result of a doubling or halving of the spatial resolution [10]. We suggest making this a general rule and defining the maximum frequency from a ’standard’ observable length or minimum resolved object sizes (MROS) by:
It means that it is unnecessary to boost the spatial resolution under a fourth of the length to observe. We tested the robustness of this parameter and found that as long as it is higher than 2, end-results of the Mission-Quality Criterion are very close.
3.3 Noise formula developments and simplification
Since the noise is spatially homogeneous, it would be sufficient to describe the domain D as a new disk centered upon the object disk. Its surface will be expressed as a multiple of the object disk surface:
The Fourier Transform of the disk object is:
MROS/2 being the radius of the disk and J1 being the Bessel function of order 1. Unfortunately since the primitive of this function is an Hypergeometric function, which is rarely natively implemented, its integration will be numerical. It is thus simple to numerically evaluate the denominator quantity of Eq. 16):
And by using the Parseval relation we obtain a very simple and familiar equation (see Eq. 19), except for its multiplier K. Since we didn’t found theoretical foundations for this value, it will be determined experimentally.
4. The subjective interpreting rating scale
The theoretical measure of Mission-Quality needs to be confronted to the reference of a scale representative of the success of the Observation Mission. Photo-interpreters favor a probabilistic rating scale of three paces: Possible, Probable and Sure. To insure more precise measurements, we asked them to divide each scale in two. We gave them a numerical scale (see Table 1) which, we hypothesize, is roughly linear in the perceptual space of the Quality of the Observation Mission.
Table 1. Rating scale used to evaluate the quality of images toward the observation mission
By the own admission of Photo-Interpreters, the “Almost Probable” level is the absolute minimum usable level. All lower score translates into failure to comply with the Observation Mission, i.e. Photo-Interpreters refuse to note anything in their report.
5. Validation of the Mission-Quality criterion by professional photo-interpreters
We need now to calibrate the numerical scale of the Mission-Quality criterion with the subjective interpreting rating scale. We have to select an Observation Mission, choose relevant imagery and possibly process it to simulate more Imaging Systems and observations conditions. Typically, such process is made of filtering, resampling and noise addition. All these images will be shown to photo-interpreters which will rate them with the Subjective Interpreting Rating Scale.
5.1 Images creation and properties relative to the observation mission
Since we own quality infrared images of military vehicles, we decided to test the STANAG 3769 Observation Mission of Vehicles Recognition which requires being able to determine shape, appearance, length-width ratio and the presence of major vehicular components. In this respect we chose a single image of various vehicles taken in the 3–5 µm Band around 15 h (local time) in the region of Champagne. It has been acquired by our low-noise bands II and III infrared air carried calibrated sensor Timbre-Poste. This image contains eleven relevant targets; french tanks, armoured troops carriers (VAB), trucks and light armour vehicles (VBL). These targets present comparable radiometric contrasts relative to their immediate environment, thus enabling photo-interpreters to give a single rating to the scene. Since shape, appearance and length-width ratio are fundamental to the Vehicles Recognition Observation Mission; we postulate that it is the average contrast between the whole shape and its immediate surrounding (typically five to ten times the target surface) which are relevant.
The use of a single image simplifies the procedure and makes it more reliable. And the professionalism of our military trained observers allows them to ignore any information they might have picked on other instances of this image.
We have simulated 60 different imaging systems strategies by varying important design parameters and propagating our reference image through them. The variation of the f-number of the imaging system (5, 10, 15, 20 and 25) governs the fuzziness of the image; the detector size (10, 20 and 30 microns) governs both fuzziness and aliasing. And the variation of the noise quadratic mean (0, 50, 100 and 150 digitization levels) governs the radiometric resolution of the imaging system. We defined thus twelve different ratios of optical cut-off frequency over Nyquist frequency ranging from 0.2 to 3.0 for a circular entrance pupil.
5.2 The experimental setup
The images thus defined were provided sequentially (from the worst to the best, in our initial opinion) to three professional photo-interpreters trained on military targets. The Geospatial Information System used was ER-MAPPER [11] and unlimited time was given to analyze each image, with unlimited use of spatial tools such as zoom or sharpening filters and radiometric tools such as histogram handling. With this kind of tools, these professionals are able to extract every significant drop of information from images they study. Since we postulated a linear perceptual space for the Interpreting Rating Scale we use, Photo-Interpreters measures are then averaged before confrontation with the theoretical Mission-Quality.
5.3 Comparison of experimental data with Mission-Quality predictions
Parameters necessary to compute the Mission-Quality for the Observation Mission of “Vehicles Recognition” are recalled Table 2.
Table 2. Values of imaging systems parameters
The simplest way to evaluate the pertinence of our model is to compute the correlation between measures and corresponding predictions. The knowledge of the constant K would be determined by the value giving the best correlation. Thus we plotted the squared K constant against the correlation it imparts to the Mission-Quality model (see Figure 2).
Fig. 2. Correlation between photo-interpreters average ratings and Mission-Quality predictions against different constant values defining the noise component of the Mission-Quality criterion.
Since the correlation shows wide variations, it is pertinent to our goal and it suggests that K takes a value of √6. It is now possible to plot photo-interpreters ratings against the Mission-Quality predictions (see Figure 3).
Fig. 3. Averaged photo-interpreters ratings of presented images versus its modified Mission-Quality predictions. A red line is drawn, linking the two unambiguous extreme points of the plot.
Such plot, if noisy, is remarkable since it indicates that there is a linear relation between photo-interpeters ratings and Mission-Quality:
Unfortunately, by incorrectly foreseeing a better Photo-interpreters efficiency we didn’t generate good enough images to get results higher than a Photo-interpreter rating of 4. Further confirmation will come from the study of the Mission-Quality behavior relative to each of the three different parameters, it is important to check that each one keeps the same relation between observers’ ratings and Mission-Quality results. Figure 4 plots these figures for each different Noise power and shows adequacy with the general behavior.
Fig. 4. (a). Averaged photo-interpreters ratings of presented images versus its modified Mission-Quality value with a Noise power of 0 digitization levels. (b). Noise power of 50 digitization levels. (c). Noise power of 100 digitization levels. (d). Noise power of 150 digitization levels.
Likewise, the behavior of the Mission-Quality relative to the f-number is consistent with the general behavior (see Fig. 5).
Fig. 5. (a). Averaged photo-interpreters ratings of presented images versus its modified Mission-Quality value with a f-number of 5. (b) f-number of 10. (c) f-number of 15. (d) f-number of 20.
And at last, the behavior of the Mission-Quality relative to the sampling strategy is also consistent with the general behavior (see Fig. 6).
Fig. 6. (a). Averaged photo-interpreters ratings of presented images versus its modified Mission-Quality value with a detector size of 10 microns. (b) Detector size of 20 microns. (c) Detector size of 30 microns.
Overall, the correlation between Photo-Interpreters’ ratings is 0.96 and the mean error is 0.26 in Photo-interpreters rating unit. This result is very satisfactory both in model adequateness and in the ease of use of the linear relation discovered between the observers judgment and the Mission-Quality prediction.
5.4 Consequence
Our model gives directly the minimum Contrast to Noise Ratio for resolved targets by taking into account only the noise component of the Mission-Quality criterion and setting it to its minimal value of 0.5. It follows that:
This value nears empirical imaging system SNR rules of conception.
5.5 Physical Interpretation of the Noise multiplicative constant
In the light of the experimental appearance of a multiplicative constant in the Mission-Quality noise component, we can advocate that the noise embarrasses the disk observation on another disk of radius √6 times higher than the radius of the observed disk. It might simply mean that this set the minimum surrounding surface needed to clearly detect the disk or detail. This surface being equal to 5 times the disk surface to observe. But only psychophysical studies could definitely prove this hypothesis. The final expression of the noise component is:
5.6 Mission-Quality Summary
6. Conclusion
From its earlier development stage we have matured the Mission-Quality Criteria into a tool very near full validation, whose strengths lie in its capability to compare very different imaging strategies, its ease of calculation, which allows the exploration of numerous imaging systems design possibilities and, paradoxically, by its explicit demand for a target bound spatial and radiometric requisite. In brief, The Mission-Quality criterion allows the easy insertion of users’ needs into the decision loop. Now, we aim towards a formal validation process with a wider group of photo-interpreters.
Acknowledgments
We would like to thank the photo-interpreters who have tolerated our peculiar questions and tried nevertheless to answer them; ADC Maxime Lafarge, PM Marc Lafitte and SGC Paul Pruvost of CF3I.
References and links
1. A. P. Kattnig, O. Ferhani, and J. Primot, “Mission-driven evaluation of imaging system quality,” J. Opt. Soc. Am. A 18, 3007–3017 (2001). [CrossRef]
2. A. P. Kattnig, O. Ferhani, and J. Primot, “A telescope design and performance analysis tool: the mission-quality criterion,” Proc. SPIE 5497, 396–404 (2004). [CrossRef]
3. A. Torralba and P. Sinha, “Statistical context priming for object detection,” Eighth International Conference on Computer Vision (ICCV’01), 763–770 (2001).
4. C. L. Fales, F. O. Huck, and R. W. Samms, “Imaging system design for improved information capacity,” Appl. Opt. 23, 872–888 (1984). [CrossRef] [PubMed]
5. STANAG 3769 (about the minimal resolution needed for photographic interpretation), NATO Standardization Agreements (NATO, Brussels).
6. A. van Meeteren, “Characterization of task performance with viewing instruments,” J. Opt. Soc. Am. A 7, 2016–2023 (1990). [CrossRef] [PubMed]
7. D. Sheffer and D. Ingman, “The informational difference concept in analyzing target recognition issues,” J. Opt. Soc. Am. A 14, 1431–1438 (1997). [CrossRef]
8. A. H. Lettington, D. Dunn, A. M. Fairhurst, and Y. Fang, “Proposed performance measures for imaging systems with discrete detector arrays,” J. Mod. Opt. 48, 115–123 (2001).
9. Imagery Resolution Assessments and Reporting (IRARS) Committee, “National Imagery Interpretability Rating Scale (NIIRS)”, http://www.fas.org/irp/imint/niirs_c/guide.htm.
10. J. C. Leachtenauer, W. Malila, J. Irvine, L. Colburn, and N. Salvaggio, “General image-quality equation: GIQE,” Appl. Opt. 36, 8322–8328 (1997). [CrossRef]
11. www.ermapper.com
