Open Access
Add to BibSonomy
Abstract
Sensitivity to micro-vibration generated in an optical alignment ground facility increases with the optical aperture of the electro-optical payload mounted on the remote sensing satellite. The micro-vibration distorts the edge spread function (ESF) and changes the modulation transfer function (MTF) value during MTF measurements. Here, an ESF reconstruction network (ESFNet) is proposed to reconstruct the ESF distorted by micro-vibration. A proxy ground-truth ESF dataset for stable training of the proposed ESFNet via a supervised learning method is introduced. The proposed method is robust against both overfitting and underfitting for distorted parts of the ESF caused by micro-vibration. Experiments using ESF obtained in the facility with real micro-vibration show that the measurement error of MTF is around 0.02% when the ESFs are reconstructed using the proposed ESFNet.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Modulation transfer function (MTF) is one of the critical quality criteria for evaluating the spatial resolution of electro-optical systems (EOS) [1–4]. There are several methods for measuring MTF, and appropriate methods should be selected and applied according to the characteristics of each system domain. For electro-optical satellite payload, push broom image sensors are widely used, and the line rate of the push broom sensor is set in response to the moving speed of the knife-edge target. In this case, MTF measurement uses the edge image created by the moving knife-edge target [5].
MTF measurement using an edge image calculates MTF through the sharpness of a straight edge center line between two bright and dark surfaces. As shown in Fig. 1, edge spread function (ESF) is measured through the pixel value obtained along the edge center line, and then line spread function (LSF) is calculated by differentiating the ESF. Finally, the MTF curve is obtained by converting the LSF into a frequency domain through fast Fourier transform (FFT).
Fig. 1. Calculation steps of MTF: (a) edge spread function, (b) line spread function, and (c) modulation transfer function.
The measurement precision of the MTF depends on the quality of the measured ESF. However, the facility operating conditions significantly affect the measurement of ESF in the actual EOS alignment environment. Recently, the optical aperture of EOS mounted on satellites has increased to more than 1 m for acquiring detailed images. Accordingly, optical sensitivity has increased rapidly [6]. For this reason, micro-vibrations generated by the movements of workers inside the facility, overhead cranes, and air conditioning system get added as noise during ESF measurement. Most optical alignment tasks were performed on an optical table equipped with vibration isolators to reduce the influence caused by such micro-vibration. The isolators attenuate vibration below 10 Hz by more than 95%. However, the air conditioning system and overhead crane of the optical alignment ground facility are the most representative global vibration sources; it is therefore difficult to realize effective attenuation by using the isolators under the optical table because they vibrate the building's frame structure itself. This appears to be a random noise effect because it causes residual vibration according to the frequency component of micro-vibration and accumulates owing to the characteristics of continuously applied global vibration. Figure 2 shows how micro-vibration can distort ESF during its measurement. The center of the edge, which should be a sharp decline, is distorted and bent. These get added as high-frequency components during the frequency domain conversion of the MTF measuring process, preventing the MTF curve from smoothly declining and changing the MTF value itself at the Nyquist frequency. Many engineering factors, such as the structure of the clean room, crane access, and blocking of the vibration sources, are considered at the design stage of the facility building to reduce micro-vibration and atmospheric turbulence [7]. Moreover, during MTF measurement, all peripheral devices are stopped and the movement of the experimental participants is controlled to minimize the occurrence of micro-vibration. However, since vibration sources such as the air conditioners of the building or coolers for cooling experimental equipment may still exist, it is impossible to block all micro-vibrations completely.
Conventional studies have focused on eliminating noise or aliasing that may occur in the generation of pixel values by digital imaging systems or reducing misalignment errors in slanted projection to measure ESF [8]. Additionally, when calculating LSF using the ESF, a differential operation is performed, during which slight noise that may exist in the ESF can be amplified to increase the error in the generated LSF.
Generally, a curve-fitting algorithm is applied to exclude this error. Hang Li et al. defined a transfer function model for each component of the digital imaging system. Based on this, an ideal knife-edge function was generated to estimate ESF [9]. Tiecheng Li et al. performed ESF fitting using various analytic function models, and among them, the Fermi function result showed the highest precision [10]. Françoise Viallefont-Robinet et al. showed that noisy ESFs could be effectively fitted through the cubic smoothing spline algorithm [1]. However, the curve fitting-based ESF reconstruction techniques proposed in previous studies have been applied to prevent the generation and amplification of sampling noise in digital imaging systems, and the micro-vibration effect is not assumed. Furthermore, underfitting may occur when reconstructing ESF distorted by micro-vibration through existing curve-fitting algorithms, as shown in Fig. 3(c), which can distort the inherent MTF characteristic curve of the original digital imaging system. Conversely, as shown in Fig. 3(d), overfitting may also occur, and the effect of micro-vibration may not be sufficiently excluded.
Fig. 3. ESF reconstruction using existing curve-fitting algorithms: (a) Original ESF. (b) Noise-added ESF. (c) Curve-fitted ESF using Fermi function (case of underfitting). (d) Curve-fitted ESF using cubic smoothing spline (case of overfitting).
This study proposes an ESF reconstruction network (ESFNet) that can alleviate the distortion caused by micro-vibration by applying the convolutional neural network (CNN) technique. First, we introduce a proxy ground-truth ESF (PGT ESF) for training the proposed network in a supervised learning manner and construct a training dataset based on the PGT ESF [11]. Second, the proposed ESFNet eliminates the ESF distortion caused by micro-vibration inside the EOS alignment facility and sampling noise of digital EOS with higher precision than the conventionally used curve fitting-based ESF reconstruction methods.
The rest of this paper is organized as follows. Section 2 describes the proposed ESFNet, including dataset preparation using PGT ESF. Section 3 presents the experiments for the performance evaluation of the proposed scheme. Finally, section 4 presents the conclusions drawn from this study.
2. ESF reconstruction method based on the deep learning network
2.1 Dataset preparation using proxy ground-truth ESF
To obtain desired results via a deep learning network, it is necessary to generate a training dataset through an appropriate method. Particularly, to train a deep learning network through a supervised learning method, pairs of ESF distorted by micro-vibration and ground-truth ESF data must be input into the network. However, it is challenging to obtain an ideal ground-truth ESF in which there is no influence by micro-vibration in an actual operating environment of EOS. To address this problem, this study extended the conventional method of obtaining ESF only from the edge-centered pixel. As shown in Fig. 4, ESF was extracted from the center of the edge in a range of ±6 pixels to obtain a total of 13 raw ESFs.
The 13 raw ESFs represented signals of knife-edge targets taken at slightly different times depending on the nature of the push broom sensor. In other words, the 13 raw ESFs include influences caused by micro-vibration at different time points. The magnitude and frequency of the micro-vibration are not a function of time, and they appear to be a random noise effect because they change immediately with changes in the internal and external situations of the optical alignment facility. Therefore, the effects of the micro-vibration applied to the EOS decrease when several ESF signals are averaged. The 13 raw ESFs were obtained from each edge image using the abovementioned method to generate the training data. MTF values were calculated using the 13 raw ESFs, sorted based on the obtained MTF values, and the mean for the middle 5 raw ESFs, excluding the top 4 and bottom 4 MTF values, was defined as proxy ground-truth ESF (PGT ESF).
Although PGT ESF was defined using only 5 raw ESFs obtained from different pixels, the original signal characteristics of the knife-edge target were sufficiently reflected because non-uniformity correction (NUC) was performed on each pixel output during the sensor's factory calibration phase [12]. The effect of micro-vibration was sufficiently excluded by calculating the average of 5 raw ESFs. The PGT ESF signal generated through the method described above is shown in Fig. 5. The training dataset for ESFNet consisted of thousands of pairs of raw ESFs extracted from images taken of knife-edge targets under different focus situations and PGT ESFs calculated using the abovementioned method.
2.2 Network design and training
Figure 6 shows a block diagram for training the proposed ESFNet using the PGT ESF dataset. ESF Raw extracted from the image in which the knife-edge target was taken as input to the ESFNet, and ESF Out was generated as an output. Subsequently, the loss was calculated using Eq. (1).
Here, N denotes the length of the sequence constituting the input raw ESF, yi denotes the i-th element of PGT ESF, xi denotes the i-th element of the input raw ESF, and function f denotes the operation by the ESFNet. The calculated loss was back-propagated to update the parameters of ESFNet. After the loss calculation was complete, ESF Out generated from the ESFNet and PGT ESF obtained from the training dataset generated LSF Out and LSF PGT, respectively, through differential operations. The subsequent steps performed FFT operations on these LSFs to generate MTF Out and MTF PGT, respectively. These steps were added to check the progress of the training and improve the convenience of use, and they were not involved in the training of ESFNet.
Figure 7 shows a detailed structure for ESFNet. Since the ESF used as an input in this study is 1-D data, the network was constructed by applying 1-D convolutional layers [13–15]. Each 1-D convolutional layer had a stride size of 1 and a zero-padding to keep the output sequence of the same length as the input. Additionally, its kernel size was set to 49 at the first layer and 25 at the rest so that the input raw ESF features could be sufficiently included. It was designed to sufficiently reflect the input raw ESF characteristics and prevent underfitting by adding one skip connection that moves the feature in the middle of the network to the output as it is. During the training phase, the mini-batch size was used as 1, and the learning rate started with 1e-4 and multiplied by 0.97 per 500 iterations to gradually become smaller. The total number of training dataset images taken of the knife-edge was 1980, and the losses converged after three epochs of training. A test dataset consisting of a total of 1080 images was prepared for performance evaluation of trained ESFNet, which was not used in the training process. The training dataset is configured to robustly suppress micro-vibrations to a sufficient degree when outliers are input during inference process by trained ESFNet. In other words, the training dataset included most of the usage conditions that could exist in the actual optical alignment ground facility, so that the ESFNet would not be overfitted. Therefore, training and test datasets were produced considering the temperature and humidity inside and outside the optical alignment ground facility, location, and movement direction of the overhead crane.
3. Experiments
Experiments were conducted to evaluate the ESF reconstruction performance of the proposed ESFNet. The experiments used ESFs with realistic simulated micro-vibration and ESFs obtained from actual EOS with real micro-vibration. To generate the ESFs with realistic simulated micro-vibration, 13 raw ESFs obtained using actual EOS at different times were normalized and averaged to create an ESF with minimal impact of micro-vibration. To simulate the influence of micro-vibration, random sampled real numbers within the range of [0, 0.03] were added. Figure 8 shows a generated ESF without the presence of micro-vibration, an ESF with realistic simulated micro-vibration noise, and how micro-vibration affects MTF values. Figure 9 illustrates ESFs obtained from actual EOS with real micro-vibrations in the test dataset to be used in the experiments. Figure 9(a) shows the effect of air conditioning system operation on ESF distortion, and Fig. 9(b) shows the effect of overhead crane operation on ESF distortion. Generally, noise caused by micro-vibration tends to increase the MTF value slightly as it is added to the original raw ESF signal as a high-frequency component. Few studies have been reported applying deep learning network-based ESF reconstruction techniques. Therefore, performance comparisons were conducted by applying the most widely used cubic smoothing spline and Fermi function-based curve fitting in the EOS industry.
Fig. 8. Generated ESF for the ESFNet performance evaluation: (a) ESF without the presence of micro-vibration and its LSF and MTF. (b) Realistic simulated micro-vibration noise-added ESF and its LSF and MTF.
Fig. 9. ESFs obtained from actual EOS with real micro-vibrations: (a) effect of air conditioning system operation on ESF distortion. (b) effect of overhead crane operation on ESF distortion
Figure 10 shows the reconstruction result of ESF for each algorithm when the realistic simulated micro-vibration noise-added ESF, shown in Fig. 8(b), is input. Figure 10(a) shows the reconstructed ESF when the input-generated ESF was curved-fitted using the Fermi function. The LSF and MTF were calculated using the reconstructed ESF. Figure 10(b) refers to the ESF, LSF, and MTF reconstructed when the input-generated ESF was reconstructed using the cubic smoothing spline. Finally, Fig. 10(c) shows the ESF reconstructed by the proposed ESFNet using the input-generated ESF and the LSF and MTF calculated using the reconstructed ESF. In each figure, the solid black line represents the ESF without the presence of micro-vibration noise, and its calculated LSF and MTF. The solid red line represents the ESF reconstructed by each algorithm, and their calculated LSF and MTF. The ESF reconstruction method using the Fermi function showed an underfitting phenomenon that did not sufficiently fit the generated ESF. As a result, the reconstructed MTF was measured lower than the original MTF value calculated using the ESF without the presence of micro-vibration. The ESF reconstruction method with the cubic smoothing spline showed an overfitting phenomenon in which distortion caused by added noise was reflected in the fitting curve. Accordingly, the MTF measured by the reconstructed ESF follows the MTF value measured by the generated ESF. In contrast, the ESF reconstruction method using the proposed ESFNet accurately expressed its unique ESF curve characteristics. Accordingly, when MTF was calculated with ESF reconstructed through ESFNet, the result was most similar to the MTF value by the ESF without the presence of micro-vibration.
Fig. 10. Reconstruction results of ESF for each algorithm when the generated ESF is input: (a) Reconstruction using Fermi function. (b) Reconstruction using Cubic smoothing spline. (c) Reconstruction using proposed ESFNet.
Table 1 shows the average MTF values calculated after measuring MTF five times through different algorithms in three fields of CCD image sensors. The added noise caused by micro-vibration produced an error of 0.01 at the MTF value. ESF reconstruction using the Fermi function produced an error of 0.044, much larger than 0.01, making it ineffective in the reconstruction of ESF distorted by micro-vibration. For ESF reconstruction by the cubic smoothing spline, an MTF error of 0.01 propagated because the distortion caused by the added noise remained in the output ESF fitting curve. Reconstructing noise-added ESF via ESFNet had the most similar MTF value to that calculated by using ESF without noise. Thus, ESF reconstruction using ESFNet can effectively control the distortion with an average MTF error of 0.001.
Table 1. Average MTF values calculated through different algorithms in three fields of CCD
Figure 11 shows the performance comparison of ESFs reconstructed when raw ESFs obtained in EOS alignment facilities, where micro-vibration exists, are input to the existing fitting algorithms and the proposed ESFNet. For ESF reconstruction using the cubic smoothing spline, an overfitting phenomenon followed the slope of the edge distorted by micro-vibration. Accordingly, the high-frequency component caused by micro-vibration can affect the MTF value.
Fig. 11. Reconstruction performance comparison for ESF with actual micro-vibration noise: (a) ESF reconstructed using the cubic smoothing spline. (b) ESF reconstructed using the Fermi function. (c) ESF reconstructed using the proposed ESFNet.
The reconstructed ESF curve using the Fermi function had the ideal shape of ESF. However, there was an underfitting phenomenon that did not sufficiently reflect the characteristics of the original ESF of the EOS to be measured. In contrast, for ESF reconstruction using the proposed ESFNet, the original characteristics of EOS were sufficiently reflected and the distortion part because of micro-vibration was not followed. Accordingly, the influence of micro-vibration can be excluded when using the proposed ESFNet.
Table 2 shows the average MTF values after 10 MTF measurements were performed in three CCD fields for five days in presence of actual micro-vibration. However, unlike experiments with generated ESFs, the MTF error cannot be calculated because the MTF values of the conditions where micro-vibration is not present are unknown when using ESFs obtained in actual alignment environments. Therefore, an MTF budget analysis was performed to estimate the system MTF. As shown in Table 3, the system MTF can be estimated by obtaining the MTFs and tolerances of each module analytically constituting the EOS and then multiplying them by a cascade process [16,17]. The analytical MTF value of the EOS was calculated as 0.0948. Based on this analytical MTF, the ESF reconstruction method using the proposed ESFNet was confirmed to have the lowest error of 0.0002. As ESFNet has sufficient robustness, distorted ESF can be effectively reconstructed even for outlier situations that may be encountered during actual optical alignment ground facility operations.
Table 2. Comparison of MTF reconstruction performance of each algorithm using actual ESF input affected by micro-vibration
Table 3. MTF budget analysis for estimating system MTF (@ Nyquist Freq.)
Through-focus MTF (TF-MTF) approach is widely used to find the exact focus position when aligning the EOS. The TF-MTF method moves the knife-edge target along the focus direction of the collimator over several steps, generates an edge image for each step, and takes it to measure the MTF. The MTFs measured along the focus direction over several steps draw a quadratic curve in the plane, and the peak point of this quadratic curve becomes the optimal focus position for alignment of the EOS. Figure 12 shows the quadratic curve of MTF obtained using the TF-MTF approach. The focus range was divided into nine steps, and MTF measurements were repeated twice in the forward and backward directions of each step. Figure 12(a) shows the result of TF-MTF applying ESF reconstruction using cubic smoothing spline. Since the influence of micro-vibration remained on the spline, the deviation of the MTF value measured at each step was large. Accordingly, the MTF quadratic curve was inaccurate. Figure 12(b) shows a TF-MTF quadratic curve using ESF reconstruction based on the Fermi function; the quadratic curve was more accurate as micro-vibration noise was removed. However, the MTF value measured for each step was distorted and lower because of the underfitting phenomenon. In contrast, for ESF reconstruction using the proposed ESFNet, the deviation between the MTF values measured at each focus step was the smallest, and the TF-MTF quadratic curve was precisely expressed. The system MTF value at the peak point of the quadratic curve also showed the smallest error with the MTF value derived by the analysis method.
Fig. 12. Through-focus MTF results with ESF reconstruction: (a) ESF reconstruction using cubic smoothing spline. (b) ESF reconstruction using Fermi function. (c) ESF reconstruction using the proposed ESFNet.
4. Conclusion
A deep-learning-based ESF reconstruction approach has been proposed to reduce the effects of the inevitable micro-vibration on MTF measurement of high-resolution EOS. A PGT ESF dataset was constructed for intuitive and practical network training, and the proposed network was trained with a supervised learning method using the PGT ESF dataset. Conventional ESF reconstruction algorithms have been difficult to effectively measure MTF because of their underfitting and overfitting to micro-vibration. The proposed ESFNet was able to prevent underfitting and overfitting, and it could effectively reconstruct distorted ESF by reflecting the characteristics of EOS. Experiments using simulated ESF showed that for ESF reconstruction using the proposed ESFNet, the MTF measurement error was 0.001, which was significantly lower than 0.01 with the cubic smoothing spline and 0.044 with the Fermi function. Experimental results using ESF obtained in facilities with real micro-vibration also showed that the MTF value was measured at 0.095 when ESF was reconstructed using the proposed ESFNet, which had the lowest measurement error with the system MTF of 0.0948 calculated using analytical methods. Nevertheless, these results must be interpreted carefully, and various limitations should be kept in mind. If there is a fixed-pattern micro-vibration source in the EOS alignment facility, PGT ESF cannot sufficiently exclude the effects of micro-vibration, which may lead to performance degradation of the proposed ESFNet. Further, ESFNet trained in a supervised learning method can show MTF values of different patterns when a micro-vibration source is added or removed.
Funding
National Research Foundation of Korea (NRF-2017M1A3A4A09093832).
Acknowledgments
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2017M1A3A4A09093832). The authors would like to thank Editage for English language editing.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time because the agency that supported this study requested confidentiality.
References
1. F. Viallefont-Robinet, D. Helder, R. Fraisse, A. Newbury, F. van den Bergh, D. Lee, and S. Saunier, “Comparison of MTF measurements using edge method: towards reference data set,” Opt. Express 26(26), 33625–33648 (2018). [CrossRef]
2. T. L. Williams, The Optical Transfer Function of Imaging Systems (Routledge, 2018).
3. Z. Zhou, F. Gao, H. Zhao, L. Zhang, L. Ren, Z. Li, M. U. Ghani, and H. Liu, “Monotone spline regression for accurate MTF measurement at low frequencies,” Opt. Express 22(19), 22446–22455 (2014). [CrossRef]
4. Z. Zhou, Q. Zhu, H. Zhao, L. Zhang, W. Ma, and F. Gao, “Techniques to Improve the Accuracy of Presampling MTF Measurement in Digital X-ray Imaging Based on Constrained Spline Regression,” IEEE Trans. Biomed. Eng. 61(4), 1339–1349 (2014). [CrossRef]
5. ISO 12233:2017, Photography—Electronic still picture imaging—Resolution and spatial frequency responses, ISO, 2017.
6. K. Jacobsen, “High resolution satellite imaging systems-an overview,” Photogrammetrie Fernerkundung Geoinformation 6, 487–496 (2005).
7. R. Bell Jr., S. B. Hutchinson, and S. Borota, “LOTIS facility successfully reaches initial operation capability,” in International Conference on Space Optics—ICSO 2010 (2019) Vol. 10565.
8. K. Masaoka, “Edge-based modulation transfer function measurement method using a variable oversampling ratio,” Opt. Express 29(23), 37628–37638 (2021). [CrossRef]
9. H. Li, C. Yan, and J. Shao, “Measurement of the Modulation Transfer Function of Infrared Imaging System by Modified Slant Edge Method,” J. Opt. Soc. Korea 20(3), 381–388 (2016). [CrossRef]
10. Tiecheng Li and Huajun Feng, “Comparison of different analytical edge spread function models for MTF calculation using curve-fitting,” MIPPR 2009: Remote Sensing and GIS Data Processing and Other Applications (2009) Vol. 7498.
11. H. Oh, S. Ahn, S. Lee, and A. C. Bovik, “Deep Visual Discomfort Predictor for Stereoscopic 3D Images,” IEEE Trans. on Image Process. 27(11), 5420–5432 (2018). [CrossRef]
12. S. Kim and Tae-Gyu Chang, “Nonuniformity correction scheme considering the effects of internal thermal stray light,” Optical Engineering 1, 013104 (2017). [CrossRef]
13. S. Kiranyaz, T. Ince, O. Abdeljaber, O. Avci, and M. Gabbouj, “1-D Convolutional Neural Networks for Signal Processing Applications,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019).
14. O. Abdeljaber, O. Avci, S. Kiranyaz, M. Gabbouj, and D. J. Inman, “Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks,” J. Sound Vib. 388, 154–170 (2017). [CrossRef]
15. S. Kiranyaz, T. Ince, and M. Gabbouj, “Real-time patient-specific ECG classification by 1-D convolutional neural networks,” IEEE Trans. Biomed. Eng. 63(3), 664–675 (2016). [CrossRef]
16. T. Choi and D. L. Helder, “Generic sensor modeling for modulation transfer function (MTF) estimation,” Pecora 16 (2005).
17. T. B. Andersen and Z. A. Granger, “Camera System MTF: combining optic with detector,” in Current Developments in Lens Design and Optical Engineering XVIII (2017) Vol. 10375.
