Abstract

Storage, processing, and transfer of huge matrices are becoming challenging tasks in the process analytical technology and scientific research. Matrix compression can solve these problems successfully. We developed a novel compression method of spectral data matrix based on its low-rank approximation and the fast Fourier transform of the singular vectors. This method differs from the known ones in that it does not require restoring the low-rank approximated matrix for further Fourier processing. Therefore, the compression ratio increases. A compromise between the losses of the accuracy of the data matrix restoring and the compression ratio was achieved by selecting the processing parameters. The method was applied to multivariate chemometrics analysis of the cow milk for determining fat and protein content using two data matrices (the file sizes were 5.7 and 12.0 MB) restored from their compressed form. The corresponding compression ratios were about 52 and 114, while the loss of accuracy of the analysis was less than 1% compared with processing of the non-compressed matrix. A huge, simulated matrix, compressed from 400 MB to 1.9 MB, was successfully used for multivariate calibration and segment cross-validation. The data set simulated a large matrix of 10 000 low-noise infrared spectra, measured in the range 4000–400 cm⁻¹ with a resolution of 0.5 cm⁻¹. The corresponding file was compressed from 262.8 MB to 19.8 MB. The discrepancies between original and restored spectra were less than the standard deviation of the noise. The method developed in the article clearly demonstrated its potential for future applications to chemometrics-enhanced spectrometric analysis with limited options of memory size and data transfer rate. The algorithm used the standard routines of Matlab software.

© 2021 The Author(s)

Fourier-transform method of data compression and temporal fringe pattern analysis

Tuck Wah Ng and Kar Tien Ang
Appl. Opt. 44(33) 7043-7049 (2005)

Denoising method for a lidar bathymetry system based on a low-rank recovery of non-local data structures

Bin Hu, Yiqiang Zhao, Rui Chen, Qiang Liu, Pinquan Wang, and Qi Zhang
Appl. Opt. 61(1) 69-76 (2022)

GPU-accelerated non-uniform fast Fourier transform-based compressive sensing spectral domain optical coherence tomography

Daguang Xu, Yong Huang, and Jin U. Kang
Opt. Express 22(12) 14871-14884 (2014)

Joint sparse and low rank recovery algorithm for compressive hyperspectral imaging

Tatiana Gelvez, Hoover Rueda, and Henry Arguello
Appl. Opt. 56(24) 6785-6795 (2017)

Compressed FTIR spectroscopy using low-rank matrix reconstruction

Manuel Marschall, Andrea Hornemann, Gerd Wübbeler, Arne Hoehl, Eckart Rühl, Bernd Kästner, and Clemens Elster
Opt. Express 28(26) 38762-38772 (2020)