1. Introduction

A low level of oxygen in the blood, known as hypoxemia, can lead to hypoxia, resulting in medical complications and even death [1]. There are various causes of hypoxia, the most common of which are lung disease, birth defects, and chronic obstructive pulmonary disease [1].

A strange phenomenon, known as silent hypoxemia [2], is causing confusion among physicians treating COVID-19 patients. Physicians report that patients seem to be feeling well despite having dangerously low blood oxygen saturation levels, which would usually leave them unconscious, or even dead [3]. The mysterious situation defies basic biology and raises deep questions about how COVID-19 attacks the lungs and whether there can be more effective ways to diagnose and treat such patients. Today, the most common way of measuring blood oxygen saturation level is by using pulse oximetry [4], which requires direct contact with the patient’s finger.

This paper proposes a novel method for testing blood oxygen saturation levels without physical contact with the patient by using laser illumination, similar to the current non-contact body temperature measurement. The purpose of laser illumination is to sample micro-saccades oscillations in the patient's eye, and by using machine learning methods associate the oscillations detected from speckle pattern analysis with the patient's blood oxygen saturation level.

Micro-saccades are tiny movements of the eye sclera that are caused by biological and physiological traits in the human body and can even reflect their metrics [5,6]. In particular, tiny eye sclera movements can reflect the level of blood saturation and even short-term hypobaric hypoxia [7]. Micro-saccades can be detected using standard eye trackers; however, the method for relating information received from the eye tracker to blood saturation level is problematic.

Although the speckle-based remote sensing approach has been known and investigated for some time and used in various biomedical applications, we take the concept presented in other papers [8–11] to another level, in which tracking of random eye movements can allow remote sensing of oxygen level in the blood by combining speckle-based technology with our artificial intelligence algorithm.

2. Motivation and experimenting

The human sclera is not an optical quality surface and thus, when illuminated by a laser beam, the back scattered light forms secondary speckle patterns. These patterns can be imaged with a fast camera having defocused optics. Analysis of temporal changes in the spatial distribution of the random speckle patterns can be related to nano-movements in the illuminated surface (in this case the eye sclera) [8].

In our experiment, data was gathered from eight healthy human subjects, aged 22 to 73, in a controlled laboratory environment. The room in which the speckle measurements were taken was shuttered to prevent background noises. Each subject was seated 50 cm across from a sensor equipped with a camera recorder (Basler acA800-510um USB 3.0 camera), the aim being to record micro-saccades with as few disturbances as possible. The subject’s head was restrained in a headset equipped with protective gear for the purpose of directing the left eye to the sensor and mitigating involuntary head movements. The subject’s eye sclera was illuminated by a laser with an output power of 770 µW at a wavelength of 1550 nm (invisible eye safe laser). The subject’s finger was connected to the pulse oximeter (Nellcor OxiMax N-600 Pulse Oximeter) for blood oxygen saturation recordings. It is important to note that the oximeter has a SpO2 accuracy tolerance of ± 2 digits, or about 1%.

Each subject was recorded for two cases: i) normal blood oxygen saturation level; and ii) relatively low oxygen saturation level. The first case was measured with the subject breathing normally, and the second after a number of prolonged respiratory arrests. It should be noted that slow breathing was performed after extended cessation of breathing in order to enable inhalation of oxygen before the breath is held further, while following up indications on the oximeter. The normal blood oxygen saturation level threshold was set for a (BOS) blood oxygen saturation of ≥96%, and the oxygen saturation level was considered to be relatively low when the instrument showed a BOS of less than 96%. Figure 1 shows a subject participating in the data acquisition process.

 

Fig. 1. Experimental setup: (a) recording speckle patterns reflected from eye sclera of a person; (b) laboratory equipment and headset

Download Full Size | PDF

The defocused camera capturing the illuminated eye movements was controlled with MATLAB code, producing a video of 500 frames per second quality (FPS) for the duration of 10 seconds for each sample taken with 32×32 resolution. During the recording, the subject was asked to remain still.

It is important to note that though our sensor may be more advanced, our optics is relatively simple. If an alternative image processing concept were to be applied, expensive optical equipment with a large focal length would be needed to provide images with a sufficiently high spatial resolution at a large inspection distance of at least few meters. In our case, the capability of sensing precise eye movement is obtained at large distances without the need for high quality optics since we do not capture images of the eye but only analyze back reflected speckle patterns.

The original data itself contained about 100 videos (2GB), and about 2 million 32×32 size frames collected with the same hardware on different dates. Each subject's video contained a unique identification, including the subject's name, duration of measurement, and recorded level of BOS. In order to prevent data leakage as part of the preparations for the training process, the videos were divided into two sets before dividing them into frames. The data of six subjects was included in the training set while that of the remaining two subjects was included in the test set.

In the interest of data variation and with a view to avoiding the pitfall of ingraining in the deep learning model, we used biases that might be inherent in the experiment procedure itself. Data gathering was carried out on separate dates, with each subject being recorded several times in one continuous session [9].

3. Method

3.1. Data exploration and pre-processing

The data exploration and processing phase is an important step in the process of building a classifier, especially in the field of machine learning. In this study, our initial source of data was laboratory recorded videos of temporal variations in speckle patterns. We first separated the videos into two sets, one for the training phase and the other for the model testing phase (each video appears in one of the above two groups).

In the pre-processing data phase, the frames were extracted from each video according to its recording rate (FPS). The extracted frames were saved as PNG files, a format that preserves image quality by preventing loss of information [12].

In the data exploration phase, we attempted to find a correlation between each frame group belonging to the same section of each video. We used a 2-dimensional cross-correlational function for each of the two consecutive frames in order to detect the difference in their direction of motion and mimic the movement of the micro-saccades, as can be seen in Fig. 2(a).

 

Fig. 2. Calculation of micro-saccades movement correlation for 10 sec. recordings: (a) finding 2D correlation between two consecutive frames; (b) micro-saccades movement during normal oxygen levels in the blood; (c) micro-saccades movement during relatively low oxygen levels in the blood

Download Full Size | PDF

An ordinary image of the eye contains much irrelevant information, such as noise. In order to select pupil movements, complex image processing operations (e.g. cleaning noise, edges detection, etc.) had to be performed, involving a considerable amount of time for extracting the relevant content (due to the nature of the image). In contrast to the speckle signal image, shown in Fig. 2(a), the relevant information could be obtained after performing a 2-dimensional cross-correlation operation between successive frames. Each frame had a known recording time that corresponds to the time of the blood oxygen saturation value, measured using the oximeter. By parallelizing the times and the numeric output of the oximeter, we labeled each frame according to the settings as explained above.

A visual examination of the correlation between the tiny movements (micro-saccades) represented by the camera pixels, when the blood oxygen saturation level is normal, showed short, consistent and varied movements in space, as seen in Fig. 2(b). In contrast, when the blood oxygen saturation level was relatively low, as seen in Fig. 2(c), the body response was in the form of tiny movements that were more scattered and had a number of smaller deviations.

Our camera sensor translated the input to a pixel of 4.8 µm x 4.8 µm size. Therefore, the range of eye movements was about 4-5 pixels or 20-25 µm – see Figs. 2(b) and 3(c).

 

Fig. 3. Classifier architecture: (a) our deep learning model; (b) convolutional block with Conv2D layer, activation and pooling functions

Download Full Size | PDF

After analyzing the various data videos, we found significant differences in the pattern of the micro-saccades when the blood oxygen saturation level was normal and when it was relatively low. The differences in the speckle patterns suggested that this was a problem that a deep learning model could handle fairly easily. It was based on this hypothesis that we selected the algorithm to build the classification model.

3.2. Algorithm

In solving the problem and in constructing our classifier we used methodology and models from the field of deep learning. The rationale for choosing this approach stemmed from the nature of our data and the type of problem we were attempting to solve.

The speckle signal is expressed through image and deep neural networks and convolution-based neural networks. These are models of the deep learning field, having many advantages when dealing with this type of input, as explained in Ref. [8]. Also, biomedical indices in humans are not continuous over time, but are interrupted by a few noises. A deep learning model, unlike a simple statistical classifier, has one clear advantage, which is to learn repetitive patterns in the data, even if it is noisy.

We used a very small convnet [13], as can be seen in Fig. 3(a), with a few layers and filters per layer, alongside a data dropout function [14]. Dropout also helped to reduce overfitting [14] by learning a fraction of the weights in the network in each training iteration:

Equation (1) shows least square loss for a dropout network, where:

${I_i}$ is a single training input

$t$ is the true label for this input

${w_i}$ is the weight of the input

$\partial $ is the dropout rate, where $\partial $ ∼ Bernoulli (p). This means that $\partial $ is equal to 1 with probability p and 0 otherwise.

Our model architecture included a simple stack of three 2D convolution layers [14] with ReLU activation [15] that speeded up the training process in our CNN model by simplifying the gradient computation process:

Equation (2) represents a ReLU function, being either 0 or 1 depending on the sign of x, while x is our input.

The computational step of a ReLU function is also easy: any negative inputs are set to 0, which means we have no exponentials, and no multiplication or division operations in our activation functions.

The ReLU function is followed by a max-pooling [16] layer for variance and reduction of computation complexity and for extracting low level features from neighborhood pixels [16]. Max-pooling extracts important features, such as pixel edges, for identifying a change in micro-saccades movement.

On top of our model, we attached two fully connected layers [13], as can be seen in Fig. 3(a). We ended the model with a single unit and sigmoid activation [17], which is appropriate for binary classification and reflects our problem definition – classification of normal and relatively low blood oxygen saturation level. We also used the binary cross-entropy loss function [18] and RMSprop optimizer [19,20] to train our model:

Equation (3) represents the binary cross-entropy loss function, where:

y is the label

p(y) is the predicted probability of the input being the “relatively low oxygen level” case for all N points

log(p(y)) is the log probability of y being the “relatively low oxygen level” case

log(1-p(y)) is the log probability of y being the “normal oxygen level” case.

Equation (4) represents the root mean square (RMSprop) of the updated math equations where:

E[g] is the moving average of squared gradients

δC/δw is the gradient of the cost function with respect to the weight

$\mu $ is the learning rate

$\beta $ is the moving average parameter

4. Experimental results

The trained model based on the six tested subjects was evaluated on the test set containing samples from two remaining subjects being tested but excluded from the training process. The best model achieved with our data reached an accuracy of 68% on the test set frame classification task, as can be seen in Fig. 4(a), and an accuracy of 100% on the test set of the video classification task.

 

Fig. 4. Model training graph and results: (a) model results on test set; (b) confusion matrix of our best model and the test set; (c) ML oxygen level saturation predictions on two subjects’ test data; (d) model training process graph including training and validation cost functions and their accuracy

Download Full Size | PDF

When examining the model's success for each frame, we treated each frame individually, which means that its score does not affect the other frames that come before or after it or even belong to the same recorded video. The confusion matrix of the classification results on our frames test set is presented in Fig. 4(b).

On testing the model's success for each recorded video, we predicted the oxygen saturation level of each subject's video frames [Fig. 4(c)]. For each video we grouped the model predictions on its frames, divided them into two columns, one for each case, and checked if most of the frames were classified correctly according to the video's label corresponding to the blood oxygen saturation level recordings. Using this method, we were able to isolate model noise and treat each recording video as an input to our model. This reflected the solution we wanted to reach and represented the introduced remote blood oxygen saturation level test for a patient.

Based on the results of our successful model it can be deduced that the difference in micro-saccades motility between the two oxygen saturation classes (labels) is a very important feature for our model – especially for the 2D convolution layer with its specialized cross correlation – having the highest impact on model output since it is the one that learns the patterns of the different inputs.

Figure 4(a) shows our model performance using well-known metrics for testing the success of models in the field of deep learning in particular [21]. In addition, a low level of false positive and false negative rates may be observed, reflecting the reliability of the model and its handling of the various data noises. The two rightmost samples in Fig. 4(c) show model predictions on the frames of two recording videos for one of the subjects with normal oxygen saturation level. A very small difference between normal and relatively low oxygen saturation columns for the two subjects can be discerned and can be explained by two possible arguments: (i) the amount of data for the relatively low case is smaller than for the normal case because of the challenge of intentionally lowering blood oxygen saturation levels (without illness). This claim can be refuted by the fact that during the training process we weighted the classes in the loss function according to the amount of data in the training set; (ii) the physiological reaction resulted in the appearance of micro-saccades during stress in a subject, caused by a relatively low blood oxygen saturation level. The unique physiological behavior of the micro-saccades when the blood oxygen saturation level is a relatively low is dominant but inconsistent throughout the measurement [7], meaning that a random sample of the micro-saccades may look as if the blood oxygen saturation level is normal. Since our model treats each frame individually, this physiological behavior will be classified as normative motion of the micro-saccades and will skew the model results.

The second argument appears to be more plausible but remains hypothetical since the ways of deep learning are not fully studied. If there was a definitive way to separate the two cases, we would not use this model and could build a fairly iterative algorithm instead.

Figure 4(d) shows the duration of our model training process. On analyzing the test accuracy curve reaching the model’s training saturation at a relatively fast stage of the training process, it can be seen that it levels off in the first epoch. The fact that the training accuracy curve continued to climb while the test set accuracy curve reached a plateau shows that a higher accuracy can be achieved in the model by a number of possible actions: improving the model with Long Short-Term Memory layers, adding convolution layers, enriching the dataset, etc.

It is also important to note that the training process was performed on expensive computing resources, and in order to save on resources by maintaining a greener environment training was stopped after we achieved satisfactory results in the course of training the model and when the test-set accuracy curve passed a certain pre-set threshold.

5. Conclusions

This paper presents further development of a novel approach linking micro-saccades with a reduced blood oxygen saturation level, which can lead to hypoxia. The experimental validation, based on remote sensing and analyses of anomalies in micro-saccades found on the recorded speckle patterns, were for the first time demonstrated on a small, predefined number of cases and used to relate those anomalies to changes in the oxygen level in the blood stream.

The system consists of a laser illuminating eye sclera, a defocused infrared digital camera with simple optics capturing speckle patterns, and a computer equipped with MATLAB and ML algorithms.

Normal and relatively low blood oxygen levels were tested in eight healthy subjects, aged 22 to 73, with normal breathing and by restricting the breathing rate to a level when the blood saturation reading dropped below 96%.

Micro-saccades anomalies were analyzed based on the dynamic speckles, showing highly accurate performance in separating between the different oxygen saturation cases. Common to all participants, the input data showed a reduction in hypoxia validation, but still gave a clear indication of the physiological condition. Therefore, the tests should be extended to include subjects with silent hypoxemia and updated. Due to the fact that micro-saccades could also indicate the existence of complementary diseases, different groups of participants should be tested and compared. This will allow the effect of a stress condition occurring during breath restriction to be eliminated.

This method is applicable to medical institutions and locations (e.g. shopping centers, educational institutions, etc.) where medical metrics, such as body temperature, are tested during the COVID-19 pandemic, when avoidance of direct contact is required. Our method could be combined with a remote temperature measurement device, offering a more reliable module for early tracking of COVID-19 patients.

The clear difference between normal and relatively low blood oxygen saturation levels shown in this research presents an opening for a future higher resolution research study, in which it can be subdivided into additional cases corresponding to numerical grading, as the shown on the reference pulse oximeter device. The full code employed in our research can be found in Ref. [22]. In addition, one can run customized training using individually collected data and with the right configuration and hyper-parameters.