1. Introduction

Since the emergence of COVID-19, hundreds of guidelines have been published by the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC) [1,2]. The Interim Guidance for United States Healthcare Facilities recommends aggressive universal source control measures and well-equipped triage procedures at closed area entrances to screen individuals for fever. Fever is defined as either measured temperature greater than or equal to 100° F (∼38°C), or subjective fever and symptoms. Respiratory symptoms consistent with COVID-19 are cough, shortness of breath, and sore throat [1]. Screening for the detection of febrile persons entering closed facilities remains problematic, particularly when paired with CDC and WHO physical distancing guidelines. The CDC has advised maintaining spatial separation of 6 ft. (∼1.83 meters), and the WHO recommended keeping a distance of at least 1 m from other persons [1–3]. As a result, noncontact infrared thermometers (NCITs) have been identified as the most practical solution considering their short response time, intrinsic simplicity, and safety for humans.

Despite their widespread use, the reliability of infrared-based remote temperature measurement has been the subject of many debates. Various studies have been done to examine the reliability of NCIT instruments and factors that affect their accuracy [4–7]. The accuracy of the temperature measured by NCITs can be affected by many factors, such as sensor’s hardware (Δsensor) or software (Δalgorithm) inaccuracy, environmental conditions (Δenvironment), or a subject’s features (Δsubject) and measurement setup (Δuser), like measurement distance and angle [6,7].

Numerous studies have been conducted to isolate each factor and test its influence on the accuracy of the NCITs measurements. Stacey J. L. Sullivan et al. [7], showed significant differences in measurements between six NCIT models in a control experimental environment (results not affected by Δuser, Δenviromental). The study showed bias ranged from just under − 0.9°C (under-reporting) to just over 0.2°C (over-reporting) with respect to the accurate reference temperature of each subject. J. Spindel et al. [8] found a statistically significant correlation between human body temperature and the outside air temperature, relative humidity, and wind velocity, neglecting the effect of the Δuser, Δsensor, Δalgorithm. In their study, 4,430 humans were screened at different entry booths that were each close in distance to the outside. The mean body temperature measurements were higher by 0.34°C [P < .001, CI 0.31-0.38] in the control group that was placed in a controlled temperature environment far from the outside.

S. Khan et al. [9] used a controlled experimental hospital environment to test the influence of subjects’ features on the NCIT measurements (Δsubject). The study compares the temperature mean differences (the NCIT vs the reference temperature of each subject) in relation to the participants’ characteristics. It shows that female participants had a higher mean difference in temperature (0.32°C) compared to male participants (0.21°C). Also, participants with light skin color had a higher mean difference in temperature (0.27°C) compared to participants with medium and darker skin colors (0.12°C). A. S. Hussain et al. [10] conducted a study to examine the effect that distance has on the NCITs’ measurement accuracy (Δuser, distance) in a controlled experimental environment. Of 51 healthy, non-febrile healthcare workers surveyed, the mean reference temporal artery temperature was 98.4°F (95% confidence interval (CI) = [98.2,98.6] °F). While the mean NCIT temperatures measured from 1ft, 3ft, and 6ft distances were 92.2 °F (95% CI = [91.8 92.67] °F), 91.3 F0 (95% CI = [90.8 91.8] °F), and 89.6 °F (95% CI = [89.2 90.1] °F), respectively. From statistical analysis, the only method in sufficient agreement with the reference standard was NCIT at 1 ft. F.

Piccinini et al. [11] conducted similar research and reached the same conclusions. In their study, four NCITs models were tested on three subjects. During the experiment, measurements were taken from different distances ranging from 1 cm to 10 cm. With the thermometer maintained perpendicular to the forehead, the distance was measured using a goniometer. Five measurements were acquired for each device in ten different positions, revealing that a difference of merely a few centimeters yields large variations in body temperature measurement. The two aforementioned studies reveal one of the prominent weaknesses of common NCIT devices measurement accuracy, decreased significantly with increased distance. The field of view (FOV) feature of the NCITs can explain this phenomenon: every NCIT instrument has a FOV, an angle of vision at which it will average all the temperatures it observes.

Depending on the distance from the sensor to the object, the camera could cover only the object’s temperature completely (in case of close distance), or record a wider view where the object covers only a part of the field. In this case, the measured temperature will contain false data, confusing the result. This paper aims to prove the feasibility of an accurate temperature measurement system based on speckle pattern analysis. This method is widely used for the remote sensing of biomedical parameters [12,13] and enables the detection and measurement of human temperature from an extended distance compared to existing methods. The measurement distance in our experimental setup was 50 cm (∼1.65 ft.), which is already greater than the 1 ft. maximum reliable distance of the NCITs measurement tested in the aforementioned studies [10,11]. However, due to the specifics of speckle pattern analysis, it could be invariant to the measuring distance.

To the best of the authors’ knowledge, no technology exists for remote temperature measurement based on optical sensing. A preliminary evaluation of the viability and feasibility of creating this novel technology is provided by the research described below.

2. Theoretical background

2.1 Body thermoregulation

When a human body is said to be in a thermal “steady state”, both superficial and deep body temperatures are stable and undergo slight temporal variations. This implies that the physical removal of heat by the environment is compensating for the heat produced by the body. Diseases can disrupt the body’s equilibrium, causing it to increase or decrease in temperature (hyperthermia or hypothermia).

To prevent its temperature from deviating, the body triggers corrective (or regulatory) mechanisms. These feedback mechanisms are called “thermoregulation,” a process in which the human body maintains its core internal temperature [14]. The mechanism that the body activates is a vascular reaction mechanism: I. In the case of potential hypothermia, when the environment removes more heat than the heat produced by the body, the physiological reaction is vasoconstriction of the skin vessels, which reduces cutaneous blood flow. This leads to a lower skin temperature, allowing better heat transfer with the cooler environment. II. In the case of potential hyperthermia, when the body ambiance removes less heat than the heat produced by the body, the physiological reaction is vasodilation of the skin vessels, which triggers increased blood flow in the skin. This increases skin temperature, allowing for better heat exchange with the hotter ambiance. A simplified formulation of the thermal steady state can be expressed as follows:

2.2 Speckle patterns analysis

Speckles have come into prominence since the invention of lasers and are used in a variety of applications: microscopy, imaging, and optical manipulations [15]. The speckles are random patterns produced by self-interference of wave fronts with the same wavelength, having different phases and amplitudes. Speckle patterns are produced due to the roughness of the area illuminated by a spatially coherent spot of a laser beam. When added together on the CMOS detector plane, the wave fronts provide a random intensity pattern due to the interference phenomenon, resulting in speckle patterns known as secondary speckles [13]. Z. Zalevsky et al. [16] presented a novel technique for measuring the vibrations of remote objects: the system configuration requires not to focus the camera on the object, but rather to have the camera focused on the far or close field so that the object itself is defocused. This makes the movement of the object (its vibrations) cause a lateral shift of the speckle pattern. In fact, instead of constantly changing the speckle pattern, the object only moves or vibrates along the transversal plane due to the defocusing. This is a very important feature since it allows extraction of the trajectory movement by tracking the maxima intensity spots. The object’s tilt movement is expressed as follows:

Using the same configuration, we illuminated the subject's forehead tissues with a laser beam. When projected on skin or other biological tissue, such coherent illumination may result in a speckle pattern due to the reflection and scattering of the beam inside the tissue, exposing the sub-skin scattering properties. The speckle pattern analysis may be applied for exploring temporal and spatial properties of the tissue. In temperature measurement, the optical scattering and transmission characteristics may vary when the temperature of the scattering body is changing due to variations in the blood and mass transfer within the skin. For example, velocity contents of scatters’ movement are highly related to the skin temperature via statistics of Brownian motion, causing image blurring that is also correlated with the skin temperature. In addition, tissue scattering or absorption may also cause a measurable fluctuation in the characteristics of a speckle pattern, speckle size, distribution, etc. [17] Combining and analyzing these easily detected parameters may form a novel mechanism for not only remote, but also distant human temperature measurements. The traditional methods of speckle pattern analysis usually use 1D cross-correlation signal analysis, which leads to a partial loss of the information existing in the 2D speckle pattern images. The introduction of AI methods can allow the direct processing of speckle pattern recordings.

2.3 AI speckle pattern analysis

Extraction and comprehensive analysis of the parameters affecting the body temperature by speckle pattern analysis requires the intensive processing of each recorded speckle pattern extracted separately from the image source. This led us to research the neural network (NN) field and to propose a NN that extracts the best feature vectors from each recorded speckle pattern. The final model we used, utilize 2 types of NN architectures: Convolutional NN and Recurrent NN. Each series of speckle patterns (the input to the model is a series of images) will be represented as features vector extracted by the model. Then, a temperature classification decision is made based on the extracted features vector.

2.3.1 Convolutional neural networks

Convolutional neural network(CNN) are one of the most well-known and usable architectures for performing representation learning on an image without minimal preprocessing. The strength of CNN architectures lies in the “convolution layer”, where the neurons form a two-dimensional filter applied across the input data. By selecting an effective training method and a loss function, each convolution layer extracts unique visual features that represent the image in terms of the problem that the architecture designer tells [18]

Other layers used in the CNN architectures are “down sampling” layers (like the “max-pooling layer”), “non-linear” layers (like “RelU” [19]) and “fully connected” layers, which complete the representation learning of the image and usually create a single feature vector of the image. The last layer of those architectures is usually the “decision layer”. This layer determines the classification or regression value according to the features of the image being fed to the layer from the entire network.

In the model proposed below, the role of the CNN network is to extract features from the image space (spatial dimension) from each individual speckle pattern in an input speckle pattern’s sequence.

2.3.2 NN—recurrent neural network

Our starting hypothesis suggests that body temperature affects the sequence of the recorded speckle patterns that are back-scattered from the human forehead. Dilatation or contraction of blood vessels during body temperature regulation gets translated into the variation of the recorded sequence of speckle patterns, which we would like to translate into a features vector for future temperature classification. In order to extract features that measure the change in the speckle patterns, the sequence of patterns should be treated as a time series, and each series of images should be represented by features reflecting the time dimension and not only the spatial dimension of each speckle image separately. Therefore, the model we built contains Recurrent Neural Network (RNN) architecture, which is designed to make a representation that also considers the time dimension (“the sequence of events”) [20–22].

The most common standard neural network type is feed-forward network, where set of neurons are organized in layers: input layer, output layer, and at least one intermediate hidden layer. Feed-forward neural networks are limited to static classification tasks. Therefore, they are limited to providing a static mapping between input and output. To model time prediction tasks we need a so-called dynamic classifier. RNNs extend feed-forward neural networks toward dynamic classification. To gain this property, RNN feed signals from previous “timestamps” back into the network. Below is the output formulation of a single neuron unit in a RNN architecture:

The current hidden state ${h^{(t )}}$ (the output for the current sequence from timestamp 0 to timestamp t) is a function f of the previous hidden state ${h^{({t - 1} )}}$ and the current input ${x^{(t )}}$. $\theta $ are parameters of the function $f$

2.3.2.1 GRU—gated recurrent unit

Recurrent NNs have faulty short-term memory. If a sequence is long enough, simple RNNs struggle to carry information from earlier time steps to later ones. This problem is known as the “vanishing gradient” problem [23]. GRUs were created as the solution to this short-term memory flaw. They have internal mechanisms called “gates” that can regulate the flow of information. These gates can learn which data in a sequence is important to keep and which to reject. This enables the transfer of relevant information further through a long chain of sequences to make predictions. Our final model contains an RNN architecture with GRU units. Here is a simplified single GRU unit formulation:

The current hidden state ${h^{(t )}}$ (the output for the current sequence from timestamp 0 to timestamp t) is a linear interpolation of the previous hidden state ${h^{({t - 1} )}}$ and ${\tilde{h}^{(t )}}$. ${\tilde{h}^{(t )}}$ is the memory content of the current hidden state dependent on the current input (till a moment t) and the previous hidden state ${h^{({t - 1} )}}$. ${Z^{(t )}}$ is the “update gate” that decides how the unit updates its content [24]

3. Methods

3.1 Optical measurements and data acquisition

The setup for human skin temperature measurement consists of several key components:

A block diagram of the experimental setup is presented in Fig. 1.

 

Fig. 1. Experimental setup for temperature measurement.

Download Full Size | PDF

The data for human temperature measurement was collected from five subjects in a controlled lab environment. Each subject was tested by the following experimental procedure: the laboratory was darkened to minimize background noise and each subject was seated 50 cm across from a sensor equipped with a camera recorder. Since the goal was to record the forehead with as few disturbances as possible, the subject’s head was placed on a chin stand equipped with protective padding, facing the sensor with their head movement restricted. In order to simulate a fever scenario in a varied temperature range of 37° C-40° C, the subject’s forehead was heated by a hair dryer. For normal temperature measurements of ∼36°C, the subject’s forehead remained unheated. After performing the selected heating scenario, the subject was illuminated by a green laser. The setup of one subject participating in the preliminary data acquisition is shown in Fig. 2.

 

Fig. 2. Setup of the system with its optics and the laser illuminating the subject’s forehead.

Download Full Size | PDF

When a fever is simulated, the forehead temperature rapidly declines following the termination of heat. To label the frames of each recording with the corresponding temperature label, we designed and assembled a syncing circuit. The synchronizing mechanism work as follows: The computer used MATLAB to send a command to the pulse generator to produce an electrical pulse every 6 seconds for the duration of the 120 seconds recording. Each pulse (20 in total) triggered the IRT gun which measured the forehead. Simultaneously, the computer activated the camera to record the speckle patterns. The sampling periods and the reference temperatures were assigned to the corresponding frames of the video recordings.

The camera, controlled through MATLAB, produced a video of 1000 frames per second quality, for 120 seconds. Each sample was taken in 64X64 pixel resolution. During the recording, the subject remained still. In the interest of varying the data and avoiding a pitfall of ingraining in the deep learning model, we used biases that might be inherent in the experiment procedure itself. The data gathering was carried out on separate dates and at different hours, whereas in one continuous session each subject was recorded in different heating scenarios multiple times.

At the final phase, each video, was cut down to 64 image-long sequences labeled in 1° C resolution – [ 36°-37°, 37°-38°, 38°-39°, 39°-40°]. The resolution that was selected enables disregarding the reference IRT device's inaccuracy (<= 0.5° C). Additionally, we decided to take only samples recorded within two consecutive reference temperatures from the same class in order to allow as many valid and consistent observations as feasible. Due to the brief and fragile nature of these samples, samples from temperatures greater than 40° C were deleted in order to prevent an imbalance in the data collection. The different sequences contained 0 to 62 images overlapping with other sequences from the same video.

3.2 Sequenced convolutional-GRU model architecture

Following the data acquisition, we built a GRU-convolutional neural network architecture. The combination of the convolution layers and the GRU layer in the proposed architecture allowed the extraction of two types of features: features in the “image dimension” (features from each speckle pattern separately), and features in the “time dimension” (features between and across the input speckle pattern sequence’s). First, each image from the sequence of each sample passes through a CNN architecture, creating a separate feature vector for each pattern. Then, the features vectors enter a bidirectional GRU-based RNN architecture, extracting features in the temporal dimension from all the sequences together. Hence, the GRU’s single output features vector contains two types of features representing the subject's temperature. At the final stage of the model, we concatenated some fully connected layers eventually down sampling the feature vector to a 1xN probabilities vector. Where N is the number of classes (temperature resolution) in our case N = 4. Each cell of the final vector corresponds to a single temperature label. The value of each cell represents the probability of the input vector belonging to the corresponding temperature label. The model’s block diagram is presented in Fig. 3.

 

Fig. 3. Convolutional – GRU based recurrent neural network model architecture.

Download Full Size | PDF

For each tested subject we created separate train and test sets. To prevent overfitting of the model to the training data, the training set and the test set were divided to contain sequences of frames from different videos without overlap.

4. Results and discussion

The proposed model was examined in two steps. The model was trained and tested on each subject separately, and afterwards on the samples related to all subjects.

4.1 Model verification for a particular subject

Verification of the model on the test data recorded from one subject, under one-degree temperature classification resolution, yielded an accuracy of ∼61%. The following is a confusion matrix of the model after applying a naïve “max probability” decision on its output vectors.

The confusion matrix (Fig. 4) allows to conclude:

 

Fig. 4. "Max probability” confusion matrix for subject #1.

Download Full Size | PDF

 

Fig. 5. Subject #1 ROC Curves plot.

Download Full Size | PDF

The presented confusion matrix accuracy (the average of the matrix’s diagonal) uses a naive decision to “take the maximum” probability without tuning decision thresholds. The model provides 4 probabilities (4 × 1 output vector) whose sum is equal to one. Each output component represents the probability that the examined image sequence belongs to the corresponding temperature label of the four classes {0, 1, 2, 3}: {36°, 37^o, 38^o, 39^o}. The maximum probability temperature is chosen to be the input sample’s temperature. Changing the decision thresholds without relying on the maximum only can further improve the ratio between the true positive rate and the false positive rate. The following is a ROC graph, representing the ratio of the rates for different decision thresholds:

With respect to the ROC micro and macro averages, Fig. 5 shows that the area under the curve reaches 0.85 out of 1, indicating that the right threshold adjustment allows to get a true positive- false positive ratio of 85%. Moreover, our model receives input of 64 speckle pattern images representing a very short sample of 0.064 sec. recorded under 1000 fps. A sample that is one-second-long could be divided into ∼15 short samples, which would enable the possibility to classify and decide by a majority of votes and to further improve the accuracy.

The diagonal of the confusion matrix above shows that the model is mainly confused with adjacent temperatures near the diagonal. For 37° samples, most of the confusion occurred between classes 37° and 38°. For 38° samples, most of the confusion occurred between classes 37° and 38°; and for the 39° samples, most of the confusion occured between class 39° and class 38°. After applying better thresholds, the true positive – false positive ratio of the highest class (39°) got the highest score of 0.83 among the high fever classes. This phenomenon can be attributed to the instantaneous high temperature and the simulation noise:

Figure 6 shows the declining slope of the temperature drop during 80 sec. recording. For class #3 ($40 < T \le 39.5$, brown arrow) the temperature drop is maximal. The slope of class #2 ($39 < T \le 38.5$, orange arrow) is lower and the slope of class #1 ($38 < T \le 37.5$, pink arrow) is the lowest. We assume that the model identifies the high simulation noise (decline rate) of the highest fever scenario and uses it as a feature of this class in order to classify with higher accuracy, compared to the other fever scenario classes. Table 1 shows a statistical summary of the temperature drop rate per temperature’s class:

 

Fig. 6. High fever scenario video samples – temporal temperature drop for subject #1.

Download Full Size | PDF

 

Fig. 7. a. ROC Curves plot for subject #2; b. ROC Curves plot for subject #4; c. ROC Curves plot for subject #3; d. ROC Curves plot for subject #5.

Download Full Size | PDF

Table 1. Temperature drop rate statistical summary

View Table | View all tables in this article

It can be seen that the statistical summary corresponds to the slope eye test. Figure 7 contains the ROC curves for the remaining four tested subjects.

Summary of the temperature classification is presented in Table 2.

Table 2. Summary of the classification results

View Table | View all tables in this article

In all the values obtained in the table, the precision is bounded by 5%.

The summarized results led us to the following conclusions:

4.2 Multi-subject temperature measurement model

After testing the model for each subject separately, we trained the model using the data collected from four subjects and tested it on the fifth subject. The results show partial inconsistent with the single subject results. The confusion matrix and ROC curve of the model on the tested subject’s samples are presented in Figs. 8 and 9.

 

Fig. 8. Multi subject model's confusion matrix.

Download Full Size | PDF

 

Fig. 9. Multi subject model's - ROC curves.

Download Full Size | PDF

Figure 8 shows that the multi-subject model succeeded in preserving the “screening” property of the single-subject models, as the model successfully distinguished between normal body temperature and those of fever. Additionally, the ROC graph shows a micro and macro average of 83%, similar to the results of the models trained on a single-subject.

However, unlike the single's models, under the high temperatures (fever scenarios), the multi-model did not identify the highest level (39°) better than the other temperatures (37°, 38°). It seems that the model tends to give a “middle “ score of 38° to the test samples that it identifies as high-temperature samples. This tendency is evidently due to the confusion matrix, where nearly all samples from the fever scenarios received the prediction of 38°. The ROC graph also displays that the 38° class receives the highest true positive – false positive ratio among the fever scenarios degrees.

This inconsistency has a positive side: We assume that the simulation noise variation in the train set caused the model to learn how to ignore the “simulation noise” and to exclude it from the features of the class 39°. However, further research is required because it is hard to explain why class 38° received a better classification than the other high-temperature classes.

5. Conclusion and future work

In this study, we demonstrated the feasibility of remote measurement of human body temperature using an optical sensing system. The sensor illuminates the subject's forehead tissue with a laser beam, producing speckle patterns and reflecting the sub-skin scattering properties. The recorded speckle pattern videos were subdivided into small sequences and fed into the hybrid convolution-recurrent neural network-based model. When trained on a number of subjects, the proposed model presented an accuracy of >99% for the separation between normal skin temperature of ∼36° C and skin temperature higher than 38° C.

These findings demonstrate the viability of developing a remote optical sensing system that, in comparison to existing IRT devices, will permit a large increase in measurement distance while preserving the same degree of precision. This was displayed in the experimental setup, where we sampled the subjects using the optical system from a distance of 50cm, whereas the IRT gun sampled it from 5cm. The primary advantage of the speckle-based temperature measurement system relates to the spatial invariance.

We trained the model to predict the subject's temperature with a resolution of 1°C. The model was trained in two phases: in the first phase, it was trained each time on a single subject and presented a mean true positive – false positive ratio of over 80%, and in the second phase, the model was trained on the data from 4 out of the 5 subjects, then tested on the fifth subject. The model showed an 83% true positive – false positive ratio.

Using an ordinary, tailored model with no prior knowledge of speckle properties, we achieved promising results. Throughout the study, we diagnosed several factors affecting temperature prediction, primarily as a result of the simulation noise that came from the artificial forehead heating procedure and its instability. Future work may include testing an extended and representative number of participants with normal body temperatures and with varying temperatures of disease-related fever. This will enable the elimination of the noise and the temperature decay caused by the fever simulation

These tests were conducted to evaluate the possibility of general human-body temperature measurement. However, in this research, we trained and tested our model on a small controlled sample size. A generalized single model that will work with high accuracy on a varying number of subjects will require larger and costlier resources. Future work in this direction must include massive data collection, and the new data set must contain many samples from each temperature range. To avoid overfitting, the data set should also contain samples from a large population with varying characteristics: age, skin color, gender, and more. Collecting such datasets can unlock more research directions such as higher temperature resolution measurement (< 1° C) under a more precise reference device.

Additional research without focusing on the forehead and on a long distance, more than 1m away, could be done in the future. Temperature measurement of a moving subject and in an uncontrolled dark experimental settings is also a further important goal.