The Emotional Expression Response of a Patient Based on their Facial Expression<br/>—Focus on Music Stimuli

Objective: To determine whether patients with unresponsive wakefulness syndrome respond to auditory stimuli based on changes in facial expressions. Participants: Six patients diagnosed with disorders of consciousness due to brain injury with permission for participation from their doctor. Design: In this hypothesis-driven observational study, the facial expressions of patients with unresponsive wakefulness syndrome were video-recorded for 5 min before and during auditory stimulation in three consecutive weekly sessions. Main Measures: Facial muscle movement was quantified using FaceReader ^&#174; software (Noldus, Wageningen, Netherlands). Valence/action unit values were plotted to detail facial expression changes. Heart rate values were also plotted. These parameters were compared before and after stimulation. Results: No significant differences in valence integral values or average heart rate were observed between the pre- and intra-stimulus conditions. However, valence signals increased in approximately half of the sessions, indicating that some patients with unresponsive wakefulness syndrome may exhibit emotional responses to auditory stimuli. Analysis of action unit integral values indicated that movement of the eyebrows and eyelids on the upper part of the face occurred during auditory stimulation. Furthermore, the types of auditory stimuli differed depending on the session for the 12 sessions of voice stimuli, whereas the changes in average heart rate differed in each of nine sessions of music stimuli. Because the changes in average heart rate were similar, it is possible that musical stimuli are more suitable than voice stimuli. Conclusion: Some patients with unresponsive wakefulness syndrome may have an emotional response to auditory stimuli. Our findings indicate that it may be possible to distinguish the emotional expression response of a patient based on their facial expression.

• Facial Expression
• Unresponsive Wakefulness Syndrome
• Music Stimuli

1. Introduction

Transition from impaired consciousness to a vegetative state creates a complex atmosphere where family caregivers have conflicting feelings, including a sense of emotional burden of care and expectations for recovery. Family visits may decrease as patients pass the acute stage and their overall condition becomes chronic, probably owing to the lack of a significant change or response from these patients. Therefore, it may be crucial for caregivers to feel that the patient is recognizing and responding to external stimuli.

We focused on facial expression analysis as an objective and quantitative method for evaluating the response of patients with unresponsive wakefulness syndrome (UWS) to external stimuli. Ribeiro et al. [1] observed changes in vital signs (blood pressure, heart rate, respiratory rate, and oxygen saturation) and facial expression after music stimuli in 26 patients with severe brain injury. However, in that study, most changes in facial expressions were subjectively evaluated. Facial analysis software has been developed to allow automatic and quantitative assessment of facial expression. FaceReader^® (Noldus, Wageningen, Netherlands; version 6) reads movements of the facial muscles, eyebrows, and mouth angles; subsequently, it integrates them to objectively evaluate the emotional response. A previous study reported 90% consistency between facial expression classification through FaceReader^® and manual facial expression classification based on the Facial Action Coding System (FACS) [2] [3] [4]. In particular, emotions can be evaluated more accurately by using a facial analysis software than by visual evaluation performed by humans, without resorting to the facial expression [5]. Diagnostic imaging of individuals with disorders of consciousness (DOC) caused by severe brain injury has revealed that auditory stimuli yield considerable activity in the brain’s auditory area [6]. A recent study noted that listening to one’s preferred music is particularly stimulating and enhances the hearing-recognition-related response of patients with disorders of consciousness [7]. Therefore, daily-life auditory stimuli are considered suitable and minimally invasive approaches of yielding static facial images necessary for facial analysis.

This study focused on “emotion” as a change in the valence value (comfort-discomfort index, which is described later) based on the classification proposed by Banham-Bridges [8]. In healthy individuals, changes in facial expression can be attributed to changes in emotions underlying conscious or unconscious responses to external stimuli (facial muscle reactions). However, it remains challenging to clearly distinguish between conscious and unconscious responses to external stimuli through facial expression changes in patients with UWS. Consequently, we defined emotional expression as a change in the valence value from the baseline based on valence values integrating changes from each facial expression muscle. Moreover, we defined facial expression motion as the degree of change in facial muscle movement, such as the action unit integral value, which is unrelated to emotional expression.

This study aimed to quantitatively measure facial expression changes in response to auditory stimuli in patients with UWS and evaluate the presence or absence of emotional responses. We hypothesized that auditory stimulation would yield higher valence integral, action unit integral, average heart rate, and data loss values (due to difficulty in capturing facial expressions because of active movement during auditory stimulation). These hypotheses were based on the study by de Jong et al. [9] where a mother-narrated story was presented on tape to a patient in a persistent vegetative state. Concomitantly, the regional cerebral blood flow was measured using positron emission tomography. The activation of the rostral anterior cingulate, right middle temporal, and right premotor cortices is indicative of appropriate cortical involvement in processing emotional attributes of sound or speech. Gutiérrez et al. [10] used heart rate variability to assess the responses to auditory stimulation with emotional content in patients in persistent vegetative or minimally conscious states; they found a changing pattern induced by auditory stimulation in three patients (decreased heart rate, increased heart rate variability, and decreased/increased power in the low/high frequencies, respectively), consistent with increased cardiovagal stimulation, and auditory stimuli could increase body movement.

2. Methods

Ethical considerations

This study was approved by the ethics committee of the participating facility. Consent to participate in the research was provided by the families of the patients; however, an explanation of the research was given at the bedside where feasible. The date and time of filming were determined with consideration of the patient’s condition and in consultation with the ward manager to ensure no interference with treatment. The family could withdraw from the study at any time, even after starting the filming, when the patient’s state became unstable, the family thought that the patient was being burdened, or the family members themselves felt burdened. When a patient’s eyes were closed for a prolonged period or their body movement was extreme during filming, it would be stopped and rescheduled where possible.

Participants

This study enrolled patients diagnosed with DOC due to brain injury with permission for participation from their doctor. Other inclusion criteria included functional hearing capacity as well as no history of arrhythmias or heart disease. We excluded patients with prolonged eye closure, those undergoing oral or nasal ventilation, or those with pre-injury hearing abnormalities. Data were collected between April 2017 and January 2018 in a private room or separate space at the hospital or in the patient’s home in Osaka, Japan.

Study intervention

Severe brain injury impairs the auditory brainstem’s response to sounds from the inner ear’s cochlear nerve to the midbrain; notably, the response remains until brain death, regardless of whether a person is asleep or awake [11]. Auditory stimulation was chosen based on the findings of our first research phase, where auditory stimulation was identified as a good stimulation method with minimal invasiveness.

There were two patterns of auditory stimulation used, music and voice stimuli, which are frequently used as interventions for restoring consciousness in patients with UWS. The stimuli were selected according to the circumstances and preferences of the family. Subsequently, patients’ facial expression changes were examined. Headphones were used to block other sounds and promote active listening. Regarding voice stimuli by the family, the primary caregiver who was closest to the patient recorded a 5-min recording of a letter and played it for the patient using headphones. For music stimuli, the primary caregiver played music familiar to the patient for 5 min using headphones. Based on the amount of time the patient could be supine, filming was performed for 5 min before and during auditory stimulation, with the session lasting for 10 min; this was repeated thrice a week. The volume was adjusted to 50 - 60 dB or less based on environmental standards. The timing for conducting the procedure was chosen in consultation with the head of the ward or the family. The procedure was performed during waking hours.

Study procedure

Head nurses at the hospital or home-visit nursing station, as well as doctors responsible for outpatient procedures, were asked to identify home-based and hospitalized patients based on the study’s objectives. Subsequently, the study was explained to the participants and proxies at the paticipant’s bed side. After consent was obtained, the date and time of the procedure were set. We collected the patients’ information, including data on age, sex, diagnosis, time since diagnosis, current medical records, and medical history. Moreover, the level of consciousness was evaluated based on the Tohoku Ryogo Center’s Prolonged Disorders of Consciousness Scale (Kohnan score) (Table 1).

Heart rate data were collected and analyzed using remote photoplethysmography, which is a non-invasive technique included in the analysis technology of FaceReader^® that measures small changes in color under the skin associated with changes in blood volume and oxygen saturation in blood vessels resulting from heartbeats. The consistency of the patient’s daily electrocardiogram monitoring data (cardiogram or pulse oximeter) was checked through visual examinations.

Regarding the facial expression analysis, feature points extracted on a

Table 1

Tohoku Ryogo center’s prolonged disorders of consciousness scale*

Disability grade Clinical symptoms	Extreme 10 points	Severe 9 points	Moderate 8 or 7 points		Mild 5 points		Slight 0 points
1. Voluntary movement	1) Absent 2) Acrocontracture 3) Pain reflex but slight trembling and rough breathing	1) Almost absent but parts of the extremities move minutely 2) Part of the extremity flexed and part paralyzed 3) Pain reflex or no pain reflex with clearly frowning face	1) Occasional all/partial extremity movement with no intention 2) Extremity could be paretic 3) Brushing away reaction for pain (7 points when good)		1) Occasional movement to meet an object 2) Capable of raising the arms upward or moving them in the intended direction, that is, face or head, imitating a posture of the investigator		1) Capable of movement with intention 2) Capable of unassisted posture change (partial change inclusive) 3) Moving wheelchair unassisted, even if awkwardly
2. Voluntary Ingestion	Totally incapable of masticating and swallowing; on tube nutrition (gastric/nasal feeding)	1) Almost on tube nutrition 2) Saliva swallowing or mastication is found 3) Capable of attempting slight perusal ingestion, that is, fruit juice, custard pudding, and so forth	1) Capable of masticating; even if not, almost capable of assisted auroral ingestion by swallowing, though sometimes choking 2) Insufficient peroral ingestion requires tube nutrition, (7 points if relatively accomplished)		1) Capable of unassisted ingestion by swallowing; mastication could be awkward 2) Capable of ingesting all the rice gruel served or chopped food with assistance 3) Attempting to reach mouth with a passed spoon or put the food into mouth awkwardly		Ingesting unassisted using spoon awkwardly
3. Incontinence	No observed somatic movement when evacuating/ urinating	Slight somatic movement when evacuating/urinating	After incontinence, a displeased look or some signal is observed, that is, frequent somatic movement. (7 points if relatively clear.)		1) Forced regular evacuating and urinating lead to the prevention of fecal and urinary incontinence 2) Communicating the fact in a certain way after incontinence		Except during the night, preevacuation and preurination communication is possible
4. Following with eyes and visual recognition	1) Eyes not opened 2) Eyes opened, no blink reflex	1) Eyes opened, blink reflex 2) No following ocular movement and no focusing eyes on an object	1) Looking straight toward the direction of the call 2) Following a moving object or staring at a TV, although understanding is impossible (7 points for rapid direct gaze)		1) Discriminating close relatives followed by a facial expression		1) Capable of reading easy words 2) Capable of understanding simple numbers 3) When watching TV, response and laughter are apparent
5. Meaningful speech	1) No vocalization 2) No lip movement under tracheostomy	1) Groaning without meaningful utterances 2) Lip movement observed under tracheostomy	1) A short utterance though not understandable (7 points if relatively clear) 2) Occasional inarticulate vocal response to calls (7 points if relatively clear) 3) Under tracheostomy, response to calls is through lip movement (7 points if relatively clear)		1) Occasional vocalizing of a meaningful word 2) Vocal response to calls 3) Imitating talking by the tester under tracheostomy		1) Capable of vocalizing a simple word response 2) Lip movement corresponds to what is asked
6. Follows simple directions and communicative	No reaction to being called (commands)	Some reaction to being called. Body movement, eye movement.		Can respond to calls, but communication not possible. (7 Points for clear response)		Responds to simple calls, communication at times	Responds well to calls, communication almost always possible
7. Facial expression changes	No response to ambient sound stimulations and TV sounds, among other things	Change of expression, such as smiling, crying, and anger, is not due to ambient stimulations		Change of expression is occasionally found in response to ambient stimulations (7 points for relatively clear change)		Change of expression, such as smiling, crying, and anger, closely matches an expected response to the ambient stimulation	Change of expression, such as crying and smiling, exactly matches an expected response to the ambient stimulation

*Kohnan Score. Categorization of patients according to the Persistent Consciousness Disorder Scale Score. Worst Total score is 70 points. Best score is 0 points. 70 - 65 points: Extreme cases (Completely Vegetative Symptoms), 64 - 55 points: Severe cases (Complete), 54 - 40 points: moderate cases (Incomplete), 39 - 25 points: Mild (Transitory Symptoms), below 24 points: Slight (Recovering cases)

frame-by-frame basis for videos were analyzed using FaceReader^®, which reads and quantifies human expressions and can record time-series changes in emotions with the FACS. The FACS was developed to objectively identify and classify human facial expression and emotional changes based on the movement of facial expression muscles [12]. It quantitatively describes the “smallest action unit of anatomically independent and visually identifiable facial expression motion.” Subsequently, it analyzes the action unit integral values after classifying them into 44 different action units based on set conversion rules. FaceReader^® extracted FACS-specified facial feature points from still images or videos of the patients’ faces before filming and identified 491 facial movement points (opening and closing of the mouth and both eyes, as well as raising and lowering of both eyebrows). Among the 44 action units, five levels of changes in the representative action unit could be identified. The FACS can define the action units involved in the formation of facial expressions, including happiness, anger, sadness, and pleasure, with each emotion being represented by a specific action-unit combination (Table 2).

For example, happiness can be expressed as action unit 6 + 7 + 12 or action unit 12. Additionally, 3D facial expression modeling based on these action-unit combinations can be used to quantify the intensity of arousal and basic emotions (neutral, happy, sad, angry, surprise, scared, and disgust) on a scale of −1 to 1 through comparisons with the established expression database of 10,000 people from different countries.

Valence indicates pleasure as positive and discomfort as negative. Basic emotions calculated through action unit combinations are all summed as negative, except for surprise and happiness, which are considered positive. Patients’ facial expressions were recorded at the patients’ home or hospital room using a video camera (HC-W570M; Panasonic, Kadoma, Japan) fixed to clearly show the patient’s face under direct light. During recording, the patient was wakeful in the supine position without any stimulation. The patients’ safety and provision of adequate care were emphasized, and filming was stopped when they were compromised. The total

Table 2

Each emotion expressed using a specific AU combination

Facial Expression	Pattern	Action Unit
Happy	1	AU6	Cheek Raiser	AU7	Lid Tightener	AU12	Lip Corner Puller
	2	AU12	Lip Corner Puller
	3	AU6	Cheek Raiser	AU11	Nasolabial Deepener	AU12	Lip Corner Puller
		AU25	Lip Part
Surprise	1	AU1	Inner Brow Raiser	AU2	Outer Brow Raiser	AU5	Upper Lid Raiser
		AU26	Jaw Drop
	2	AU1	Inner Brow Raiser	AU2	Outer Brow Raiser	AU5	Upper Lid Raiser
		AU27	Mouth Stretch
Scared	1	AU1	Inner Brow Raiser	AU2	Outer Brow Raiser	AU4	Brow Lowere
		AU5	Upper Lid Raiser	AU20	Lip Stretcher	AU25	Lip Part
		AU26	Jaw Drop	AU27	Mouth Stretch
	2	AU1	Inner Brow Raiser	AU2	Outer Brow Raiser	AU4	Brow Lowere
		AU5	Upper Lid Raiser	AU25	Lip Part	AU26	Jaw Drop
		AU27	Mouth Stretch
Sad	1	AU1	Inner Brow Raiser	AU4	Brow Lowere	AU11	Nasolabial Deepener
		AU15	Lip Corner Depressor	（AU54）	Head down	（AU64）	Eyes down
	2	AU1	Inner Brow Raiser	AU4	Brow Lowere	AU15	Lip Corner Depressor
		（AU54）	Head down	（AU64）	Eyes down
	3	AU6	Cheek Raiser	AU15	Lip Corner Depressor	（AU54）	Head down
		（AU64）	Eyes down
Anger	1	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU10	Upper Lip Raiser	AU22	Lip Funneler	AU23	Lip Tighter
		AU25	Lip Part	AU26	Jaw Drop
	2	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU10	Upper Lip Raiser	AU23	Lip Tighter	AU25	Lip Part
		AU26	Jaw Drop
	3	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU23	Lip Tighter	AU25	Lip Part	AU26	Jaw Drop
	4	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU17	Chin Raiser	AU23	Lip Tighter
	5	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU17	Chin Raiser	AU24	Lip Pressor
	6	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU23	Lip Tighter
	7	AU4	Brow Lowere	AU5	Upper Lid Raiser	AU7	Lid Tightener
		AU24	Lip Pressor
Disgust	1	AU9	Nose Wrinkler
	2	AU9	Nose Wrinkler	AU16	Lower Lip Depressor	AU25	Lip Part
		AU26	Jaw Drop
	3	AU9	Nose Wrinkler	AU17	Chin Raiser
	4	AU10	Upper Lip Raiser
	5	AU10	Upper Lip Raiser	AU16	Lower Lip Depressor	AU25	Lip Part
		AU26	Jaw Drop
	6	AU10	Upper Lip Raiser	AU17	Chin Raiser

Ekman & Friesen, 1978. AU, action unit.

recording duration from preparation to completion was <1 h.

Regarding the patient’s environmental factors, any major changes in temperature, humidity, sound, and luminosity on the day or during filming were recorded as stimuli using a luminance meter (SK-10LX; Sato Keiryoki MFG. Co., Ltd., Tokyo, Japan), a sound level meter (Mother Tool TM-102 Inc.), and a combined thermometer and hygrometer (SK-110TRH II Series Type 1, Sato Keiryoki MFG. Co., Ltd) installed at the patient’s bedside.

Data analysis

Every 1/15-s video footage (18,000 frames per session) was converted to a valence value ranging between −1 and 1. Moreover, action-unit integral values were determined on a five-point scale.

The time from the start of measurement to 5 min after measurement was represented on the x-axis, while valence and action-unit values were plotted on the y-axis for each timepoint. The line graphs were plotted at 5-min intervals to calculate the integral value of the area from the baseline. Missing values were calculated using the following formula:

Corrected integral value = actual integral value × 9000/(9000 − number of missing values).

Values obtained during the 5-min duration before and during the auditory stimulation were classified into before-stimulation and with-stimulation groups, respectively. Between-group comparisons were made for 1) valence integral values to verify the degree of emotional expression, 2) action-unit integral values to verify the movement of the muscles on the face and their degree of movement, 3) average heart rate values to verify the degree of response using a physiological indicator, and 4) missing values resulting from body movement or full-face movement (behavioral response) to verify the incidence of behavioral responses to auditory stimulation.

Regarding valence and action-unit integral values, only sessions with missing values of <10% of the 9000 missed frames were analyzed. Variables underwent normality testing using the Shapiro-Wilk test with subsequent analysis using the Wilcoxon signed-rank rest. All statistical analyses were performed using SPSS Statistics (IBM, Armok, NY).

3. Results

Patient characteristics are summarized in Table 3. This study included three men and three women with DOC due to brain injury. Two participants were hospital inpatients (less than 1 year from onset), while the other four resided at home. One patient’s family requested both voice and music stimuli (Patient 1). Six patients underwent 21 sessions. The average age of the patients was 45.2 ± 20.1 years (range: 21 - 76 years). Two patients were filmed at home and four at the hospital. Auditory stimuli were provided thrice to each patient. Ten sessions with missing values of <10% of the total 9000 frames are indicated using an asterisk. In each session, there was no change in the Kohnan score.

Assessments of reaction to auditory stimulation

Expression of emotional response (Table 4)

Data plotting and integration values were analyzed for ten effective sessions and significant between-group differences were observed in the degree of emotional expression (Figure 1). There was no significant between-group difference in the valence integral values (before-auditory stimulation: 36.4 ± 16.1; with-auditory stimulation: 52.8 ± 44.2; p = 0.49). Areas representing valence integral values per session are shown in gray in Figure 2. Auditory stimulation increased emotional responses in 5 (Patient-Session: 1-1, 1-3, 6-1, 6-2, and 6-3) of the 10 sessions (Patient-Session: 1-1, 1-3, 3-1, 3-2, 3-3, 4-3, 5-2, 6-1, 6-2, and 6-3). In the with-auditory stimulation group, Patients 3 and 6 showed decreased and increased emotional responses in all three sessions, respectively. The five sessions with increased valence integral levels were Patient-Session 1-1, 1-3, 6-1, 6-2, and 6-3. Patient 1 was a 42-year-old woman who had been diagnosed with brain injury 24 years earlier and lived at the home where she underwent in-home filming of voice stimulation sessions. Patient 6 was a 21-year-old man who usually lived at home but underwent filming of music stimulation sessions at a respite inpatient care center. He had been diagnosed 5 years previously with DOC caused by acute subdural hematoma after a traffic accident. The families of both patients reported reactions such as crying-like noises, producing a sound similar to a voice calling out, and staring at faces. However, in other times, they did not completely respond during excretion and treatment or when they were thought to be uncomfortable. Furthermore, among the five sessions with reduced valence integral values (Patient-Session: 3-1, 3-2, 3-3, 4-3, and 5-2), Patient 3 often responded with a voice calling out a name; however, movement and response usually stopped when the television was turned on. Although Patient 4 showed a change in facial expression depicting pain in response to barely painful stimuli, the patient appeared to be almost entirely unresponsive. Patient 5 exhibited a

Table 3

Patient characteristics and stimulation methods

Patient and session	Type of auditory stimulation	Place	Age (y)	Sex	Kohnan score		Total score	Diagnosis	Time since diagnosis (months/ years)
					Category	Score
*Patient 1, Session 1	Voice	Home	42	Female	1. Self-movement	9	52	Brain tumor	3 m/24 y
Patient 1, Session 2					2. Self-ingestion	10
*Patient 1, Session 3					3. Incontinence	10
Patient 1, Session 4	Music				4. Follow with the eyes and recognition	5
Patient1, Session 5					5. Meaningful speech	5
Patient1, Session 6					6. Following and communicative	5
					7. Changing facial expression	8
Patient 2, Session 1	Voice	Hospital	76	Male	1. Self-movement	8	58	Acute subdural hematoma	4 m/0 y
					2. Self-ingestion	10
Patient 2, Session 2					3. Incontinence	10
					4. Follow with the eyes and recognition	5
Patient 2, Session 3					5. Meaningful speech	9
					6. Following and communicative	9
					7. Changing facial expression	7
*Patient 3, Session 1	Voice	Home	39	Female	1. Self-movement	9	52	Head injury due to traffic accident	6 m/13 y
					2. Self-ingestion	10
*Patient 3, Session 2					3. Incontinence	9
					4. Follow with the eyes and recognition	5
*Patient 3, Session 3					5. Meaningful speech	7
					6. Following and communicative	5
					7. Changing facial expression	7
Patient 4, Session 1	Voice	Hospital	40	Male	1. Self-movement	7	59	Resuscitation after acute myocardial infarction	5 m/0 y
					2. Self-ingestion	10
Patient 4, Session 2					3. Incontinence	10
					4. Follow with the eyes and recognition	7
*Patient 4, Session 3					5. Meaningful speech	9
					6. Following and communicative	8
					7. Changing facial expression	8
Patient 5, Session 1	Music	Hospital (usually stays at home)	50	Female	1. Self-movement	7	58	Cerebral embolism	1 m/3 y
					2. Self-ingestion	10
*Patient 5, Session 2					3. Incontinence	10
					4. Follow with the eyes and recognition	8
Patient 5, Session 3					5. Meaningful speech	8
					6. Following and communicative	7
					7. Changing facial expression	8
*Patient 6, Session 1	Music	Hospital (usually stays at home)	21	Male	1. Self-movement	8	52	Head injury due to traffic accident	4 m/5 y
					2. Self-ingestion	10
*Patient 6, Session 2					3. Incontinence	10
					4. Follow with the eyes and recognition	5
*Patient 6, Session 3					5. Meaningful speech	7
					6. Following and communicative	5
					7. Changing facial expression	7

Patients were categorized according to Persistent Consciousness Disorder Scale Score (worst, 70 points; best 0 points). 70 - 65 points: extreme (completely vegetative symptoms), 64 - 55 points: severe (complete), 54 - 40 points: moderate (incomplete), 39 - 25 points: mild (transitory symptoms), <24 points: slight (recovering patients). *Indicates a session where the number of missing values was less than 10% of all 9000 frames.

Table 4

Areas representing the valence integral values (n = 18,000)

Case Number	Type of Audio Stimulation	Recording Location	before auditory stimulation	With auditory stimulation
*Case 1-1	Voice	Home	8.46	62.83
Case 1-2			82.54	45.99
*Case 1-3			31.88	62.19
Case 1-4	Music	Home	67.61	28.54
Case 1-5			89.99	35.33
Case 1-6			83.49	101.07
Case 2-1	Voice	hospital	12.67	14.90
Case 2-2			34.55	28.15
Case 2-3			32.26	144.57
*Case 3-1	Voice	Home	25.02	17.74
*Case 3-2			38.21	37.84
*Case 3-3			60.05	11.76
Case 4-1	Voice	hospital	16.39	14.06
Case 4-2			14.39	15.81
*Case 4-3			28.34	16.67
Case 5-1	Music	Hospital (usually at Home)	46.64	23.18
*Case 5-2			47.83	23.67
Case 5-3			12.74	37.91
*Case 6-1	Music	Hospital (usually at Home)	29.95	158.79
*Case 6-2			60.62	82.29
*Case 6-3			34.00	54.01

reaction to voice stimulus involving staring in the direction of the voice without movement. Patients with decreased emotional expression responses showed a static response, rather than a dynamic response, to stimuli. Additionally, in these patients, facial expression analysis could not determine emotion expression or perception.

Regarding the mean action-unit integral values, there was a significant between-group difference for action unit 10 only (p = 0.03). (Table 5) Although emotion assessment could not be conducted using action-unit 10 only, the lip-raising movement of facial muscles was less in the with-auditory stimulation group than in the before-auditory stimulation group. A visual image shows one of the nasolabial folds disappearing. The missing action-unit values for the five sessions that showed increased emotional expression due to auditory stimulation were <10%; moreover, there were action-unit 1, 4, and 7 responses, which correspond to upper-face movements (Figure 3). In the five sessions that showed an emotional expression response, in sessions 1, 2, and 3, Patient 6 showed significant variations in facial expression behavior in action units 1, 4, and 7, which

Table 5

Comparison of AU values before and during auditory stimulation across ten sessions

AU No.		The integral area values of AU		P
		before	during
1	Inner Brow Raiser	275.6	303.6	0.82
2	Outer Brow Raiser	256.6	193.6	0.43
4	Brow Lowere	335.3	310.9	0.81
5	Upper Lid Raiser	46.7	143.7	0.22
6	Cheek Raiser	94.7	25.1	0.53
7	Lid Tightener	265.2	244.5	0.42
9	Nose Wrinkler	9.2	0	0.34
10	Upper Lip Raiser	61.9	8.9	0.03*
12	Lip Corner Puller	23.1	31.8	0.71
14	Dimpler	94.9	64.5	0.52
15	Lip Corner Depressor	63.7	45.2	0.73
17	Chin Raiser	56.2	49.1	0.51
18	Lip Pucker	46	38.5	0.7 4
20	Lip Stretcher	1.3	8.7	0.22
23	Lip Tighter	26	20.2	0.77
24	Lip Pressor	32.5	2.3	0.32
25	Lip Part	277.2	385.4	0.61
26	Jaw Drop	192.4	258	0.62
27	Mouth Stretch	109.7	5.8	0.24

AU, action unit.

correspond to eyelid and upper-face movements.

Physiological reactions

The average heart rates in the before-auditory and with-auditory stimulation groups were 70.9 ± 6.9 and 69.4 ± 6.8 bpm, respectively, with no significant between-group difference (p = 0.33). As shown in Figure 4, there were fluctuating heart rate changes over time. The valence integral values and heart rate variations varied across the sessions. Moreover, Patients 1, 5, and 6 showed similar variation patterns for the average heart rate in both groups for all three sessions, with all sessions involving music stimuli.

4. Discussion

Real-time detection of responses to external stimuli and DOC fragmentation benefits both patients and their caregivers in case of a decline in quality of life resulting from impeded identification of a patient’s negative response to an intervention, as well as the inability of the family and caregivers to determine when the patient is feeling happy or comfortable.

This study conducted a facial expression analysis of patients with UWS at the bedside. There was no significant post-stimulation change in the valence integral levels; moreover, among the ten valid sessions, valence integral values for auditory stimulation increased in half the sessions and did not increase in the other half. This observed heterogeneity in patients responding to stimuli is consistent with that reported in previous studies.

Only action unit 10 showed a significant between-group difference among the average action-unit integral values in ten sessions. Although this was insufficient for evaluating emotion, the interpreted visual finding indicated greater disappearance of the nasolabial folds in the with-stimulation group than in the before-stimulation group. This finding suggests that the patient was responding to the stimulus.

There was no significant between-group difference in the average heart rate or valance integral values. However, two patients experienced a decrease in the average heart rate in all three sessions involving music stimuli, while one patient experienced an increase in the average heart rate in all sessions. This finding suggests that the three patients who underwent music stimulation experienced similar changes in each session. In patients with UWS, music stimulation caused unique changes in the cerebral blood flow of each person. This could be associated with the reproducibility of heart rate changes (similar heart rate change for each stimulus).

Patient 6 exhibited an increase in valence integral values, as well as a lower average heart rate, during auditory stimulation in all three sessions. The DOC in Patient 6 was caused by a subdural hematoma due to a traffic accident. Patient 6 was the youngest (21 years old) and the only one in their twenties. Emotional responses may be easier to elicit in younger patients.

Our findings show the characteristic emotional reactions that the families of each patient noticed on several occasions, consistent with the findings of previous research. Our findings show, subjectively and objectively, that patients with unresponsive arousal syndrome have a unique response. Specifically, this was due to gaining important insights from a nursing perspective. Our findings show that not only the subjective primary caregiver but also others may be able to confirm the emotional expression of the patients, which is a new finding.

5. Conclusion

In conclusion, we observed significant between-group differences in the average heart rate values and valence integral values, which represent pleasant or unpleasant emotions. Valence integral values increased in half of the sessions, which suggest that some patients showed an emotional expression response to auditory stimulation. Furthermore, AU integral values revealed upper-face, eyebrow, and eyelid movement during auditory stimulation, which could be used for discriminating between emotional expressions in clinical settings. Moreover, regarding the response of patients with UWS to external stimuli, music stimuli were considered more suitable than voice stimuli since the average heart rate fluctuations were consistent for nine music stimulation sessions (three sessions per case) and differed for the 12 voice stimulation sessions. Future clinical applications are anticipated after validation of these findings using other modalities.

6. Limitations

This study has several limitations. First, the sample size (n = 6) was small owing to altering physical conditions, as well as difficulty in gaining cooperation from recently diagnosed or home-based patients. Further multicenter studies in different cities and environments are needed to confirm the present findings. Second, data were collected based on a bedside survey using headphones. Therefore, although auditory stimuli were controlled to some degree, the possible influence of non-auditory stimuli could not be excluded. Third, this study did not employ a control group, and the results were only partially verified based on those of a previous study on healthy individuals. Despite these limitations, our results are significant given the potential impact of studies that advance behavioral assessments for patients with DOC, which may have wide implications in nursing and caregiving settings.

Support

This study was supported by the Japam Spciety for the Promotion of Science KAKENHI Grant in-Aid-for Young Scientists (B) [Grant number 16K20781, 2016].

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

1. Ribeiro, A.S., Ramos, A., Bermejo, E., et al. (2014) Effects of Different Musical Stimuli in Vital Signs and Facial Expressions in Patients with Cerebral Damage: A Pilot Study. Journal of Neuroscience Nursing, 46, 117-124. https://doi.org/10.1097/JNN.0000000000000037

2. Bijlstra, G. and Dotsch, R. (2010) FaceReader 4 Emotion Classification Performance on Images from the Radboud Faces Databases. Radboud University Nijmegen. Unpublished Manuscript. http://www.gijsbijlstra.nl/ http://ron.dotsch.org/

3. Lewinski, P., Den Uyl, T.M. and Butler, C. (2014) Automated Facial Coding: Validation of Basic Emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7, 227-236. https://doi.org/10.1037/npe0000028

4. Van Kuilenburg, H., Wiering, M. and den Uyl, M. (2005) A Model-Based Method for Automatic Facial Expression Recognition in Machine Learning. 16th European Conference on Machine Learning, Porto, October 3-7 2005, 194-205. https://doi.org/10.1007/11564096_22

5. Lewinski, P. (2015) Automated Facial Coding Software Outperforms People in Recognizing Neutral Faces as Neutral from Standardized Datasets. Frontiers in Psychology, 6, 1386. https://doi.org/10.3389/fpsyg.2015.01386

6. Boly, M., Faymonville, M.E., Peigneux, P., et al. (2004) Auditory Processing in Severely Brain Injured Patients: Differences between the Minimally Conscious State and the Persistent Vegetative State. Archives of Neurology, 61, 233-238. https://doi.org/10.1001/archneur.61.2.233

7. Heine, L., Castro, M., Martial, C., et al. (2015) Exploration of Functional Connectivity during Preferred Stimulation in Patients with Disorders of Consciousness. Frontiers in Psychology, 6, 1-11. https://doi.org/10.3389/fpsyg.2015.01704

8. Banham-Bridges, K.M. (1932) Emotional Development in Early Infancy. Child Development, 3, 324-341. https://doi.org/10.2307/1125359

9. de Jong, B.M., Willemsen, A.T. and Paans, A.M. (1997) Regional Cerebral Blood Flow Changes Related to Affective Speech Presentation in Persistent Vegetative State. Clinical Neurology and Neurosurgery, 99, 213-216. https://doi.org/10.1016/S0303-8467(97)00024-3

10. Gutiérrez, J.G., Machado, C., Estévez, M., et al. (2010) Heart Rate Variability Changes Induced by Auditory Stimulation in Persistent Vegetative State. International Journal on Disability and Human Development, 9, 213-216. https://doi.org/10.1515/IJDHD.2010.041

11. Ekman, P. and Friesen, W.V. (2003) Unmasking the Face: A Guide to Recognizing Emotions From Facial Clues. 10th Edition, ISHK, San Jose, CA.

12. Gosseries, O., Zasler, N.D. and Laureys, S. (2014) Recent Advances in Disorders of Consciousness: Focus on the Diagnosis. Brain Injury, 28, 1141-1150. https://doi.org/10.3109/02699052.2014.920522