Electrophysiology-based detection of emergency braking intention in real-world driving

In total, 25 individuals participated in this study. Five of the resulting datasets had to be excluded from further analysis due to technical problems. Therefore, the sample consisted of 20 participants (22–53 years, mean: 29.0 years, five females). Subjects were recruited from an in-house database, in which volunteers for experiments are listed. Every subject had normal or corrected-to-normal vision, reported normal hearing and had no history of psychiatric or neurological diseases. Participation was voluntary and occurred during working hours. All experimental procedures were conducted in accordance with the ethic guidelines of the declaration of Helsinki. Data were collected anonymously. Informed consent was obtained after the task had been explained. Participants were informed they had the option to end participation in the experiment at any time without any type of penalty. Participants received a gift worth approximately 20 EUR for their participation.

The study was conducted on a non-public test track in an unused military training area in Münsingen, Germany.

Car-following task (primary). The primary task was to drive in accordance with official traffic regulations. Participants had to drive three rounds on the test track, one round being 37 km long with distinct variations in road curvature and altitude. The setup consisted of two Mercedes-Benz S-class cars: the lead car was navigated by an investigator and the following car was driven by the participant. Participants were instructed to follow the lead car at a constant distance of approximately 20 m at a maximum speed of $60\;{\rm km}\;{{{\rm h}}^{-1}}$. In order to obtain a reference for the required distance, the participantʼs car was parked 20 m behind the lead car before the start of the experiment. The investigator initiated emergency braking with an interstimulus interval of 42.5 s–57.5 s (M = 50 s, uniformly distributed jitter) after receiving an acoustic trigger from a laptop, provided that the lead car had a constant velocity of $60\;{\rm km}\;{{{\rm h}}^{-1}}$ and adequate separation to the trailing vehicle. Given the lead carʼs abrupt braking from $60\;{\rm km}\;{{{\rm h}}^{-1}}$ to $40\;{\rm km}\;{{{\rm h}}^{-1}}$, the participant was instructed also to brake immediately as soon as the brake light of the lead car flashed, irrespective of the actual distance between cars. After every emergency braking, the investigator accelerated back to $60\;{\rm km}\;{{{\rm h}}^{-1}}$ using cruise control.

Auditory task (secondary). As a temporary secondary task, participants listened to parts of an audio book. They were instructed to alternately detect two frequent German words by pressing a button that was attached to their left index finger with their thumb. The primary car-following task had to be prioritized over the auditory secondary task at all times.

Structure of the experiment. Participants had to drive three rounds on the test track, with short breaks between each round, which occurred after about 40 and 80 min of driving. The experiment consisted of 16 blocks, superimposed on the continuous driving task. In every block, participants drove for 3 min without performing the auditory secondary task and for 3 min with secondary task (see figure 1). The beginning and the end of every 3 min interval was announced verbally. For the whole study, participants had to drive a total of 48 min in both conditions (driving only, driving with auditory secondary task).

Zoom In Zoom Out Reset image size

After agreeing to the study, participants were fitted with a 32-electrode-cap (actiCAP, Brain Products GmbH, Munich). A set of 25 EEG electrodes (F3, Fz, F4, P7, P8, T8, FC3, FC4, C3, Cz, C4, T7, CP3, CP4, FC5, P3, Pz, P4, FC6, O1, O2, Oz, CPz, PO4, PO3) was positioned according to the international 10–20 system. EEG data were recorded relative to FCz, and all impedances were maintained less than 10 k $\Omega$. Muscle activity from the right foot was measured with two electromyography (EMG) electrodes, positioned at the right musculus tibialis anterior and on the right thigh. All data were digitized at 250 Hz with a bandpass filter (low: 0.53 Hz, high: 100 Hz), and a 50 Hz notch filter was applied to remove power line interference.

Gas and brake pedal deflections as well as other technical parameters such as the timings of brakelight flashes were acquired with a sampling rate of 50 Hz through the controller area network bus units of both vehicles, and were synchronized with the EEG and EMG data. The time between the lead carʼs brake lights flashing and the brake pedal response signal from the trailing car was defined as the brake reaction time.

Statistical analysis was performed using MATLAB (The Mathworks). The EEG data were re-referenced offline to common average. The EMG data were rectified by taking the absolute values. All continuous data (EEG, EMG, gas pedal deflection and brake pedal deflection time series) were segmented into target and non-target intervals. Targets were defined as those situations, in which the braking response was given no earlier than 300 ms and no later than 1200 ms after an abrupt braking of the lead vehicle (stimulus onset). A further requirement for a valid target was that the participant had the foot on the gas pedal at the moment of brake light flashing. Targets were obtained by cutting out data from $-300\;{\rm ms}$ to 1200 ms relative to the stimulus onset in all valid target situations. Normal driving (non-target) segments were obtained by collecting all data blocks (1500 ms duration, 500 ms equidistant offset) that were at least 3000 ms apart from any stimulus. Baseline correction was performed for the EEG data segment-wise by subtracting the average EEG amplitude in the first 100 ms of the extracted interval. The total numbers of target and non-target segments per subject were ${{N}_{{\rm t}}}=134\pm 17$ and ${{N}_{{\rm nt}}}=5623\pm 615$ on average (± std), respectively.

We also investigated the performance with which emergency braking and normal driving situations could be distinguished based on the single-trial spatio-temporal dynamics of the four available measurement modalities at each stage of the emergency situation. To this end, we constructed additional sets of segments with constant length of 1500 ms. The endpoints of these segments varied in steps of 20 ms from −200 ms to 1180 ms relative to the stimulus onset. Furthermore, separate datasets were created for the four individual modalities EEG, EMG, gas and brake. The target and non-target segments were split into training and test parts, such that the training sets contained only data from the first half of driving and the test sets only data from the second half. The following procedure was performed for each time shift and for each modality. Ten discriminating time intervals were determined heuristically (Blankertz et al 2011). EEG electrodes showing strong artifacts on the training data were discarded (Haufe et al 2011). For each segment, signals of the remaining electrodes were averaged within the selected time intervals and stacked into feature vectors, which were used to train regularized linear discriminant analysis (RLDA) (Friedman 1989, Duda et al 2000) classifiers. For regularization, the automatic shrinkage technique (Ledoit and Wolf 2004, Schäfer 2005, Blankertz et al 2011) was adopted. Data from the first half of driving were used for selecting time intervals and for training the RLDA classifiers. The trained discriminant functions were applied to the corresponding test data, and the resulting outputs were used to measure classification performance. We used the area under the receiver-operating characteristics curve (AUC, Fawcett 2006) as a performance measure. Area under the curve (AUC) scores are normalized to the interval $[0,1]$ where 1 indicates perfect classification, 0 indicates perfect misclassification, and 0.5 is attained for chance-level classification. Grand-average AUC scores were calculated as the arithmetic mean across subjects.

The grand-average median response time in target situations was 720 ms, and was thus slightly longer compared to the laboratory setting of Haufe et al (2011), where it was 665 ms. This slight increase could be attributed to the additional workload introduced by the secondary task to be performed half of the time. The median response time in the driving-only blocks was 676 ms, while it was 740 ms in the blocks in which the auditory task had to be performed, including the announcements. Note that all graphical data presented in this manuscript comprise both real-world driving conditions (driving-only and auditory dual task), while separate data is presented in the supplementary data (section S1), available from stacks.iop.org/jne/11/056011/mmedia.

We computed grand-average stimulus-related behavioral and EMG signals by taking the arithmetic mean of the extracted target segments of all subjects. The resulting curves are depicted in figure 2(A) as thick lines. The corresponding curves obtained in the previous laboratory study are overlaid as thin curves. Note that, unlike in Haufe et al (2011), the average of the entire data (not only of data from the second half of driving) is depicted here. Both curves show a striking resemblance, indicating that the real-world experiment induced the same behavioral pattern as the laboratory study. This pattern is characterized by an initial burst of EMG activity, a subsequent drop of the gas pedal deflection, and finally an abrupt increase of the brake pedal deflection.

Zoom In Zoom Out Reset image size

In analogy to the behavioral and EMG channels, we computed grand-average event-related potential (ERP) curves. These are shown in figure 2(B) for a selection of 11 EEG channels. The corresponding laboratory data are again overlaid as thin curves. To make both curves comparable, the lab data were transformed to common-average reference prior to the re-analysis. Figure 2(C) depicts an alternative representation of the ERP sequences, where average voltages in five subsequent time intervals of 160 ms length are shown as a series of topographical (scalp) maps. Here, the upper panel (thick scalp outlines) depicts real-world data, while the lower panel (thin outlines) depicts the laboratory data of Haufe et al (2011).

Figures 2(B) and (C) highlight the similarity of the event-related potentials elicited by forced emergency brakings in both studies. That is, we observe a spatio-temporal ERP complex composed of the identical three subcomponents reported in Haufe et al (2011): an early symmetric negative deflection in occipito-temporal areas, a later negativity at central scalp sites and a late positive deflection in centro-parietal areas. As also noted in Haufe et al (2011), all of these three ERP components have a clear functional relevance. The early occipital negativity (visual-evoked potential, VEP) can be attributed to low-level processing of the emergency-inducing visual stimulus (the brakelight flashing). Higher-level (semantic) processing of the gravity of the emergency situation is reflected in the later centro-parietal positivity (P300). Finally, the late central negativity (readiness potential, along with other movement-related potentials) reflects the preparation and execution of the braking response performed by the right foot.

The grand-average AUC scores attained for single measurement modalities at each stage of emergency braking are depicted in figure 3(A). AUC curves obtained on the present real-world data are drawn as thick lines, while the corresponding curves obtained by Haufe et al (2011) in a laboratory setting are drawn as thin lines. As the figure shows, both curves are comparable regarding their shape and timing. For each time point, corresponding AUC scores of both datasets were compared using two-sided Wilcoxon rank sum tests. Note, that the reported results are not corrected for multiple testing, as we were not interested in an overall result here, but used the statistical tests here just as a robust quantification of the predictive value of different measures over time. The gas and brake modalities generally attained higher grand-average accuracies in the laboratory setting. This difference was significant ($p\lt 0.05$) between 860 ms and 1200 ms post-stimulus for gas and almost in the entire interval between 520 ms and 1160 ms for brake. The EEG and EMG channels attained higher AUC values in the simulated driving setting compared to the real-world driving setting in the late stages of the emergency situations. The difference was significant almost in the entire interval between 660 ms and 1200 ms post-stimulus for EEG and between 640 ms and 1200 ms for EMG. Note in this respect, however, that more EEG channels were available in the laboratory setting than in the real-world driving setting (58 compared to 25, see Haufe et al 2011). Interestingly, electrophysiological measures turned out to be more predictive in the real-world driving setting than in the laboratory setting in the early stages of emergency braking. This was the case ($p\lt 0.05$) between 140 ms and 200 ms for EEG despite the lower number of electrodes in the real-world driving setting, and almost in the entire interval between 260 ms and 460 ms for EMG. Significant time points are marked as square boxes in figure 3(A), where filled boxes represent intervals, in which higher accuracy was achieved in the real-world driving setting, while unfilled boxed represent the opposite case (higher accuracy in the laboratory setting).

Zoom In Zoom Out Reset image size

The main goal of the study was to determine whether information contained in electrophysiological channels improves the detection of emergency braking situations in real-world driving to a similar degree as in simulated driving. To investigate this, we repeated the classification analysis described above for the combination of the gas and brake channels, and for the combination of all four modalities. The resulting curves are presented in figure 3(B). Using one-sided paired Wilcoxon signed rank tests, we assessed for every time point post-stimulus, whether the AUC scores obtained for the EEG+EMG+gas+brake modality combination were higher compared to using the only gas+brake. The time intervals, in which this was the case ($p\lt 0.05$) are marked as square boxes in figure 3(B), where the boxes related to real-world driving are filled with light-gray color. The analysis revealed that the significant intervals were similar for real-world and simulated driving (180 ms to 1080 ms post-stimulus for real-world driving and 240 ms to 1060 ms post-stimulus for simulated driving).

We were also interested in the average time saved by including electrophysiological channels compared to using only behavioral channels. To investigate this issue, Haufe et al (2011) constructed two pseudo-online emergency braking detectors, and compared their performance in what one could call a realistic application scenario. They showed that, at constant detection accuracy, the inclusion of EEG and EMG channels led to 130 ms faster detections on average. Here, we took a more straightforward approach to assess this improvement; namely, we calculated the area of the polygons spanned by the AUC curves related to the EEG+EMG+gas+brake and gas+brake modality combinations. This area is marked in light gray for the real-world driving data in figure 3(B). The two endpoints of the polygons were given by the points in which the accuracy scores deviated first and later merged again. These points were $t=140\;{\rm ms}$ (${\rm AUC}=0.525$) and $t=1\;000\;{\rm ms}$ (${\rm AUC}=0.983$) for the real-world driving dataset, and ($t=180\;{\rm ms}$, ${\rm AUC}=0.516$) and ($t=1\;180\;{\rm ms}$, ${\rm AUC}=0.997$) for the simulated driving dataset. The polygon areas were divided by the difference of the AUC scores at the two polygon endpoints to yield the braking detection time that could be saved using electrophysiology, integrated over all achievable detection accuracies. The average improvement for simulated driving was 200 ms, while for real-world driving it was 237 ms. The analysis was repeated for the driving-only test data and the driving-with-secondary-task-only test data. Notably, the improvement was larger in blocks in which the secondary task had to be performed, and during announcements (253 ms compared to 222 ms in driving-only blocks). See section S1 in the supplementary data for respective AUC curves.