False alarm reduction in critical care

During the last decade, over a period of seven years, intensive care unit (ICU) admissions at US hospitals increased by 48.8% with a mean biennial increase of 14.2%. By comparison, overall emergency department (ED) visits increased by 5.8% biennially. In absolute terms, admissions jumped from 2.79 million in 2002–2003 to 4.14 million in 2008–2009, according to data from the National Hospital Ambulatory Care Survey (Mullins et al 2013). The three most common diagnoses for ICU admissions were unspecified chest pain, congestive heart failure, and pneumonia. Utilization rates of most tests and services delivered to patients admitted to the ICU from the ED increased, with the largest increase occurring in computed tomography (CT) and magnetic resonance imaging (MRI), which increased from 16.8% in 2002/2003 to 37.4% in 2008/2009, a 6.9% mean biennial increase. These findings suggested emergency physicians were sending more patients on to the ICU. The increase might be the result of an older, sicker population that needs more care (Mullins et al 2013).

ICU patients require a high level of acute care, with numerous bedside monitors which are continuously measuring both invasive and non-invasive variables. These monitors provide synchronous waveforms with both independent and complementary information. Huge ICU databases are therefore becoming available, and include parameters such as the electrocardiogram (ECG), the photoplethysmogram (PPG), the arterial blood pressure (ABP) waveform, and respiratory effort. In clinical practice these signals are processed individually to trigger an alarm when a derived parameter (such as heart rate) exceeds a pre-defined range. These alarms are frequently false alarms (FAs) and account for a large majority of all alarms generated in the ICU (Chambrin et al 1999).

The high rate of false alarms significantly burdens clinical staff, which can lead to decreased quality of care (Donchin and Seagull 2002, Imhoff and Kuhls 2006), impacting both the patient and the clinical staff through noise disturbances, desensitization to warnings, slowing of response times (Chambrin 2001) and missed true alarms (Allen and Murray 1996, Chambrin 2001, Hug et al 2011). ICU alarms produce sound intensities above 80 dB that can lead to sleep deprivation (Meyer et al 1994, Chambrin 2001, Parthasarathy and Tobin 2004), inferior sleep structure (Slevin et al 2000, Johnson 2001), stress for both patients and staff (Cropp et al 1994, Novaes et al 1997, Topf and Thompson 2001, Morrison et al 2003) and depressed immune systems (Berg 2001). There are also indications that the incidence of re-hospitalization is lower if disruptive noise levels are decreased during a patient's stay (Hagerman et al 2005). Furthermore, such disruptions have been shown to have an important effect on recovery and length of stay (Cropp et al 1994, Donchin and Seagull 2002). In particular, cortisol levels have been shown to be elevated (reflecting increased stress) (Topf and Thompson 2001, Morrison et al 2003), and sleep disruption has been shown to lead to longer stays in the ICU (Parthasarathy and Tobin 2004). ICU false alarm (FA) rates as high as 90% (Aboukhalil et al 2008) have been reported, with between 6% and 40% of ICU alarms having been shown to be true but clinically insignificant (requiring no immediate action) (Lawless 1994). In fact, only 2% to 9% of alarms have been found to be important for patient management (Tsien and Fackler 1997). In response to this, thresholds and filter settings for alarms are often manipulated on a case-by-case basis in response to an individual clinical user's preferences (to reduce annoyance), which may well be sub optimal in terms of the trade off between true and false alarms (Mullins et al 2013).

In the 2015 PhysioNet/Computing in Cardiology Challenge (Clifford et al 2015) (http://physionet.org/challenge/2015), we aimed to address the problem of high false alarm rates by encouraging the development of new algorithms to improve the specificity of ICU alarms. In this Challenge, we focused on five types of life-threatening arrhythmia events, which we defined as follows:

• Asystole (ASY): No heartbeats for a period of 4 s or more.
• Extreme bradycardia (EBR): Heart rate lower than 40 beats per minute; fewer than five beats occur within a period of 6 s.
• Extreme tachycardia (ETC): Heart rate higher than 140 beats per minute; more than 17 beats occur within a period of 6.85 s.
• Ventricular tachycardia (VTA): Five or more consecutive ventricular beats within a period of 2.4 s (a rate of 100 per minute)
• Ventricular fibrillation or flutter (VFB): The heart exhibits a rapid fibrillatory, flutter, or oscillatory waveform for at least 4 s.

Participants in the Challenge were given samples of ICU patient waveforms that were identified by the bedside monitor as falling into one of the above categories, and were tasked with devising an algorithm to determine which of these alarms represented true arrhythmias, and which were caused by other factors (such as noise, patient movement, leads falling off, or mis-identification of ECG features on the part of the monitor.)

The Challenge was divided into two events. Event 1 was a simulation of the real-time alarm suppression problem: the algorithm needed to determine whether the alarm was true or false based solely on the information available before the alarm was first triggered. In Event 2, algorithms were also able to see 30 seconds worth of waveform data following the time of the alarm, and could use this information to retrospectively classify the alarm as true or false. The development of an algorithm that could reliably solve either of these problems would be a major step forward in patient care.

Key to rhythm detection is accurate heart rate estimation. Several ECG R-peak detection algorithms are freely available, several of which were used in the Challenge example entries.

• eplimited (available at www.eplimited.com) (Hamilton and Tompkins 1986), which used digital filtering and a group of decision rules.
• sqrs (available on PhysioNet (Goldberger et al 2000)) (Engelse and Zeelenberg 1979), which uses a single scan of the sampled data and combines digital filter preprocessing with a detector and feature extractor based on dynamically adjusted slope and timing criteria.
• wqrs (available on PhysioNet) (Zong et al 2003b), which is based on the length transform.
• gqrs (available on PhysioNet), which consists of a QRS matched filter with a custom built set of heuristics (such as search back).
• coqrs (Nygårds and Sörnmo 1983, Clifford 2002, Oster et al 2013) based on the peak energy (no search back).
• jqrs (Behar et al 2014a, 2014b) consists of a window-based peak energy detector but with replacement of the original band-pass filter with a QRS matched filter (Mexican hat) and an additional heuristic ensuring no detection were made during flat lines.

Detection of the onset of the pulses in the ABP and PPG signals can provide further information on rhythm and rate. An open-source algorithm, wabp (Zong et al 2003a), is available from PhysioNet. The algorithm consists of three components: (1) a low-pass filter which is to suppress high frequency noise that might affect the onset detection; (2) a windowed and weighted slope sum function (SSF) which is to enhance the up-slope of the pulse and to suppress the remainder of the pressure wave; (3) a decision rule which allows for detection of each SSF pulse onset.

We provided three example Challenge entries, based on these and other open-source algorithms, and implemented in various programming languages, to serve as a basis on which participants could develop their own code.

The simplest example entry (#1) used wabp and gqrs, along with the gqfuse tool (available on PhysioNet), to analyze all available signals and select the most stable sequence of RR intervals, in order to detect asystole, bradycardia, and tachycardia. To detect the onset of VFB, this entry analyzed the ECG and pulsatile signals separately (using gqfuse for each), and searched for a 10 s interval where the QRS rate and pulse rate were equal, followed by a 3 s interval in which the QRS rate increased by at least 25% and the pulse rate decreased by at least 75%. This entry did not attempt to detect VT.

A second example entry (#2) written in MATLAB used wabp to detect the beats and used jSQI (Sun et al 2006) and a template matching SQI (Li and Clifford 2012) to estimate the signal quality from ABP and PPG channels. For the ECG, an agreement level of two R-peak detectors (ph gqrs and ph coqrs) in a 10 s window, evaluated every second, known as bSQI, was used (Behar et al 2013).

Finally, the third sample entry (#3) was provided for Octave users, with functions from the WFDB toolbox for Octave/MATLAB Silva and Moody (2014). This sample entry ran three QRS detectors from the WFDB toolbox: wqrs on signal 1, sqrs on signals 1 and 2, and gqrs on signals 1 and 2. The results of the QRS detectors were then used to compute three tachograms. A decision was made on the veracity of the alarm based on the average pair-wise correlation between the tachograms 30 s prior to the alarm (a threshold was set arbitrarily based on the training data).

2.1. Signal quality

2.2. Voting algorithms

Data for the Challenge consisted of waveform recordings from ICU patients in four hospitals in the USA and Europe, representing three major manufacturers of ICU monitoring equipment. For each arrhythmia alarm matching our selection criteria, we collected all available multi-parameter waveforms (including at least five minutes of data before and after each alarm), as well as the alarm messages themselves, and any other status messages reported by the monitor. If possible, we also collected a list of the fiducial points and types of beats that were detected by the monitor; in some cases, the monitor did not provide this information. All of the signals were filtered in order to remove spectral characteristics that might identify the manufacturer or the country of origin. They were then resampled to 250 Hz and scaled to a 16-bit range. The specific names of the various alarm annotations were also normalized to anonymize the data.

3.1. Expert labeling

3.2. Training and test data

Participants were asked to submit their entries in the form of a compressed archive that included everything needed to compile and run their program on a GNU/ Linux system, together with the results that they expected their program to produce for the records in the public training set. When an entry was uploaded, the scoring system would first attempt to compile the program and run it over a randomly selected subset of the training set; if this did not produce the expected results, evaluation stopped and the error messages were sent back to the submitter.

Once the program was successfully compiled and validated, it was then invoked for each record in the test set.¹ If the program failed to produce output for a given record, it was treated as if it had classified that alarm as true.

For each category, the entry's score was computed based on the number of true positives (true alarms classified as true), false positives (false alarms classified as true), true negatives, and false negatives. The scoring function was designed to treat false negatives—genuinely life-threatening events that the program considered unimportant—especially harshly, and was defined as:

$$\begin{eqnarray}&&\text{score}=\frac{100\cdot (TP+TN)}{TP+TN+FP+5\cdot FN}\end{eqnarray}$$

A total of 29 closed-source entries and 215 open-source entries were submitted in the Challenge in 2015. Table 2 provides a breakdown of the top scoring entries. A different contestant ranked highest in each separate alarm category, indicating that there was no best general algorithm. Interestingly, a simple majority vote of all the 38 competitors' final entries gave scores of 60.15 in the real-time event and 62.41 in the retrospective event. These moderate performances, well below the top 10 algorithms, indicating that simple voting schemes do no yield an improved performance in this context, since the performance tail is long. A voting algorithm using the N  =  13 best performing final entries ranked by their score on the training data, provided the highest scores in both event 1 (84.26) and event 2 (87.04), although N  =  11 was sufficient to beat the best performance in either event. Figure 1 illustrates the performance of a simple voting approach for both the retrospective and prospective parts of the Challenge. Note that performance only degrades above 13 algorithms.

Zoom In Zoom Out Reset image size

A total of 13 articles were reviewed and revised in time to be accepted for this special issue. Most authors had originally entered the Challenge, and submitted updated versions of their algorithms, which should be made available by the authors through their open source licenses. The top reported results on the hidden test set for each alarm type were: ASY: 97.4% (Plešsinger et al 2016), EBR: 93.8% (Krasteva et al 2015a), ETC: 100.0% (Hoog Antink et al 2016), VFB: 88.7% (Rodrigues and Couto 2016), and VTA: 76.7% (Kalidas and Tamil 2016), yielding an average best of 91.2%.

Each algorithm published in this issue is reviewed below according to four standard stages of algorithm function:

• 1.  
  Pre-processing and signal conditioning
• 2.  
  Beat detection
• 3.  
  Beat classification
• 4.  
  Alarm classification

The purpose of this standardized summary is to glimpse at a myriad of advanced approaches used by the competitors in a format that allows the reader to quickly identify both the commonalities and the originality of all the approaches. Finally, the last two articles in this review (Daluwatte et al 2016, Tsimenidis and Murray 2016) did not attempt to reduce the number of false alarms, but rather provide some useful insights into the relationship between signal quality metrics and false alarm rates.

6.1. Ansari et al (2016)

6.2. Eerikäinen et al (2016)

6.3. Fallet et al (2016)

6.4. Hoog Antink et al (2016)

6.5. Kalidas and Tamil (2016)

6.6. Krasteva et al (2016)

6.7. Liu et al (2016)

6.8. Plešsinger et al (2016)

6.9. Rodrigues and Couto (2016)

6.10. Sadr et al (2016)

6.11. Zong et al (2016)

6.12. Daluwatte et al (2016)

6.13. Tsimenidis and Murray (2016)

In summary, the PhysioNet/Computing in Cardiology Challenge 2015 provided several key additions to the field of false alarm suppression in critical care. First we note that for the top performing entrants, it was the VTA alarm that proved the hardest to classify accurately. This is partly because, at low heart rates, the signal becomes 'more normal looking' in the other signals. This was previously noted in Aboukhalil et al (2008). To-date, there has been little to address this issue, although the current Challenge has made a move towards this. Second, we note that retrospective scores were generally higher than 'real-time' scores, with the highest performing retrospective approach only suppressing 1% of the true alarms, while 80% of the false alarms were suppressed. Although debatable, this may be acceptable as a clinical algorithm if a 30 s window were acceptable. We suggest that this may spur a re-consideration of the Association for the Advancement of Medical Instrumentation (AAMI) guidelines for maximum alarm latency. Third, we note that voting algorithms together can produce superior results to even the best algorithm. Such an approach can also lead to a more robust implementation, although it may be significantly more computationally intensive. It is also important to note that too many naive voters (more than 13 in the case of the 2015 Challenge) can reduce the accuracy of the label or answer. In Zhu et al (2014) and Zhu et al (2015) a voting system for algorithm (and human) annotations of physiological data was described, which incorporates both the physiology and the individual annotator's accuracy as a function of objective features (such as signal quality) to produce a weighted voting scheme to guarantee that all voters add extra information. We suggest that such approaches will become ever more important as computational power becomes increasingly less expensive. We also note that this means that all competitors in the challenge added something to the final answer!

Finally we note some limitations of the competition. A larger database is needed with more patients, longer recordings, more leads and abnormalities (such as arrhythmias). We intend to work with industry and researchers alike to enhance the Challenge database in all these areas and would be grateful for continued contributions of data and source code, which we will post together with all the open source algorithms and annotated data from the 2015 PhysioNet/Computing in Cardiology Challenge. The latter can be found on PhysioNet's website at http://physionet.org/challenge/2015.

This work was funded in part by the National Institutes of Health, grant R01-GM104987 and by the National Institute of General Medical Sciences, under NIH cooperative agreement U01-EB-008577 and NIH grant R01-EB-001659. We also would like to thank Arvind Ananthan, Jaka Cikač, Barbara Drew, Quan Ding, Scott Eaton, Xiao Hu, Franc Jager, Cadathur 'Raj' Rajagopalan and Anže Rezelj, as well as GE Healthcare, MathWorks, Mindray North America and Philips Healthcare for their valuable assistance. Finally, we thank all the competitors themselves, without whom there would be no competition.