A review of rapid serial visual presentation-based brain–computer interfaces

Event-related potentials (ERPs) are electroencephalography (EEG) signal amplitude variations in the EEG associated with the onset of a stimulus (usually auditory or visual) presented to a person. ERPs are typically smaller in amplitude (<10 µV) in comparison to the ongoing EEG activity (~50–100 µV) they are embedded within (Huang et al 2008, Acqualagna and Blankertz 2011). As ERPs are locked in phase and time to specific events, they can be measured by averaging epochs over repeated trials (Huang et al 2011, Cecotti et al 2012, 2014). Shared EEG signal features are accentuated and noise attenuated (Luck 2005, Cohen 2014). The outcome is represented by a temporal waveform with a sequence of positive and negative voltage deflections labeled as ERP components. ERPs are representative of summated cortical neural processing and behavioral counterparts, such as attentional orientation (Wolpaw and Wolpaw 2012, Cohen 2014).

The stream of images presented within an RSVP paradigm comprise frequent non-target images and infrequent target images; different ERP components are associated with target and non-target stimuli (Bigdely-Shamlo et al 2008, Cohen 2014, Sajda et al 2014). BCI signal processing algorithms are used to recognize spatio-temporal electrophysiological responses and link them to target image identification, ideally on a single trial basis (Manor et al 2016).

The most commonly exploited ERP in RSVP-based BCI applications is the P300. The P300 appears at approximately 250–750 ms post target stimulus (Polich and Donchin 1988, Leutgeb et al 2009, Ming et al 2010, Zhang et al 2012). As specified by Polich and Donchin (1988) during the P300 experiment (commonly referred to as the 'oddball' paradigm), participants must classify a series of stimuli that fall into one of two classes: targets and non-targets. Targets appear more infrequently than non-targets (typically ~5–10% of total stimuli in the RSVP paradigm) and should be recognizably different. It is known that P300 responses can be suppressed in an RSVP task if the time between two targets is  <0.5 s, which is known as attentional blink (Raymond et al 1992, Kranczioch et al 2003). The amplitude and the latency of the P300 are influenced by the target discriminability and the target-to-target interval in the sequence. The latency of the P300 is affected by stimulus complexity (McCarthy and Donchin 1981, Luck et al 2000). The P300 amplitude can vary as a result of multiple factors (Johnson 1986), such as:

• subjective probability—the expectedness of an event;
• stimulus meaning—comprised of task complexity, stimulus complexity and stimulus value;
• information transmission—the amount of stimulus information a participant registers in relation to the information contained within a stimulus.

BCIs can be of three different types: active, reactive or passive (Zander et al 2010). An active BCI is purposefully controlled by the user through intentional modulation of neural activity, often independent of external events. Contrastingly, reactive BCIs generate outputs from neural activity evoked in response to external events, enabling indirect control by the user. Passive BCI makes use of implicit information and generate outputs from neural activity without purposeful control by the user. Active/reactive BCIs are commonly aimed at users with restricted movement abilities who intentionally try to control brain activity, whereas implicit or passive BCIs are more commonly targeted towards applications that are also of interest to able-bodied users (Zander and Kothe 2011, Sasane and Schwabe 2012).

RSVP-based BCIs have two presentation modes: static mode in which images appear and disappear without moving; and moving mode where targets within short moving clips have to be identified (Sajda et al 2010, Cecotti et al 2012, Weiden et al 2012). Both presentation modes can be used with or without a button press. With a button press, users indicate manually, by pressing a button, when they observe a target stimulus. A button press is used to establish baseline performance, reaction time and/or to enhance performance (discussed further in section 5.1).

2.3.1. Static.

2.3.2. Moving.

When designing an RSVP-based BCI, three different types of cognitive blindness should be considered namely, the attentional blink, change blindness and saccadic blindness. Generally, RSVP is a paradigm used to study the attentional blink, which is a phenomena that occurs when a participant's attention is grabbed by an initial target image and a further target image may not be detectable for up to 500 ms after the first (Raymond et al 1992). Depending upon the duration of stimuli presentation the ration of target images/total images will change (e.g. if images are being presented at a duration of 100 ms then there must be a minimum of five images between targets 1 and 2. In a sequence of 100 images there can be a maximum of 20 target images. Whereas if images are presented at 200 ms this limits the maximum number of targets to 10/100 images in total).

Change blindness occurs when a participant is viewing two images that vary in a non-trivial fashion, and has to identify the image differences. Change blindness can occur when confronted by images, motion pictures, and real world interactions. Humans have the capacity to get the gist of a scene quickly but are unable to identify particular within-scene features (Simons and Levin 1997, Oliva 2005). For example, when two images are presented for 100 ms each and participants are required to identify a non-trivial variation as the images are interchangeably presented, participants can take between 10–20 s to identify the variation. This latency period in identifying non-trivial variations in imagery can be augmented through use of distractors or motion pictures (Rensink 2000). In the context of designing an RSVP paradigm change blindness is of interest, as it will take longer for a user to identify a target within an image if it does not pop out from the rest of the image. Distractors within the image or cluttered images, will increase the time it takes a user to recognize a target, reducing the performance of the RSVP paradigm.

Saccadic blindness is a form of change blindness described by Chahine and Krekelberg (2009) where 'humans move their eyes about three times each second. Those rapid eye movements called saccades help to increase our perceptual resolution by placing different parts of the world on the high-resolution fovea. As these eye movements are performed, the image is swept across the retina, yet we perceive a stable world with no apparent blurring or motion'. Saccadic blindness thus refers to the loss of image when a person saccades between two locations. Evidence shows that saccadic blindness can occur 50 ms before saccades and up to 50 ms after saccades (Diamond et al 2000). Thus, it is important that stimuli have a duration greater than 50 ms to bypass saccadic blindness, unless participants are instructed to attend a focus point and the task is gaze independent and thus does not demand saccades (such as during the canonical RSVP paradigm (section 5.4)).

Having considered some of the factors influencing RSVP-based BCI designs, the remainder of the paper focuses on a bibliometric study of the RSVP literature highlighting the key methodological parameters and study trends. Studies are compared and contrasted on an intra- and inter-application basis. Later sections focus on study design parameters and provide contextualized recommendations for researchers in the field.

A button press may be used in conjunction with either of the aforementioned presentation modes (section 2.2), and entails users having to click a button when they see a target. This mode is used as a baseline to estimate the behavioral performance and the difficulty of the task. In most research studies, participants undergo an experimental trial without a button press and a follow-on trial with a button press.

A button press can be used in RSVP-based BCI research in combination with the participant's EEG responses in order to monitor attention (Marathe et al 2014). The combination of EEG and button press can lead to increased performance in RSVP-based BCIs. Tasks that require sustained attention can cause participants to suffer from lapses in vigilance due to fatigue, workload or visual distractors (Boksem et al 2005). The button press can be used to determine if there is a tipping point during the presentations when participants are unable to consciously detect target stimuli, while still identifying targets via EEG recordings (Potter et al 2014). However, the core advantage of the RSVP-based BCIs is the enhanced speed of using a neural signature instead of a behavioral response to determine if a user has detected an intended image of interest.

Forty of the studies reported use static mode as a method of presentation; six of these papers used moving mode in conjunction with static mode while one study exclusively used moving mode. Moving mode is more complex than static mode as participants have to take in an entire scene rather than specific images. Moving mode uses motion onset in conjunction with the P300 for scenes in which the targets are moving, yielding a more realistic setting to validate RSVP-based BCIs (Weiden et al 2012). All papers employing moving mode were found within the surveillance application category; this is unsurprising as the moving mode offers the opportunity to detect targets in realistic surveillance situations where movements of people or vehicles are of interest. For the other application areas, i.e. medical, categorization, etc the static mode is likely to be the most appropriate.

Won et al (2017) compared motion RSVP to standard RSVP, with the motion-type RSVP being the rapid presentation of letters of the alphabet, numbers 1–9 and a hyphen '-' used to separate words, in six different color groups in one of six directions in line with the hands of a clock, i.e. 2, 4, 6, 8, 10, 12, whilst participants focused on a central point. An increase in performance accuracy with motion-type RSVP versus static-type was demonstrated, which could be accounted for by the shorter latency and greater amplitudes of ERP components in the motion-type variation (Won et al 2017).

Out of the studies found, 22 used a button press while 23 did not. 70% of surveillance applications used a button press. In categorization studies and face recognition studies the majority of applications used a button press. 89% of RSVP-speller applications did not use a button press. Typically, the BCI studies that involve spellers focus on movement-free communication and high information transfer rates. Having a button press for confirmation of targets is not standard practice in such applications (Orhan et al 2012, Oken et al 2014). In many of the studies that did not utilize a button press, researchers were focused on different aspects of the RSVP paradigm other than reaction time. For example, researchers focused on the comparison of two classification methods, image duration, etc (Sajda et al 2010, Cecotti et al 2014). Combining EEG responses with button press can improve accuracy although more signal processing is required in order to remove noise that occurs as a result of participant movement (Healy and Smeaton 2011). Button press confirmation is unnecessary unless an assessment of physical reaction time is an important aspect of the study.

Maguire and Howe (2016) instructed participants to use a button press following image blocks to indicate if a target was consciously perceived as present or absent. Such an approach is useful when studying RSVP-based parameters and the limits of perception. However, button press responses might be less useful than EEG responses during RSVP for data labeling or image sorting, where the focus is to label individual images within the burst. Nonetheless, Bigdely-Shamlo et al (2008) applied an image burst approach where a button press at the end of the image burst was used to determine if the participant saw a target image or not. The authors showed that airplanes could be detected in aerial shots with image bursts lasting 4100 ms and images presented at 12 Hz. The button press served well in determining correct and incorrect responses. In practice, however, a button press may be superfluous or infeasible.

A body of researchers is of the opinion that RSVP-related EEG accuracy must surpass button press accuracy in order to be useful. However, this need not be the case as Gerson et al (2006) report no significant differences in triage performance based on EEG recordings or button presses. Nevertheless button-based triage performance is superior for participants that correctly respond to a high percentage of target images. Conversely, EEG-based triage alone is shown to be ideal for the subset of participants who responded correctly to fewer images Gerson et al (2006). Hence, the most reliable strategy for image triaging in an RSVP-based paradigm may be through reacting to the target image by real-time button presses in conjunction with an EEG-based detection method. Target identification reflected in EEG responses can be confirmed by a button press, and through signal processing techniques both reported and missed targets can be identified.

Studies such as Marathe et al (2014) proposed methods for integrating button press information with EEG-based RSVP classifiers to improve overall target detection performance. However, challenges arise when overlaying ERP and behavioral responses, such as issues concerning stimulation presentation speed and behavioral latency (Files and Marathe 2016). Crucially Files and Marathe (2016) demonstrated that techniques for measuring real-time button press accuracy start to fail at higher presentation rates. Given evidence of human capacity for semantic processing during 20 Hz image streams (approximately 50 ms per image) and response times (RTs) often being an order of magnitude greater than EEG responses, button presses may be unsuitable for faster RSVP-based image triaging.

Pending further studies investigating the reliability of fast detection of neural correlates, EEG-based responses have the potential to exceed button press. However, it is not necessary for EEG-based RSVP paradigms to surpass button press performance and evidence suggests that a complement of both modalities at comfortable lower presentation rates may indeed be the best approach. Nevertheless, ideally studies would contain an EEG-only block and EEG plus button press block, where the button press follows the target and not the image burst. This would facilitate more accurate evaluation of differences and correlations between behavioral and neural response times. Interesting, Bohannon et al (2017), presented a heterogeneous multi-agent system comprising computer vision, human and BCI agents, and showed that heterogeneous multi-agent image systems may achieve human level accuracies in significantly less time than a single human agent by balancing the trade-off between time-cost and accuracy. In such cases a human–computer interaction may occur in the form of button press if the confidence in the response of other, more rapid agents such as RSVP-BCI agents or computer vision algorithm is low for a particular sequence of stimuli.

Surveillance is the largest RSVP BCI system application reported in this review, reflected as such by the discussion length of this subsection (Sajda et al 2003, Erdogmus et al 2006, Gerson et al 2006, Poolman et al 2008, Bigdely-Shamlo et al 2008, Sajda et al 2010, Huang et al 2011, Cecotti et al 2012, Weiden et al 2012, Matran-Fernandez and Poli 2014, Marathe et al 2014, Rosenthal et al 2014, Yu et al 2014, Marathe et al 2015, Barngrover et al 2016, Cecotti 2016, Files and Marathe 2016).

In a surveillance application study carried out by Huang et al (2011) the targets were surface-to-air missile sites. Target and non-target images shared low-level features such as local textures, which enhanced complexity. Nonetheless target images were set apart due to large-scale features such as unambiguous road layouts. Another example of surveillance targets denoted by Bigdely-Shamlo et al (2008) is where overlapping clips of London satellite images were superimposed with small target airplane images, which could vary in location and angle within an elliptical focal area. Correspondingly, in Barngrover et al (2016), the prime goal was to correctly identify sonar images of mine-like objects on the seabed. Accordingly, a three-stage BCI system was developed whereby the initial stages entailed computer vision procedures, e.g. Haar-like feature classification whereby pixel intensities of adjacent regions are summed and then the difference between regions is computed, in order to segregate images into image chips. These image chips were then fed into an RSVP type paradigm exposed to human judgment, followed by a final classification using a support vector machine (SVM).

In the categorization, application type images were sorted into different groups (Cecotti et al 2011). Alpert et al (2014) conducted a study whereby five image categories were presented: cars, painted eggs, faces, planes, and clock faces (Sajda et al 2014). A second study by Alpert et al (2014), containing target (cars) and non-target image (scrambled images of the same car) categories, was conducted. In both RSVP experiments, the proposed spatially weighted Fisher linear discriminant–principal component analysis (SWFP) classifier correctly classified a significantly higher number of images than the hierarchical discriminant component analysis (HDCA) algorithm. In terms of categorization, empirical grounds were provided for potential intuitive claims, stating that target categorization is more efficient when there is only one target image type, or distractors are scrambled variations of the target image as opposed to different images all together (Sajda et al 2014).

Face recognition applications have been used to seek out whether a recognition response can be delineated from an uninterrupted stream of faces, whereby each face cannot be independently recognized (Touryan et al 2011). Two of the three studies evaluated utilized face recognition RSVP paradigm spin offs with celebrity/familiar faces as targets and novel, or other familiar or celebrity faces as distractors (Touryan et al 2011, Cai et al 2013). Cecotti et al (2011) utilized novel faces as targets amongst cars with both stimuli types presented with and without noise. Utilizing the RSVP paradigm for face recognition applications is an unconventional approach; nonetheless the ERP itself has been used exhaustively to study neural correlates of recognition and declarative memory (Yovel and Paller 2004, Guo et al 2005, MacKenzie and Donaldson 2007, Dias and Parra et al 2011). Specifically, early and later components of the ERP have been associated with the psychological constructs of familiarity and recollection, respectively (Smith 1993, Rugg et al 1998). There is thus substantial potential for the utility of the RSVP-based BCI paradigm for applications in facial recognition. In the future, RSVP-based BCI face recognition may be apposite in a real world setting in conjunction with security-based identity applications to recognize people of interest. Furthermore, Touryan et al (2011) claimed that, based on the success of their study, RSVP paradigm-based EEG classification methods could potentially be applied to the neural substrates of memory. Indeed, some studies show augmentation in the posterior positivity of ERP components for faces that are later remembered (Paller and Wagner 2002, Yovel and Paller 2004). That is to say, components of ERPs triggered by an initial stimulus may provide an indication of whether memory consolidation of the said stimulus will take place, which provides an interesting avenue for utilizing RSVP-based BCI systems for enhancing human performance. Based on these studies, it is clear that relatively novel face recognition paradigms have achieved success when used in RSVP-based BCIs.

RSVP-based BCIs that assist with finding targets within images to support clinical diagnosis has received attention (Stoica et al 2013), for example, in the development of more efficient breast cancer screening methods (Hope et al 2013). Hope et al (2013) is the only paper evaluated from the field of medical image analysis and hence described in detail. During an initial sub-study participants were shown mammogram images, where target lesions were present or absent. In a subsequent study, target red or green stimuli were displayed among a set of random non-target blobs. These studies facilitated comparison between 'masses' and 'no masses' in mammograms, and strong color-based images versus random distractors. Images were presented against a grey background in three-second bursts of 30 images (100 ms per image). A difference in the amplitude of the P300 potential was observed across studies, with a larger amplitude difference between target and non-target images in the mammogram study. The researchers attributed this to the semantic association with mammogram images, in contrast to the lack thereof in the colored image-based study.

The number of stimuli refers to the total number of stimuli, i.e. the same stimulus can be shown several times. An exception to this is RSVP-speller studies where researchers only report on the number of symbols used, i.e. 28 symbols—26 letters of the alphabet, space and backspace (Hild et al 2011). In the RSVP-speller studies reviewed, the number of times each symbol was shown was not explicit. RSVP-speller applications are likely to have significantly fewer stimuli than the other aforementioned applications as participants are spelling out a specific word or sentence, which only has a small number of target letters/words. The integration of language models into RSVP-speller applications enables ERP classifiers to utilize the abundance of sequential dependencies embedded in language to minimize the number of trials required to classify letters as targets or non-targets (Orhan et al 2011, Kindermans et al 2014). Some systems, such as the RSVP keyboard (described in Hild et al (2011), Orhan et al (2012a), Oken et al (2014)) display only a subset of available characters in each sequence. This sequence length can be automatically defined or be a pre-defined parameter chosen by the researcher. The next letter in a sequence becomes highly predictable in specific contexts, therefore it is not necessary to display every character in the RSVP speller. Studies show that target characters are generally displayed more than once before the character is selected. The length of a sequence and the ratio of target to non-target stimuli can have an effect on the typing rate/performance. In an online study by Acqualagna et al (2011), participants were shown 30 symbols that were randomly shuffled ten times before a symbol was selected through classification and presented on screen. Orhan et al (2012), carried out an offline study whereby two healthy participants where shown three sequences (consisting of 26 randomly ordered letters of the alphabet). The results of this study showed that the number of correctly identified symbols more than doubled when using three sequences instead of one sequence to identify targets.

Task complexity is enhanced by the multiplicity of target categories. In Poolman, et al (2008) there were two blocks of target presentations: a helipad block with a 4% target prevalence; and a surface-to-air missile and anti-aircraft artillery block with a 1% target prevalence. Additionally, in Cecotti et al (2012) the targets were 50% vehicles, 50% people, with 50% being stationary and 50% moving. Further to this, (Weiden et al 2012) demonstrated that presenting kinetic images during the RSVP paradigm as opposed to stationary images increased the performance of EEG-based detection, and that this is negatively correlated with the cognitive load associated with the presented stimuli. In RSVP-speller applications task complexity varies based on what instructions participants are given, e.g. (1) participants may be asked to 'spell dog'; (2) 'type a word related to weather'; (3) participants can be given a word bank containing 20 words and asked to 'spell a word found within this word bank'. Half of the RSVP-speller-based BCI studies evaluated involved user-defined sequence lengths (instructions 2 and 3) (Acqualagnav et al 2010, Hild et al 2011, Orhan et al 2012, Oken et al 2014), while the other half involved users being given a target word/sentence to spell (instruction 1). If a participant has to remember the sentence or how to spell a long or unfamiliar word this can increase the complexity of a task (i.e. dog is much easier to spell than idiosyncrasy) (Primativo et al 2016). Note however that these different complexities in instructions are only present for evaluation/training tasks with the RSVP-BCI spellers. For their real use, participants choose themselves what they want to spell. The RSVP-based text application allows the number of stimuli before a target stimulus be reduced (i.e. letters such as 'z' that are less commonly used can be shown less frequently).

Excluding RSVP-speller applications, as it is already known that they do not require the same number of stimuli as the other applications, the number of stimuli used typically varied between studies from approximately 800 in the surveillance application study by Sajda et al (2010) to 26 100 in a categorization application study by Sajda et al (2014). The most common target stimuli percentage range was 1–10% found in 61% of the studies reviewed, followed by 11–20% then  >20%. There are a number of studies that focus specifically on the percentage of target stimuli. In a study by Cecotti et al (2011), researchers investigated the influence of target probability when categorizing face and car images. In this study, researchers used spatially filtered EEG signals as the input for a Bayesian classifier. Using eight healthy participants, this method was evaluated using four probabilities of target stimuli conditions, i.e. 0.05, 0.10, 0.25, or 0.50. It was found that the target probability had an effect on the participant's ability to detect targets and on behavioral performance. The best mean AUC (0.82) was achieved using the 0.1 probability condition. The results showed that the percentage of targets shown in an RSVP paradigm has an effect on participants' performance. As number and percentage of target stimuli used can have an effect on the complexity of a task, it is important to keep the percentage of targets to  <10% to evoke the P300 and maximize detection rates. This was proposed to be in line with well-established P3 measures, whereby bigger gaps between target trials reduce peak latency and increase amplitude (Gonsalvez and Polich 2002).

A key factor of the RSVP paradigm is the rate of presentation, as the focus of this paradigm is presenting data at a rapid rate so that large datasets can be analyzed in short periods. The duration for which stimuli were presented varied from 50 to 500 ms (Sajda et al 2003, Touryan et al 2011, Cai et al 2013). The upper limits for the presentation time of stimuli during the RSVP paradigm is ill-defined in the literature; however we found 500 ms per image to be the maximum RSVP duration used across all RSVP studies. The duration of stimuli typically differs between applications. Table 3 shows that the most common duration of stimuli was between 100–199 ms per image. The quickest duration of 50 ms per image was used in a study by Sajda et al (2003) where two participants were asked to identify scenes containing people in natural scenes. In each trial, the duration of the stimulus presentation was decreased from 200 to 100 to 50 ms per image. The results of this study showed that both participants had reduced performance for faster stimulus presentations, i.e. 50 ms. This would suggest that the most suitable duration for RSVP-based BCI applications is 100–200 ms, to balance the trade-off between accuracy and speed.

Overall, these limited findings are suggestive of presentation rates of  >10 Hz being infeasible for identification of neural correlates that allow successful identification of targets. Despite the low a participant number in Sajda et al (2003), validation for this upper cut-off presentation rate may be provided by Raymond et al (1992), where the attentional blink was first described. An RSVP paradigm was undertaken whereby the participant must register a target white letter in a stream of black letters and a second target 'X' amongst this stream. It was found that if the 'X' appeared within ~100–500 ms of the initial target, errors in indicating whether the 'X' was present or not were likely to be made even when the first target was correctly identified (Raymond et al 1992). This is not to say that humans cannot correctly process information presented at  >10 Hz. Forster (1970), has shown that participants can process words presented in a sentence at 16 Hz (16 words per second). However, the sentence structure may have influenced the correct detection rate, which has an average of four words per second for simple sentence structures and three words for complex sentences. Detection rates improve when presented at a slower pace, e.g. four relevant words per second, with masks (not relevant words) presented between relevant words. Additionally, Fine and Peli (1995) showed that humans can process words at 20 Hz in an RSVP paradigm.

Potter et al (2014) assessed the minimum viewing time needed for visual comprehension using RSVP of a series of 6 or 12 pictures presented at between 13 and 80 ms per picture, with no inter-stimulus interval. They found that observers could determine the presence or absence of a specific picture even when the pictures in the sequence were presented for just 13 ms each. The results suggest that humans are capable of detecting meaning in RSVP at 13 ms per picture. However, the finding challenges established feedback theories of visual perception. Specifically, research assert that neural activity needs to propagate from the primary visual cortex to higher cortical areas and back to the primary visual cortex before recognition can occur at the level of detail required for an individual picture to be detected, Maguire and Howe (2016). Maguire and Howe (2016) supported Potter et al (2014) in that the duration of this feedback process is likely  ⩾50 ms, and suggest that this is feasible based on work done by Lamme and Roelfsema (2000). Explicitly, Lamme and Roelfsema (2000) estimated that response latencies at any hierarchical level of the visual system are ~10 ms. Therefore, assuming that a minimum of five levels must be traversed as activity propagates from the V1 to higher cortical areas and back again, this feedback process is unlikely to occur in  <50 ms. However, Maguire and Howe (2016) suggested a potential confound of Potter et al (2014), which was that pictures in the RSVP sequence, on occasion, contained areas with no high-contrast edges and hence may not have adequately masked proceeding pictures. Consequently, Maguire and Howe (2016) replicated the study rectifying the edges to ensure high-contrast covering the entire image. They were unable to find any evidence that meaning can be detected in an RSVP stream at 13 ms, or even 27 ms, per image but at 53 and 80 ms this is possible. Upon this basis, the limits of RSVP processing could be reduced to a minimum of ~20 Hz. Nonetheless, further study is needed to investigate the limits of human capability to rapidly distinguish target from non-target information, in comparison to the limit in detecting target related ERPs versus non-target ERPs at 20 Hz presentation rates.

In all three face recognition studies, each face image was displayed for 500 ms (Cecotti et al 2011, Touryan et al 2011, Cai et al 2013). In two of the studies there was no ISI (Cecotti et al 2011, Touryan et al 2011), and in the other an ISI of 500 ms was given to ensure ample time for image processing (Cai et al 2013). The speed at which face images were shown was reduced in comparison to the other RSVP applications. RSVP spellers most commonly use a duration of 400 ms; RSVP-spellers can benefit from slower stimulus duration with the incorporation of a language model to enable the prediction of relevant letters. The estimation of performance can be challenging in the RSVP paradigm when the ISI is small, as assigning a behavioral response (i.e. button press) to the correct image cannot be done with certainty. A solution to this problem is to assign behavioral responses to each image, therefore researchers are able to establish hits or false alarms (Touryan et al 2011). When two targets are temporally adjacent with a SOA of 80 ms, participants are able to identify one of the two targets but not both. SOA should be at least 400 ms and target images should not be shown straight after each other (Raymond et al 1992). Acqualagnav et al (2010), had a four factorial design looking at classification accuracy when the letters presented as no-color or color letters at either 83 or 133 ms with an ISI of 33 ms (Acqualagnav et al 2010). The number of sequence stimuli was presented for enhanced accuracy rate in selecting letter of choice. After 10 sequences ~90% mean accuracy was reached in 133 ms color presentation mode (100% for 6/9 participants). After ten sequences in 133 ms no color presentation mode ~80% mean accuracy was reached (100% in 3/9 participants). Whilst at presentation rates of 83 ms mean accuracy rate was ~70% and the there was no significant effect of color. This formulation is based on the chance rate of 3.33% (i.e. 1 in 30). This implies that cultured letters enhances performance accuracy but not past a certain speed of stimulus presentation.

There is likely a significant interaction between the difficulty of target identification and presentation rate. For example, the optimal presentation rate for a given stimulus set is highly dependent on the difficulty of identifying targets within that set (Ward et al 1997). Image sets with low clutter, high contrast, no occlusion, and large target size are likely amenable to faster presentation rates; while image sets with high clutter, low contrast, high levels of occlusion, with small target sizes will require slower presentation rates (Rousselet et al 2004, Serre et al 2007, Hart et al 2013, Liu and Kwon 2016). A more conclusive analysis of the effect of stimulus presentation duration for each application type could be derived by varying the presentation rate duration between 100, 200, and 500 ms, whilst other parameters remain fixed. With regards to temporal proximity of target images, 500 ms should be taken to be the minimum to maximize performance.

Another RSVP design aspect to be considered is stimulus size. There is a large variation in image sizes ranging from 256  ×  256 pixels in a categorization application to 960  ×  600 pixels in a surveillance applications. In general, surveillance applications use larger images than the other applications described. The most common image size used is 500  ×  500 pixels. This is only used in static surveillance applications and all surveillance studies using this image size achieved a high accuracy (>80%). The other applications used smaller image sizes such as 360  ×  360 pixels and achieved high accuracies (i.e. 91% and 89.7%). Therefore, it can be concluded that for surveillance studies, image size should be at least 500  ×  500 pixels, although for all other applications the image size may be smaller. A more complex task is where a target stimulus is presented in the background of a larger image eliciting the N2 ERP. Early components such as the P1 and N2 are sensitive to the spatial location of the stimuli (Saavedra and Bougrain 2012).

One issue with reporting only image size is that it is always relevant to the distance viewed from screen and its location on the screen with respect to the viewer, i.e. the visual angle. The visual angle is the angle an image subtends at the eye, reported in degrees of arc. In a study by Dias and Parra (2011) it was shown that participants performed best (90%) when the target stimulus was centered. Performance consistently decreased to 50% in all participants as target stimulus were placed further away from the center (4° of visual angle), this dropped further when target stimulus was placed at 8° of visual angle. Although performance drops significantly participants are still able to detect target stimulus shown in their peripheral visual field even at such rapid paces. Many papers report that the visual angle of the stimuli can have an effect on performance. As a general principle, targets must appear larger or be more distinct for detection at the outer edge of the visual field. The visual angle can thus be deemed the most important measure as it accounts for distance from screen, image location on screen and image size. Authors are therefore encouraged to report visual angle, as reporting image size alone is not useful without the availability of distance from the screen. For RSVP-speller studies, none of the papers found reported on the size of the image or font, however some reported the visual angle.

Many different types of target images have been identified within this review. The majority of research focuses on a two-class problem, i.e. detecting target images in sequences of non-target images that are completely different from each other. However, in real-life situations, non-target images are likely to share some of the same characteristics as target images (Marathe et al 2015). These presentation sequences appear to be more like moving images than static images. In Marathe et al (2015) a more complex surveillance task was carried out where, in the first task, participants were required to detect targets when targets are the only infrequent image whilst, in the second task, targets were presented with non-targets (i.e. the target image could be found in the background of a larger image). Participants were required to ignore everything else in the image, a much more difficult task, and consequently the amplitude of the P300 was reduced. The results of this study found that the introduction of the infrequent non-target stimuli in the scene yielded a substantial slowing of the reaction time. Surveillance applications commonly use stimuli that are more complex where trained participants, such as intelligence analysts, outperform novice participants, as they are able to give meaning to the stimuli. The RSVP-speller applications present their letters as images one at a time on screen (Hild et al 2011). Due to the nature of the RSVP paradigm, it is important that these letters are shown in a random order as participants pre-empting a target can have an effect on ERP responses (Oken et al 2014). Data categorization applications had the most variance between the different types of stimuli presented to a participant. However, these stimuli tend to be everyday items that participants can easily recognize.

All applications have certain requirements in terms of speed and type of images displayed, which, as outlined above, can influence the ERP and therefore also variations in performance as measured by detection accuracy. The signal processing framework plays an important role in being able to cope with variations in ERP and maximizing performance. There is a likely tradeoff between the design parameters used as described above and the level of sophistication built into the signal processing framework, which often varies across studies. Here we review some of the approaches applied.

5.7.1. Pre-processing.

5.7.2. Feature extraction.

5.7.3. Classification.

The parameters reviewed here have been selected because they have an effect on one or all of the following aspects of the RSVP paradigm: task complexity, stimulus complexity, stimulus saliency or information transmission. Performance within RSVP-based BCIs is measured as the participant's ability to correctly identify oddball images in a sequence. RSVP-based BCIs use two different measurements of performance such as accuracy (percentage of targets that are correctly identified using EEG) and ROC curves. 10% of papers assessed in this review did not report at least one out of these performance measures (ROC/percentage accuracy). The accuracies of the different studies need to be put in context, as all the reviewed parameters and other observed parameters i.e. number of trials and participants will influence study accuracy. In table 4 parameter recommendations are provided for designing RSVP-based BCIs within the different application types and these have been discussed thoroughly throughout section 5. In particular, table 4 suggests the parameters to use for each application, according to those leading to the best detection performances (accuracy or AUC) in studies comparatively. If no formal comparisons between parameters were available for a specific application or parameter, the most popular parameter values that yield good performances are mentioned.

Applying BCI systems commercially and outside the lab in real-world scenarios will ideally require the system to be robust during the execution of tasks of increasing difficulty. Section 5 summarized the five applications areas that have been studied to the greatest extent in the context of RSVP-based BCIs. Specifically, this section tackles intra-application comparisons of various aspects of the papers that met the inclusion/exclusion criteria. A few of the papers found in this review carried out more than one study in different application types. The most common type of application found was surveillance applications, followed by RSVP-speller applications and categorization applications; after this were face recognition and lastly medical applications. Although there is are a relatively limited number of studies, the design parameters and the focal points of different applications vary widely.