When running an EEG-based BCI session – especially your first one – it can happen that the observed accuracy of the BCI leaves a lot to be desired. This can be normal and within the limitations of the current state of the technology. The accuracy can also be very low or essentially random. This may suggest a problem in the BCI system. In this short overview we briefly recall what is meant by accuracy and then discuss some error sources that can cause the accuracy to be lower than it could be. We present some ideas what to do in different cases.
Disclaimer: This post reflects the opinions of the writer(s). Expert opinion may vary between experts.
What is accuracy?
Usually BCI system accuracy is measured by counting the number of intended (correctly classified) choices divided by the total number of choices made. The resulting ratio is called accuracy, although other quality measures are possible, such as a bit rate, which also takes into account how long did it take to make each choice.
Let us assume for the moment that we have “a decent BCI system” (here a system consists of user+hardware+software+DSP pipeline) with a two-class balanced design. This roughly means that the user wants to make an equal numbers of choices between two classes. This could be the case in ‘left and right’ motor imagery, for example. In such a case, the expected random accuracy by tossing a coin is 50%. The coin-tosser may nevertheless get lucky, and if the amount of tosses is small, obtain a better than 50% accuracy by chance (see e.g. Müller-Putz & al: “Better than random? A closer look on BCI results”, 2008). If the design is unbalanced – such as in P300 where most flashes are ‘nontargets’ – the yardstick changes accordingly.
A casual off-the-sleeve estimate could be that a BCI system – when working normally – might have an accuracy between 70-90% for a balanced design. If the obtained accuracy is lower than that, or otherwise considered unacceptable, it becomes necessary to try to address the situation. This can be attempted by tracing down the error source(s) and addressing the problems found.
Different error sources
The low accuracy can be due to various problems in different ‘components’ of the BCI system: the user, the hardware, or the software. Here we outline some sources of errors, how to spot them, and give some basic advice what could be tried to address them.
For attaining a thorough understanding of the challenges of BCI systems, we suggest to read a textbook on the subject and to carry out empirical experiments.
User-related (human) errors
One potential source of challenges to the BCI system is the user (in the sense that the BCI system is having hard time with the specific users’ signal). The user-related errors can be due to e.g.
- Inherent variability in user physiology. Users have different head shapes, cortical volumes and brain folding, and hence the electrical signal transmission from the generative sources inside the brain volume to the surface are subjected to different volume conduction transforms in different users. Thus, physiological differences is one reason why different users have very different EEGs. The volume conduction can be very destructive to the signal and generally works as a strong lowpass filter (see e.g. Nunez: “Electric Fields of the Brain: The Neurophysics of EEG, 2nd Edition”, 2005).
Fixing it: There may be not much that can be done with this source of errors; if the differences between the electrical activities related to the different classes are not sufficiently reflected in the EEG due to physiological reasons. There is no way an EEG-based interface can do better than what is allowed by the data.
- User skill, motivation and state. These can affect the BCI results. This is especially the case with BCI paradigms such as motor imagery which require the user to actively carry out some mental activity intensely enough. This activity can be difficult and tiring for the user. If not well instructed, the user may also be using other mental strategies than intended by the BCI designer (e.g. visual imagery instead of kinesthetic imagery).
Fixing it: In classical motor imagery experiments, the users underwent long and repeated training sessions, and the ultimate accuracies reported were those that these users obtained after hours practice, training over several days. Not all users are able to learn to control BCI, even after training. Making the BCI session/feedback engaging and having a motivated, non-tired, and well-tutored user can help.
Bottom line: If some user doesn’t attain acceptable results and other issues have been ruled out (e.g. the BCI has been validated to work on an acceptable level with users in general), the choices may be such as 1) try to use another assistive technology for the user 2) try another user — it depends on what the goal is.
Acquisition related issues
- Electrode positioning. EEG correlates of different BCI paradigms are typically localized differently on the scalp. The electrodes need to be placed appropriately for good results.
Fixing it: Make sure the positioning you use makes sense wrt the BCI paradigm used. Check the literature.
- Electrode conductivity. Electrodes need to have good conductivity or your data will consists only of noise.
Fixing it: Check the impedance values. They should be low enough. Consult the literature for acceptable numbers. Check that the signal visually ‘looks’ like EEG. Typical checks include e.g. verifying that eye- and jaw artifacts (blink, clench) appear in the signal, as well as checking for a strong ‘alpha’ frequency component on electrodes placed on top of the occipital lobe when the user has eyes closed. Note that the artifact test is not sufficient — artifacts may appear even if the amplifier is not working correctly. Check that no channel visually differs a lot from the other channels. If some do, this typically indicates an error of electrode conductivity, as EEG channels tend to be very correlated or anticorrelated and should subsequently have similar ‘gist’. Wild hairstyles with which the electrode tends to float above the skin or move during the session are a potential source of trouble. It is especially important to get the conductivity of any potential reference or ground electrodes good. For wet electrodes, impedance problems are in the first place often fixed by adding enough gel. Broken electrodes or wires need to be replaced.
- Electrical interference. EEG hardware designs usually try to cancel outside interference from the recording environment. Still some amplifiers and headsets can be subject to it.
Fixing it. Make sure that mains-connected EEG amplifier outputs are clean from the the 50hz/60hz power noise. This can be done in software using a notch filter. Additionally, make sure the electrode cables are at a good distance from electrical components that could cause interference. Such can be for example cheap power transformers (e.g. for powering USB drives), microwave ovens, etc. If uncertain, unplug.
- The amplifier. The amplifier may be working incorrectly. A quick way to test this is to switch to another amplifier which is known to be working. A more involved approach to investigate amplifier-related issues is to have a signal generator (physical hardware) making known low voltage signals and feeding these through a physical head phantom that models the volume conduction. The sampled signals are read out from the amp and compared to what is expected based on the controlled signal generation. Alternatively, the signal source can be directly connected to the electrodes. In both cases, be careful not to fry the amplifier with too strong voltages. In general this approach requires help from the trained professionals of the electrical engineering department. It shouldn’t be tried at home. In case of high-end amplifiers, you should request the manufacturer to test that the amplifier operates within normal parameters.
Fixing it: If the amplifier is found faulty, its time to send it back to the manufacturer for repairs or buy a new one.
- Bluetooth / wireless. If your headset relies on wireless communication, this can be fickle and be harmed by objects such as tables, monitors, user’s own body, etc getting on the signal transmission path.
Fixing it: Try to ensure a clear, unblocked airpath between the transmitter and the receiver. If possible, use cable instead of wireless.
- The acquisition computer. Some BCI paradigms such as P300 may be very sensitive to timing. E.g. in P300, the event markers need to be very precisely aligned in the recorded EEG signal.
Fixing it: Make sure the acquisition computer is not doing any ‘funny stuff’ during the BCI session. A pedantic experimenter will always do the following before a BCI session: disable scheduled tasks, make sure virus scan or other harddisk grinding operations do not start or run in the background, unplug the internet, and close all other applications. Make the CPU run at ‘Performance’ setting (‘Power options’ in Windows) so the CPU doesn’t pump its frequency up and down during the session.
Software/DSP related errors
- The software platform bugs. Although many BCI platforms are rigorously tested, due to the ‘normal BCI accuracy’ tending to be less than perfect, it is unfortunately easy to overlook some errors: they can get lost between the other issues.
- The DSP pipeline that processes the signal may have bugs of its own related to its design.
Fixing it: Troubleshooting instructions can be followed to discover potential issues in the system and the pipeline. One strong debugging approach is using simulated data. Simulation has the benefit that you can know what is in the data since you designed how it was generated, and hence you can spot when your model behavior differs from what you expect. OpenViBE can do some simple artificial data generation. Alternatively, you can try e.g. simBCI. If bugs are found in the BCI platform or its components, it is probably best to file a bug report to the developers of the platform.
- The used parameters in the pipeline may be unoptimal for the user in question.
Fixing it: Parameters can be tuned by hand or with machine learning. Due to per-user differences, in BCI it is common practice to fit new parameters/classifiers in each session, as even the same brain can change over time. It is also difficult to keep the electrode positions always the same. Different filters / machine learning algorithms / models could be tried to see if one obtains better accuracy. One way to try to improve the parameter aspect is try to train the models with more recorded data, or only with the latest data (for ‘nonstationarity’ of the brain; the classifier either needs to be invariant or to change over time).
- Firewalls. The firewall of your system may prevent BCI components (AS, Designer, BCI application) from communicating. Make sure this is not the case. Depending on the connection (e.g. LSL, TCP/IP, VRPN, …) you may find third party tools to check that some connection point is receiving or sending data successfully.
Deficiencies in BCI accuracies can be due to various reasons. Finding the reasons and addressing them can be a lot of research-like work involving meticulously testing with different users, hardware, pieces of software and DSP solutions. Not many of the abovementioned error sources can be ruled out without doing proper testing of each of them.
Do not hesitate to post comments, further ideas and solutions on the OpenViBE forum.