P300: CoAdapt P300 Speller (OpenViBE v0.18 only)

Posted on July 29, 2014 by Loïc Mahe
  • NB: Document based on OpenViBE 0.18.0

p300sceen

The CoAdapt P300 speller is currently working only in the 0.18 version of OpenViBE. The contributing authors are currently developing an even more robust P300 speller that is also easy to use.

CoAdapt P300 Stimulator and protocol – tutorial
by Maureen Clerc, Dieter Devlaminck & Loïc Mahé @ Inria Sophia Antipolis-Méditerranée.

Code developped by Inria & Inserm, funded by CoAdapt project (ANR-09-EMER-002).

Overview

This tutorial describes a P300 speller protocol based on the CoAdapt P300 Stimulator, as featured in the scenarios under coadapt-p300-stimulator. For a technical sketch on how the involved classes interact, see here.

The CoAdapt P300 Stimulator is an application independent from OpenViBE,
that handles the following features of a P300 speller system:
– the keyboard display;
– the flashing of the keyboard elements;
– the element selection.

References:
E. Thomas, M. Clerc, A. Carpentier, E. Daucé, D. Devlaminck, R. Munos, Optimizing P300-speller sequences by RIP-ping groups apart, 6th International IEEE/EMBS Conference on Neural Engineering (NER) 2013, DOI 10.1109/NER.2013.6696120

E. Thomas, E. Daucé, D. Devlaminck, L. Mahé, A. Carpentier, R. Munos, M. Perrin, E. Maby, J. Mattout, T. Papadopoulo, M. Clerc. CO-ADAPT P300 speller: optimized flashing sequences and online learning. Int. Brain-Computer Interface Conference, Graz, Austria, 2014.

Compilation

This application introduces a few new dependencies. They are automatically installed on Linux (via the linux-install-dependencies script) and you can choose to install them on Windows via the installer. On Windows, the libraries are downloaded from the inria server but be aware that only 32bits version compiled against Visual Studio 2010 are available. The libraries used are :

  • GLFW : is used to initialized the OpenGL context of the display. It is the only mandatory dependency of this application;
  • Presage : is the word prediction library. If it is not present, the word prediction module is not compiled and you will not be able to use it;
  • Inpout32 : on Windows, this library allows the use of hardware tagging (provided your computer and your acquisition device can communicate through parallel port).

System description

The CoAdapt P300 Stimulator is designed to be run in combination with
– the OpenViBE acquisition server;
– the OpenViBE designer (for signal processing and classification).

These three software components communicate thus:

p300_communication

Tagging ensures correct determination of the timing of flashes, with two possible modes:

  •  software tagging (default) –  using boost interprocess queue (see openvibe-external-stimulation-connection-example)
  • hardware tagging (to be used preferably  –  provided that the amplifier can receive stimuli and that the computer handles parallel port connection)

Shared memory is used for communication from the designer to the P300 stimulator, also using boost. For this to work, a  Shared Memory Writer box must be running in the designer, and it is mandatory to launch the designer before the P300 stimulator because it is the box that creates the shared memory. For now it is the only mean of communication between the designer and the CoAdapt stimulator and being a shared memory, it is mandatory that the designer and the CoAdapt stimulator are run on the same computer.

Configuration files

Configuration parameters are set in several files
(in dist/share/openvibe/applications/externalP300Stimulator ) :

  • interface-properties.xml defines the tagging mode, the keyboard layout, the photodiode area activation, the use of fullscreen, etc … See file comments for detailed explanations;
  • stimulator-properties.xml defines the flashing time parameters and evidence accumulation : number of flashes per symbol, flash and inter-flash duration, early stopping and its threshold;
  • P300AcquisitionConfig.conf defines tokens used by the two above xml files, and handled by the OpenViBE configuration manager.

Operation modes

There are three main operation modes:

  • Calibration mode where target symbols are presented, and the user does not receive any feedback;
  • Copy mode where target symbols are presented, and the user receives feedback after each symbol presentation;
  • Free spelling mode where there is no target symbol presented, and the user receives feedback after each selected symbol.

Keyboard layout

p300displaycomponents

The Keyboard layout is made up of several areas:

Keyboard area where symbols are displayed and flashes occur.
Target area (calibration and copy mode) where target symbols are kept on display – the last being the one currently attended.
Feedback area (copy and free mode) where selected symbols are kept on display.
Diode area mainly for timing tests: a photodiode can be attached to this area of the screen. If the diode area is enabled (in interface-properties.xml ), it will light up simultaneously with  each group of symbols flashed.

P300 speller protocol

Setting up the Acquisition Server :

The tagging mode for the communication from the CoAdapt P300 stimulator to the Acquisition Server must be set in interface-properties.xml :

  • hardware tagging is the preferable mode if the amplifier can receive stimuli and the computer can handle parallel port connection. In this case it is advised not to use drift correction. (this can be set in the acquisition server preferences)
  • software tagging is the default mode. Because it is less precise than the hardware tagging,  it is advised to use drift correction.

Calibration mode

In Calibration mode, the user counts flashes of letters spelt out from a predefined word. In P300AcquisitionConfig.conf :

CoAdaptP300_AcquisitionMode=Cal
CoAdaptP300_WordToSpell=TEST
CoAdaptP300_NrOfRepetitions=20
CoAdaptP300_StimulatorMode=Online

Each letter of the word will flash the same number of times (set by NrOfRepetitions). Any word can be used as WordToSpell, but keep in mind that the calibration phase must generate enough data to train a reliable classifier.

– Launch the OpenViBE acquisition server. Be sure to enable external stimulations for the tagging.
– Launch the OpenViBE designer and the p300-coadapt-online-with(out)-monitoring.xml scenario.
– Launch the CoAdapt P300 stimulator.

– When the user is ready, press “S” (on real keyboard…) to start the calibration phase.

With the monitoring on, a display will appear. The right panel shows the EEG, and the left panel, green (resp. red) curves representing the EEG of one electrode, averaged over target (resp. non-target) trials.

p300erp

– Press ‘ESC’ to stop the CoAdapt P300 stimulator (at the end of the calibration word, or anytime).
-After having stopped the CoAdapt P300 stimulator, you may proceed to the training phase. You must quit the stimulator (by pressing escape) because doing so causes the stimulator to send the OVA_StimulationId_ExperimentStop via tagging to the designer, which in turn will write it to the recorded file. This stimulation is used by the training boxes to launch the training.

The EEG files are recorded in openvibe native format along with flash-group-definition_$core{date}_$core{time}.csv files that contains the group of letters flashed. In those files, each line correspond to a flash with 1 being the letters flashed and 0 the one not flashed. For offline analysis later you may want to keep those in addition to the EEG data, and the different configuration files as well.

– In P300AcquisitionConfig.conf put as token TrainFile the name of the file just recorded  at the end of the calibration phase. You will have to quit and launch again the designer for the change in the configuration file to be taken into account.

CoAdaptP300_TrainFile=

– Run the p300-coadapt-train-xdawn.xml scenario  to learn X-Dawn filters
– Run the  p300-coadapt-train-classifier.xml scenario to train the LDA weights
You may fast forward them, the OVA_StimulationId_ExperimentStop (as mentionned before) will launch the training and when this training is done, the scenarios will stop automatically.

Replaying signals:

The p300-coadapt-replay.xml displays the signals from the training file, filtered by X-Dawn, and averaged across trials. The green curve (averaged over target trials) should be separated from the red curve (averaged over non-target trials).

p300erp2

This scenario also allows to redisplay what the user saw, with these additional steps :
– Configure the P300AcquisitionConfig.conf as

CoAdaptP300_AcquisitionMode= Cal or Copy  #(the mode you used when recording)
CoAdaptP300_StimulatorMode=Replay
CoAdaptP300_FlashGroupDefinitionFile=   ... 
             #(the csv file attached with the ov file you replay)
CoAdaptP300_FlashMode= file  #(the flash groups are retrieved from the above file)

-Launch the scenario then launch the CoAdapt stimulator. The application will read the group of letters from the csv file and display the flashes as the user saw them [though as noted in the scenario, timing will be a little off]. Note that the shared memory mayl sometimes skip a letter even at normal speed, so fast forward is not advised.

You are now ready to use the other modes.

Copy mode

A good way to see if the system is well calibrated is for the user to reproduce a target using Copy mode.

CoAdaptP300_AcquisitionMode=Cop
CoAdaptP300_WordToSpell=COPYWORD
CoAdaptP300_NrOfRepetitions=20
CoAdaptP300_EarlyStopping=True 
CoAdaptP300_StopCondition=0.9

When EarlyStopping is set to True, each symbol flashes between 2 and NrOfRepetitions times, and stops flashing as soon as the system has enough evidence on the symbols.
CoAdaptP300_StopCondition is an Early Stopping threshold. It should be between 0 and 1.  A high value produces more flashes, and less errors (speed vs. accuracy tradeoff). Typical values range from 0.9 to 0.99.

Free mode

In Free mode, the user can write any combination of symbols. No target symbol is displayed.

CoAdaptP300_AcquisitionMode=Fr
CoAdaptP300_WordToSpell=

It is better to leave the CoAdaptP300_WordToSpell token empty, otherwise the system will stop as soon as the number of letters in WordToSpell is achieved.

Notes

Keyboard layouts :

Keyboards generally come in two versions: alphabetical (abc-) or azerty (French ordering).
Keyboards with ‘-undo’ support an ‘UNDO’ key to revert the last action
Keyboards with ‘-wp’ support Word Prediction: extra slots that can hold predicted words.

Customizing keyboards :

A keyboard is composed of keys displayed on a grid. At the beginning of the file, the default properties for keys are defined.
A key can have several states :
noflash is the default state
flash is when the letter is in the current group.
target : at the beginning of a new selection, indicates which symbol to target
correct_feedback : when the seleted symbol is the target
wrong_feedback : when the selected symbol is not the target

Each of these states has different properties (color, size, …).
Then we define the positions and sizes of the different areas.
You are free to set the keys as you want in the keyboard area’s limits.

Each key corresponds to an action. The most common action, ‘write’, means, upon selection, to copy the content into the ‘Feedback area’ of the Keyboard display.
Other possible actions are ‘undo’ (revert ‘Feedback area’ to previous state) and ‘wordprediction’ which indicate the key is to be filled with the most probable words (if any).

Word prediction :

Provided presage is installed on your system, the word prediction module is compiled and you can use it. To use this feature you will need a keyboard layout with slots for predicted words. In general, these keyboard have -wp- (for word prediction) in their names. You also need a dictionary generated by presage. You can create one yourself using presage or use the ones we provide :
french dictionary
english dictionary
Finally, set the name of the dictionary you want to use in presage.xml

This entry was posted in Deprecated and old documents. Bookmark the permalink.