P300 speller and LDA accuracy

Working with OpenViBE signal processing scenarios and doing scenario/BCI design
stede
Posts: 10
Joined: Tue Dec 06, 2011 9:37 pm

P300 speller and LDA accuracy

Post by stede »

Hi there,

First of all big thanks for developing OV, it's really powerful and instructive. I do have a really simple, if not stupid question regarding the callibration of the p300 speller scenario. When using LDA I get quite reasonable performances between 65 and 85%, depending on the subject, the EEG system and the features selected, all fine so far. But what exactly does the LDA accuracy reflect here (I think I do understand LDA and cross-validation), is it related to the number of letters exclusively (6x6), the number of letters used in the training session (say, 10), or also related to the number of repetititons (number each row and column is repeated per letter)? I use no epoch averaging (that is, I use single trials), and we run 6 repetitions (and use the voting classifier with 6 repetitions as well, which is probably irrelevant for the classifier training question, it just comes to my mind, as we use the modified online scenario for offline classifier training). A related question when running P300 speller online: What exactly would be chance performance with a 6 x 6 speller, 6 repetitions and no averaging? Any help, or link to the corresponding documentation, is greatly appreciated.

Thanks much,
stede

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: P300 speller and LDA accuracy

Post by lbonnet »

Hi stede, welcome on board!

The LDA performance here corresponds to the detection of one flash to be P300 or not. On a 6x6 matrix flashed by row and column, the chance level is 5/6 = 0.76 (if you always classify Non-P300, you're right 5/6 times).

With a relatively low performance (let's say 85%) you can achieve good letter selection by higher number of repetitions. We use 12 repetition as a default, to-be-sure setup.
I use no epoch averaging (that is, I use single trials)
I'm not sure we have the same definition of single trial. For me it means you only try to select a letter with one repetition (1 flashing sequence).
If you remove the Epoch Averaging from the scenarios and connect directly the Stimulation Based Epoching to the feature aggregator, I guess you are giving to the classifier a big feature vector (one epoch of 600ms contains 0.6*samp.freq/4) instead of only one value (the average).

I hope this helps a bit :)

Laurent
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

stede
Posts: 10
Joined: Tue Dec 06, 2011 9:37 pm

Re: P300 speller and LDA accuracy

Post by stede »

Dear Laurent,

Thanks very much for your helpful reply. Just to be sure I get it right:
lbonnet wrote: The LDA performance here corresponds to the detection of one flash to be P300 or not. On a 6x6 matrix flashed by row and column, the chance level is 5/6 = 0.76
I guess you mean 1/6 = .17 chance level that one (out of six) flashes evokes a P3, correct?

lbonnet wrote: With a relatively low performance (let's say 85%) you can achieve good letter selection by higher number of repetitions. We use 12 repetition as a default setup.
We played with the number of repetitions, but are not entirely sure how the data across repetitions are handled. Does the voting classifier (assuming 12 repetitions) vote based on the average of the 12 values, or the maximum value for each row/column?
lbonnet wrote:
I use no epoch averaging (that is, I use single trials)
I'm not sure we have the same definition of single trial. For me it means you only try to select a letter with one repetition (1 flashing sequence).
If you remove the Epoch Averaging from the scenarios and connect directly the Stimulation Based Epoching to the feature aggregator, I guess you are giving to the classifier a big feature vector (one epoch of 600ms contains 0.6*samp.freq/4) instead of only one value (the average).
We mean the same thing here. I was just refering to the Epoch Averaging with the Epoch count set to 1. From what you write this should result in averaging all frames within each trial, but no averaging across trials. I got a bit confused by the documentation of this box... If I would set the Epoch count to, say, 2, all frames within the epoch window would be averaged, and the average across two adjacent trials (corresponding to the same stimulation) would be calculated, correct?

Thanks much,
setde

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: P300 speller and LDA accuracy

Post by lbonnet »

Hi setde,
I guess you mean 1/6 = .17 chance level that one (out of six) flashes evokes a P3, correct?
I meant the chance level for the classifier is 5/6. If you take a stupid classifier that always classifies a trial to be non-p300, it will be right 5/6 times, as only 2 flashes out of 12 (for 6 rows and 6 columns) should trigger a P3 response. The k-fold test performed by the classifier trainer at the end of the training gives you a performance related to the number of time the classification was right on a test set of trials.
We played with the number of repetitions, but are not entirely sure how the data across repetitions are handled. Does the voting classifier (assuming 12 repetitions) vote based on the average of the 12 values, or the maximum value for each row/column?
The row voting classifier take the classification state of all row classifiers. For a LDA, this classification state is the distance to the hyperplan computed (value < 0 means class1; value > 0 means class2).
To keep it simple, the voting classifier sums all classification states received for every input and choose to send the stimulation corresponding to the input with highest score.
I was just refering to the Epoch Averaging with the Epoch count set to 1. From what you write this should result in averaging all frames within each trial, but no averaging across trials. I got a bit confused by the documentation of this box... If I would set the Epoch count to, say, 2, all frames within the epoch window would be averaged, and the average across two adjacent trials (corresponding to the same stimulation) would be calculated, correct?
Yes indeed ! A Block Average with only 1 epoch does nothing at all, it's the very same block. But if you take 2, you average every 2 trials on the "same flash".

NB: All the Epoch Average boxes in the scenario are configured with the same config file ({Path_Samples}/bci/p300-speller-xDAWN/p300-epoch-average.cfg). Changing this file will change every boxes in the scenarios, in case you want to try.

Cheers

Laurent
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

fabien.lotte
Posts: 112
Joined: Sun Mar 14, 2010 12:58 pm

Re: P300 speller and LDA accuracy

Post by fabien.lotte »

lbonnet wrote:
I guess you mean 1/6 = .17 chance level that one (out of six) flashes evokes a P3, correct?
I meant the chance level for the classifier is 5/6. If you take a stupid classifier that always classifies a trial to be non-p300, it will be right 5/6 times, as only 2 flashes out of 12 (for 6 rows and 6 columns) should trigger a P3 response. The k-fold test performed by the classifier trainer at the end of the training gives you a performance related to the number of time the classification was right on a test set of trials.

I didn't realise before that the classification accuracy displayed in the P300 speller was the single trial classification accuracy (detection of P300 or not). Since this is the case, I think displaying a classification accuracy is not really an appropriate performance measure, because, as you mentioned, the data is highly unbalanced (5 times more non-P300 than P300). Indeed, because of this unbalance problem, the classification accuracy does not mean anything: a stupid classifier which always outputs non P300 will be 83.3% right but will be totally useless, whereas a classifier which correctly detects all P300 but miss 1 out of 5 non-P300 will be much more useful but will also have an accuracy of 83.3%... I would suggest to display an alternative measure such as the area under the ROC curve or the percentage of True Positive and False Positive, which would be much more informative in this case (these measures are designed to deal with unbalanced data). Would this be possible?

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: P300 speller and LDA accuracy

Post by lbonnet »

It's OpenViBE ! everything is possible :)

This is a good idea.
We could compute for every sets in the k-fold test the TP/FP rates quiet easily . I'm not familiar with ROC curves though.

I may be over-optimistic but in my opinion it's not very complicated.
Not on the top priority list I guess.

L
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

fabien.lotte
Posts: 112
Joined: Sun Mar 14, 2010 12:58 pm

Re: P300 speller and LDA accuracy

Post by fabien.lotte »

Indeed, just computing the TP/FP for each fold of the CV would be enough and should be very easy to do. It's indeed not very important but displaying the single trial accuracy instead is rather misleading...

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: P300 speller and LDA accuracy

Post by lbonnet »

I developed the patch rapidly. It computes the Accuracy (mean + standard deviation on kfold test); plus True Positive and False Positive rates (means + standard deviations over the kfold test).
I basically take the first class as the "Target".
The classifier trainer box is theoretically compatible with all possible classifiers. For a P300/Non-P300 classifier I see what TP and FP means and why they are interesting, but for multiclass classifiers there is no TP and FP rates, am I right ? Or I must compute it for every class ? (and is it really necessary to do so?)

Sorry. Statistic newbe question...

Laurent
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

stefan
Posts: 20
Joined: Thu Jul 28, 2011 2:14 am

Re: P300 speller and LDA accuracy

Post by stefan »

Hi setde,

I'd like to thank you for your comments here that I support. I'm also working with P300 speller and face the same problem. I'm wondering if you have used the online scenario, and what is you reflection about it?
I got accuracy range of 60-85% during the training. However, I did not get any letter right when I used the online scenario :(
although, the system was trained foe many sessions and tested repeatedly.
The performance on the training was meaningless in my case.
Have you come across this problem? what EEG device do you use?

thank you and best regards,

stede
Posts: 10
Joined: Tue Dec 06, 2011 9:37 pm

Re: P300 speller and LDA accuracy

Post by stede »

Hi Stefan
You have to play with the settings a little bit to get it up and running online. Use a modified online session for the recording of the training data, otherwise there will be fundamental problems with the stimulation markers (for details pls consult the ov experts, I am ov newbie. We use brainamps but I would think that the type of amp is not crucial. Just make sure that you select appropriate channels and a reasonable number of repetitions (with the default 12 you should be able to spell with reasonable accuracy, albeit slowly).
Hope this helps,
stede

fabien.lotte
Posts: 112
Joined: Sun Mar 14, 2010 12:58 pm

Re: P300 speller and LDA accuracy

Post by fabien.lotte »

lbonnet wrote:I developed the patch rapidly. It computes the Accuracy (mean + standard deviation on kfold test); plus True Positive and False Positive rates (means + standard deviations over the kfold test).
I basically take the first class as the "Target".
The classifier trainer box is theoretically compatible with all possible classifiers. For a P300/Non-P300 classifier I see what TP and FP means and why they are interesting, but for multiclass classifiers there is no TP and FP rates, am I right ? Or I must compute it for every class ? (and is it really necessary to do so?)

Sorry. Statistic newbe question...

Laurent
Great, thanks for the patch!

Just a remark, In the case of the P300, the target class should the P300 (and not the nonP300), since we want to estimate the detection performance of the P300. Maybe it would be interesting to be able to specify the target class in the classifier box?

Otherwise, FP/TP are indeed mostly relevant for P300/nonP300 classification, but not more informative than accuracy in cases where the classes are balanced.
It could be useful for asynchronous BCI, to estimate the detection performance of the mental task used (in which case the data are generally highly unbalanced since there are much more non-control-state data than control-state data). But I don't think you need to compute it for every class, in case of multiclass classification. If you want to display a generic measure whatever the kind of classifier, you may display the confusion matrix. Everything (accuracy, FP, TP, etc.) can be derived from this matrix.

I hope this helps!

best,
Fabien

stede
Posts: 10
Joined: Tue Dec 06, 2011 9:37 pm

Re: P300 speller and LDA accuracy

Post by stede »

Thanks a lot for fixing it.... but how do I get the patch?
stede

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: P300 speller and LDA accuracy

Post by lbonnet »

Ok I see. I'll add the class (input) to be considered as "target" in the box settings.

@ stede : you cannot right now :) I patched but did not commit as I wanted fabien's feedback first on the statistics...

I do the last modifications today and commit to the SVN trunc. You will be able to get it using the latest sources from the SVN repository (see build instructions then).

Laurent
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: P300 speller and LDA accuracy

Post by lbonnet »

I'll be a bit late for that sorry... I got some work to rush right now.
I'll do it next monday !

cheers
L
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: P300 speller and LDA accuracy

Post by yrenard »

Hello guys,

sorry I'm taking the discussion very late as I got the notification of this evolution in the SVN commit mailing list.

I'd like to point out two issues and propose one alternative solution here :
  • Computing TP/FP supposes 2-class classifiers. Even if OpenViBE does not come with multi class classifiers for now, the classifier trainer box has been designed to handle multiple class classifiers and some users may use such classifiers making this TP/FP computation irrelevant
  • Adding a new setting to this box to choose which class is the target breaks the compatibility with existing scenarios, and even causes unpredictable behavior for scenarios that use classification algorithms which already add new settings (such as SVM for instance)
In my opinion, the best approach would be to compute the confusion matrix for each partition and print this to the Trace log level for interested users. The confusion matrix gives all the necessary information to know how good a classifier is.

More generally speaking, it is important to keep the boxes generic and not to make them specific to a particular problem in order to be able to re-use them later in different problems :)

Hope this helps,
Best regards,
Yann

PS: Laurent, I'll wait your feedback to know if I should update my scenarios or not ;)

Post Reply