Potential bug with Temporal Filter - GPL

[GrahamHealy]

Not sure if it's a bug but...

If I provide errornous data (I wasn't looking at the screen for training acquisition) to the p300-speller-3-train-classifier.xml scenario with the temproal filter options of highpass set at 0hz it will score at > 80%.
I tried 3 different filters and the same thing.

Also, is there any way to get ROC-AUC as an output metric?

[lbonnet]

Hi GrahamHealy !

If I provide errornous data (I wasn't looking at the screen for training acquisition) to the p300-speller-3-train-classifier.xml scenario with the temproal filter options of highpass set at 0hz it will score at > 80%.

In the P300 speller, the classifier has 6 possibilities for the row/column selection : 5 non-target, 1 target. Thus a random classifier (target or non-target) has an accuracy of 5/6 = 83%.
If you give erroneous data to the classifier trainer, you will of course score poor accuracy, and this is what happened here with your 80% : it is below the chance rate. And I am not sure it is related to the temporal filter.

Also, is there any way to get ROC-AUC as an output metric?

I am not familiar with ROC-AUC. Can you please send us a pointer to this metric ?

Best regards !

Laurent

[GrahamHealy]

Oh, that makes sense of course!
but if you set the the highpass at even 1hz the accuracy will drop to <60% (far away from the 83% likely indicating it has gathered something). This is what I found strange and unexpected! I am using the svn snapshot (openvibe-0.7.0-svn2254) just before the latest release which I built using the included scripts. I did modify some elements of the scenarios to include a channel selector. I am away from the machine and the data I recorded for but I will try this again and post outputs, along with scenario files.

As for ROC-AUC there's the wikipedia page:
http://en.wikipedia.org/wiki/Receiver_o ... acteristic
and a tool to do it in libsvm:
http://www.csie.ntu.edu.tw/~cjlin/libsv ... binary_svm
The algorithm is simple for it assuming you can rank test outputs in an ordinal fashion, and with cross validation it's acceptable to average the AUC results...
I found this to be a good paper outlining concerns in averaging ROC graphs and such:
"An introduction to ROC analysis psu.edu [PDF]T Fawcett - Pattern recognition letters, 2006 - Elsevier"
There's rakes of other litrature from the medical image perception community on ROC stuff available on google scholar too.

I hope that helps!

Looking forward to using openvibe more in the coming weeks!