MI Classification Accuracy changes with OpenVibe Versions

Working with OpenViBE signal processing scenarios and doing scenario/BCI design
Post Reply
id1016
Posts: 2
Joined: Thu Nov 23, 2017 2:01 pm

MI Classification Accuracy changes with OpenVibe Versions

Post by id1016 »

Hi.
In order to calculate the clasification accuracy of a two-class motor imagery task, I was running a scenario based on mi-csp-2-train-csp and mi-csp-3-classifier-trainer, with SVM (Nu-SVC with RBF kernel) classifier, and I obtained more than 90% of accuracy for all the subjects with OpenVibe 1.3.0. Then, I did the same thing with OpenVibe 2.0 and the accuracy is so much poorer (average of 45%).
Anyone knows which one is the real accuracy, or if there is any problem with the new version of OpenVibe with certain classifiers?


Thanks!

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: MI Classification Accuracy changes with OpenVibe Version

Post by jtlindgren »

Hello,

the cross-validation in 2.0 has changed its default mode which is now that the data is not randomly permuted. This means that now examples in train and test folds are separated by time instead of being mixed. Due to time correlations in EEG, the classifier might get some test examples correct simply because they look similar to the train example as they are simply close in time. Like, if you have feature vectors x(t1),x(t2),x(t3),...x(tn), the vector x(t2) in EEG will likely resemble its neighbours x(t1) and x(t3) much more than it resembles some far away x(tn). However, in something like motor imagery what we'd like instead is a classifier which based on, say, x(t1) to x(t3), is able to predict x(tn) correctly. If you random permute the examples (as is typically done in machine learning assuming 'identically and independently distributed' training examples; this is NOT the case with EEG), you will have a situation where your test examples are very likely close in time to some of the train examples.

To get back the old behaviour, set in openvibe.conf

Plugin_Classification_RandomizeKFoldTestData=true

and let us know if it changes the situation back to resemble the previous one. If not, there may be an issue which we need to fix.

Please also see http://openvibe.inria.fr/tutorial-how-t ... te-better/ for more background information.

Best,
Jussi

id1016
Posts: 2
Joined: Thu Nov 23, 2017 2:01 pm

Re: MI Classification Accuracy changes with OpenVibe Version

Post by id1016 »

Hi!
Thank you for you answer!
I did the change you mentioned on the openvibe.conf file and I runned the same scenario again in Openvibe 2.0 and the accuracy is still in the order of 45%.

Best regards,
Ines

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: MI Classification Accuracy changes with OpenVibe Version

Post by jtlindgren »

Hello, if you check Edit->Preferences, can you see the value changed to 'true' in the list? If it hasn't changed, its defined as false elsewhere and the behavior wont change.

Best,
Jussi

jlegeny
Posts: 239
Joined: Tue Nov 02, 2010 8:51 am
Location: Mensia Technologies Paris FR
Contact:

Re: MI Classification Accuracy changes with OpenVibe Version

Post by jlegeny »

Hello,

one thing to try to set this option is to create a file called openvibe-workspace.conf which you put just next to your xml scenario. Then put this line into the file:

Code: Select all

Plugin_Classification_RandomizeKFoldTestData=true
The token will be loaded by any scenario next to it when it is started. In the case when the randomization is enabled, you should see this message in the console: Randomizing the feature vector set

Post Reply