OpenViBE forum

The OpenViBE community
It is currently Wed Apr 25, 2018 10:27 am

All times are UTC




Post new topic Reply to topic  [ 5 posts ] 
Author Message
PostPosted: Thu Nov 23, 2017 2:26 pm 
Offline

Joined: Thu Nov 23, 2017 2:01 pm
Posts: 2
Hi.
In order to calculate the clasification accuracy of a two-class motor imagery task, I was running a scenario based on mi-csp-2-train-csp and mi-csp-3-classifier-trainer, with SVM (Nu-SVC with RBF kernel) classifier, and I obtained more than 90% of accuracy for all the subjects with OpenVibe 1.3.0. Then, I did the same thing with OpenVibe 2.0 and the accuracy is so much poorer (average of 45%).
Anyone knows which one is the real accuracy, or if there is any problem with the new version of OpenVibe with certain classifiers?


Thanks!


Top
 Profile  
Reply with quote  
PostPosted: Fri Nov 24, 2017 8:34 am 
Offline

Joined: Tue Dec 04, 2012 3:53 pm
Posts: 676
Location: INRIA Rennes, FRANCE
Hello,

the cross-validation in 2.0 has changed its default mode which is now that the data is not randomly permuted. This means that now examples in train and test folds are separated by time instead of being mixed. Due to time correlations in EEG, the classifier might get some test examples correct simply because they look similar to the train example as they are simply close in time. Like, if you have feature vectors x(t1),x(t2),x(t3),...x(tn), the vector x(t2) in EEG will likely resemble its neighbours x(t1) and x(t3) much more than it resembles some far away x(tn). However, in something like motor imagery what we'd like instead is a classifier which based on, say, x(t1) to x(t3), is able to predict x(tn) correctly. If you random permute the examples (as is typically done in machine learning assuming 'identically and independently distributed' training examples; this is NOT the case with EEG), you will have a situation where your test examples are very likely close in time to some of the train examples.

To get back the old behaviour, set in openvibe.conf

Plugin_Classification_RandomizeKFoldTestData=true

and let us know if it changes the situation back to resemble the previous one. If not, there may be an issue which we need to fix.

Please also see http://openvibe.inria.fr/tutorial-how-t ... te-better/ for more background information.

Best,
Jussi


Top
 Profile  
Reply with quote  
PostPosted: Thu Nov 30, 2017 12:03 pm 
Offline

Joined: Thu Nov 23, 2017 2:01 pm
Posts: 2
Hi!
Thank you for you answer!
I did the change you mentioned on the openvibe.conf file and I runned the same scenario again in Openvibe 2.0 and the accuracy is still in the order of 45%.

Best regards,
Ines


Top
 Profile  
Reply with quote  
PostPosted: Thu Nov 30, 2017 2:39 pm 
Offline

Joined: Tue Dec 04, 2012 3:53 pm
Posts: 676
Location: INRIA Rennes, FRANCE
Hello, if you check Edit->Preferences, can you see the value changed to 'true' in the list? If it hasn't changed, its defined as false elsewhere and the behavior wont change.

Best,
Jussi


Top
 Profile  
Reply with quote  
PostPosted: Thu Nov 30, 2017 4:53 pm 
Offline

Joined: Tue Nov 02, 2010 8:51 am
Posts: 239
Location: Mensia Technologies Paris FR
Hello,

one thing to try to set this option is to create a file called openvibe-workspace.conf which you put just next to your xml scenario. Then put this line into the file:

Code:
Plugin_Classification_RandomizeKFoldTestData=true


The token will be loaded by any scenario next to it when it is started. In the case when the randomization is enabled, you should see this message in the console: Randomizing the feature vector set

_________________
My Weblog Follow me on Twitter


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Americanized by Maƫl Soucaze.