high cross-valid accuracy poor prediction in online scenario

Obtaining data from various hardware devices
Post Reply
umar
Posts: 7
Joined: Thu Nov 29, 2018 9:32 pm

high cross-valid accuracy poor prediction in online scenario

Post by umar »

Hello,
Could someone explain why cross validation of accuracy of about 80% on xDawn p300 speller, yet it could not not correctly predict any single letter in an online scenario? Although, the class wise accuracy is very poor as it is between 4-5 % for the target, and 90-95 % for the non-target. Is this normal? Could there be any known cause of this? if not how can one improve on it? Can connectivity measure when training CSP spatial filter trainer help?
Thanks!

jtlindgren
Posts: 778
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: high cross-valid accuracy poor prediction in online scen

Post by jtlindgren »

Hi, thats often asked why the xval accuracy might exceed onlibe accuracy. Have you read this?

http://openvibe.inria.fr/tutorial-how-t ... te-better/

There could also be a bug, its not impossible. You are also not telling which amp and electrode layout you are using.

We should put these points to the FAQ or even write a separate document about what to do in case of poor perf. Maybe I'll sketch one during christmas.

Cheers,
Jussi

umar
Posts: 7
Joined: Thu Nov 29, 2018 9:32 pm

Re: high cross-valid accuracy poor prediction in online scen

Post by umar »

Hi Jussi,
Many thanks for the reply. Yes, I later came across the cross -validation doc.
Best
Umar

Post Reply