xDawn P300 Speller with OpenBCI headset predicts only 'A'

Working with OpenViBE signal processing scenarios and doing scenario/BCI design
chrisfosterelli
Posts: 1
Joined: Mon Oct 03, 2016 5:54 pm

xDawn P300 Speller with OpenBCI headset predicts only 'A'

Post by chrisfosterelli »

Hi everyone,

I am trying to use an OpenBCI headset with the xDawn P300 speller in OpenViBE, but upon running the "online" scenario the only result is appears to predict is 'A'.

Here are some notes about my particular configuration and current debugging state:
[*]I'm using OpenViBE 1.2.2
[*]I'm using the "OpenBCI" acquisition server driver
[*]In the signal monitoring scenario, I can definitely see EEG readings
[*]I'm currently using the following tutorial: http://openvibe.inria.fr/p300-speller-xdawn/
[*]I'm running OpenViBE on a Ubuntu virtual machine (with USB-passthrough for the OpenBCI dongle) contained on an OSX host
[*]The training scenario reports an 80-90% accuracy (I've tried this a few times)
[*]The online scenario never returns anything other than the letter 'A' for predictions
[*]The signal decimation has always returned an error (since sampling rate and sample count per buffer are not compatible). I have fixed this by changing the signal decimation factor from the default (4 I believe) to 2 in the three scenarios where it is relevant. I have a suspicion this may be related to the problem, but I'm not sure what the proper fix for it is.

Does anyone have any recommendations?

My current attempt at improving this is to use a native Windows 7 host machine with OpenViBE to try and reduce the number of moving parts (as maybe latency is a problem). While I'm setting that up, I thought I would see if anyone in the community has any advice they would give.

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by jtlindgren »

Hi Chris,

P300 like all BCI, is tricky to diagnose. P300 is made more difficult by the fact that the alignment of the markers (stimulations) needs generally to be quite precise. Random ideas,

1) What does the 'replay' scenario give you, does it work with the data you recorded for the training? It should.

2) get the P300 test dataset from the page

http://openvibe.inria.fr/datasets/

and run the scenarios with that. If it works, this suggests that the problem might be somewhere in the data: alignment of the stimulations or the content of the signal itself. If it doesn't, something has broken in OV or the scenarios.

3) You can also take a look at the individual flash predictions which get summed up by the .lua script. Do they make sense? Are the predictions indicating A is the correct answer?

VM might affect the timing somehow. I haven't run BCI over a VM myself. Usually I try to minimize all possible complications.


Three cents,
Jussi

cfkwok
Posts: 1
Joined: Tue Oct 18, 2016 4:00 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by cfkwok »

jtlindgren wrote:Hi Chris,

P300 like all BCI, is tricky to diagnose. P300 is made more difficult by the fact that the alignment of the markers (stimulations) needs generally to be quite precise. Random ideas,

1) What does the 'replay' scenario give you, does it work with the data you recorded for the training? It should.

2) get the P300 test dataset from the page

http://openvibe.inria.fr/datasets/

and run the scenarios with that. If it works, this suggests that the problem might be somewhere in the data: alignment of the stimulations or the content of the signal itself. If it doesn't, something has broken in OV or the scenarios.

3) You can also take a look at the individual flash predictions which get summed up by the .lua script. Do they make sense? Are the predictions indicating A is the correct answer?

VM might affect the timing somehow. I haven't run BCI over a VM myself. Usually I try to minimize all possible complications.


Three cents,
Jussi
I'm also having the same problem.

1) Replay doesn't give me any results either from my online run.
2) I tried using those XMLs and still only gives me "A," however when I use the dataset to run a replay, I can see that it works.
3) It says "A" is the wrong answer.

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by jtlindgren »

Hi, one way to see what the classifier is predicting is to set

Code: Select all

	do_debug = true
in p300-speller-accumulator.lua .

In P300, there are only two classes underneath corresponding to the state 'did the P300 phenomenon happen in the brain' with the classes being 'yes' and 'no'. The phenomena happens when the flash hits the letter the user is attending to. However, most of the flashes are on letters that the user is *not* attending. Hence, when the classifier is estimated, there are much more 'no' classes, so if the classifier can't do better, it will predict 'no'. This means that its best dumb guess is to say 'no' always. So if your classifier is just predicting the majority class, it will predict that the P300 did *not* happen ever. Further, if on every flash the answer is 'no p300' from the classifier, the result will be integrated to 'A'.

With that debug enabled, If you see only lines like

91 sec <Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Push col target 5
[ INF ] At time 165.078 sec <Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Pred fifo row 3 is nontarget

the key is the 'nontarget' there, you have essentially a situation that your classifier is always predicting the same 'no', so no row or column will get any 'votes', and finding the max over an array of zeroes in .lua returns the first index, hence A.

Anyway, thats for the explanation. Why this happens is another question. How I'd test if you have reasonable data is to do the following,

1) Do two separate recordings with p300-xdawn-1-acquisition.xml
2) Use the first recording to train the xdawn and the classifier (scenarios 2 and 3)
3) Use the first recording to run 'replay'
4) Use the second recording to run 'replay'

Now the expectation should be that if you have decent data, both steps should give decent accuracy, and step 3 should be better than step 4. If both break badly, either your data is terrible or there is a bug. If 3 works but 4 doesn't, your datasets don't resemble each other (some drift of amp or brainstate) or the classifier overfits badly. If both work, the online scenario should as well, you can try this by replacing the Acquisition Client box with the Generic Stream reader reading from a file. Note that in this case you wont get the correct predictions, but you shouldn't see 'A's either.

In any case, if you can gather a more specific understanding of what works and what doesn't, it'd be easier to home in on the problem.

ps. If anybody is reading this who has successfully used P300 in OV 1.2.x, let us know! So far I only know that it worked for me in the spring 2016, using gtec gusbamp with 16 channels placed more or less equidistant around the head.


Best,
Jussi

lefthanded
Posts: 4
Joined: Wed Aug 10, 2016 8:02 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by lefthanded »

jtlindgren wrote:
ps. If anybody is reading this who has successfully used P300 in OV 1.2.x, let us know! So far I only know that it worked for me in the spring 2016, using gtec gusbamp with 16 channels placed more or less equidistant around the head.


Best,
Jussi
I had successfully, run the P300 scenario quite esay (the standard one without xDAWN), using gtec's gusbamp with 8 channels in both online modes (copy spelling and free spelling), I run the experiments with 10 sequences and reached a good accuracy (90 %).

by the way, my OV version is 1.2.2

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

I am having exactly the same issue, did anyone manage to find a solution ?

Thanks
Patrick

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

Hi Jussi,
I am a PhD student doing my studies on BCI. I am currently using OpenBCI and was evaluating the P300 on OpenVibe, however like the post says I am always getting an A.

These are to answer your comments:
The dataset download from the website ran perfectly.
I am not running any VM. I am running a Windows 7 machine.
I have tried the first and second recording as you suggested, however both failed.

I am attaching a number of images of my setup. If you check after running the classifier *everything* is labelled as "nontarget".

Could it be the acquisition client? I am reverting to Jeremy's python script and running it on a generic streamer to see the results. Will let you know how it goes.

In the meantime if you have any other ideas, please do let me know; stuck on it at the moment.

Thanks
Patrick

Image
Image

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

I also forgot to mention, if I run the acquisition recording on the reply, it shows only A as result as well. So the problem is not actually within the train and classifier scenario.

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by jtlindgren »

Hi,

the output of the classifier trainer box suggests that it has produced a classifier which always predicts the majority label. With this classifier it is not surprising you will always see the same outcome, since it will always vote 'no flash'. As to why the classifier would be like this, I don't know. One reason could be if there is absolutely no p300 effect discernible in the data.


Best,
Jussi

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

Hi Jussi,
thanks for your prompt reply.
I can visually see the P300 effect when I put a Signal Display box when running the replay. I can send you the data if you want to check (however need your email address).

Today and tomorrow I will try to run Jeremy's python script and the Generic Streamer acquisition. He made it work with the previous version using OpenBCI.
My thought is that maybe there is something wrong with the OpenBCI acquisition?

Will let you know how it goes.

Thanks
Patrick

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by jtlindgren »

Hi Patrick,

although it'd be great to get some more test data from different systems (mine is based on gtec's gusbamp), I don't have time at the moment to investigate the issue. Basically I'd do two things,

1) check that after each processing step, the output is what you expect
2) if so, try to see if the classifier can be tweaked somehow to produce better models (you want to get away from 100% in one class kind of predictions)

What does Jeremy's script exactly do that you mention?

Cheers,
Jussi

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

Hi Jussi,
I can understand and relate :)

1) will do
2) tried to tweak the classifier, I set it from 3 output channels to 2 and to 1 and changed countless settings.
If I understand correctly if I run the acquisition (scenario 1) and I replay it in scenario 5, it should give me the correct output letters and not A, right (say target = S, results should be = S)? Or since it has not been processed, it will not display anything useful and will just always show A in this case?

Jeremy had written a python script for OpenBCI to stream data ( https://github.com/jfrey-xx/OpenBCI_Pyt ... /README.md ) and used a Generic Raw Telnet Reader acquisition to successfully make the P300: CoAdapt P300 Speller (OpenViBE v0.18 only) work. Now my thought was to use his script and generic reader to try and see if the xDawn P300 works. If it does, it might be something with the OpenBCI acquisition? If not, I am at a loss at the moment.
The other weird thing is (and don't take my word on it, I just tried it for an hour), i tried the SSVEP scenario and it did not work as well, hence why I suspected its something to do with the acquisition server?

I have used the combo of OpenBCI with ElectroCap for 2 years and managed to publish a paper and several other results, so it is not an issue with the equipment.

Thanks
Patrick

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

One last thing, I had to eliminate the "Signal Decimation" due to OpenBCI 250 sampling rate (or set it to 2 which is to only multiplier that works for both sampling rate and sample count per buffer). That shouldn't affect the results, am I correct?

Will report my results by tomorrow on the generic stream by tomorrow.

Patrick

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

Hey Jussi,
sorry for my late reply, been hectic the past weeks.

I have tried Jeremy's python (with generic acquisition) instead of OpenBCI acquisition, but it still gave me all as no hits and display only 'A' both in replay of the train and online.

Tomorrow I will be trying to revert back to Openvibe 0.1.8 and try with the "p300-coadapt-stimulator", so at least I am able to replicate Jeremy's success. If this works fine, it must be something related to the OpenVibe new version or xdawn compatibility with OpenBCI ?

Honestly I am at a loss, apart from reverting to the old version I have no idea what else to try. If you have any more ideas for me to try, please let me know.

Thanks
Patrick

knightv8
Posts: 14
Joined: Thu Jun 29, 2017 9:32 pm

Re: xDawn P300 Speller with OpenBCI headset predicts only 'A

Post by knightv8 »

Hey Jussi,
as per my previous post, I downgraded to Openvibe 0.1.8 with the "p300-codapt-stimulator" and after a couple of tries and some settings changed, it worked.

In the next few days I will reinstall the new version and try again from scratch, maybe I missed something.
If you have any other ideas, please do let me know.

Thanks
Patrick

Post Reply