About Classifier Trainer Box

[ariandy]

Hello.

I have several question about using the Classifier Trainer Box.

In Classifier Trainer Box, the parameter for "Train Trigger" is by default set to "OVTK_StimulationID_EndOfFile". From the documentation, this is stimulation to trigger the training process. So if I want to train for a Motor Imagery BCI, do I have to include "OVTK_StimulationID_EndOfFile" stimulation to the end of my XML stimulation file? I don't have clear understanding about this because that stimulation isn't listed on stimulation codes list (http://openvibe.inria.fr/documentation/ ... tions.html). What is the code for that stimulation?

And for training process, since the training is triggered by "End Of File" stimulation, so we have to wait the training process until it's done? How long does it took for 40 iterations of left and right motor imagery data?

Thank you.

[lbonnet]

Hi ariandy,

First of all, the stimulation codes list on the web site needs an update, sorry for the inconvenience. If you want a complete list of available stimulations, take a Classifier Trainer box, and try to change the Train Trigger in the settings.
And if you also need the corresponding codes, they are listed in openvibe-toolkit\trunc\src\openvibe-toolkit\ovtk_defines.h. (OVTK_StimulationId_EndOfFile = 0x00008204)

If you are using the GDF File Reader box, the OVTK_StimulationID_EndOfFile is automatically sent by the reader box when it reaches the end of file. So if the train trigger of the Classifier Trainer box is set to OVTK_StimulationID_EndOfFile, the training process will start when the file is completely read. You will have then the results of the process printed in the designer console (accuracy of each partition, global accuracy, etc.). The time taken by the training process really depends on the size of the feature vectors and the number of partitions used.

If you are reading another format, you will have to configure the Classifier Trainer Box with a particular Train Trigger (the default one is actually OVTK_StimulationId_Train with code 0x00008201), and add this stimulation at the end of your XML stimulation file (used in the acquisition process). Of course, when you are acquiring your data, you must save it along with the stimulations given by the XML Stimulation Scenario Player.

Take a look at the P300-speller example.
- p300-speller-1-acquisition.xml :
In the acquisition scenario, the XML stim file used in the XML Stimulation Scenario Player is p300-speller-stimulations-targets.xml. In this file the last stimulation sent is OVTK_StimulationId_Train. Both EEG data and stimulations are written with a Generic Stream Writer box.

- p300-speller-2-train-classifier.xml
The Classifier Trainer box has a train trigger set to OVTK_StimulationId_Train. The trainer scenario reads the "generic stream" file recorded, and when the OVTK_StimulationId_Train stim is reached (it should be at the end of the file...), the training process starts.

I hope this will help to understand the training process.

Regards,
Laurent-

[ariandy]

Dear Laurent,

Thank you for your fast response and clear explanation, it helps me understanding the process.

I used the GDF file reader box, but I didn't stop my EEG recording session right after my XML stimulations over, so maybe about last half of my GDF file is mostly unused. And I didn't knew that i should put an ending stimulation in XML stimulation file. To start the training process immediately (in the middle of GDF file reading), can I pass a stimulation to the Classifier Trainer Box via Stimulation Multiplexer Box?

And also I'm a bit curious about something not related to Classifier Trainer Box. Since I already have my classification constants, how can I build an online BCI just like from the sample scenario, but without cues from XML stimulation? So the user/subject could independently imagining left/right hand movement, and OpenViBE shows the classification outputs in real-time. Without the computer ordering the subject to imagine left/right movement. That would be great for Lab exhibition/show off, or could be used to control external device like robot vehicle.

Thank you.

[lbonnet]

To start the training process immediately (in the middle of GDF file reading), can I pass a stimulation to the Classifier Trainer Box via Stimulation Multiplexer Box?

yes, you can. If so, the training process will rely on the first-half part of the EEG data.
When the train trigger is detected by the classifier trainer, the file reading is paused, and the trainer box do its job with the data it received so far. When the training is done, the EEG file reading is resumed. If you trigger again the classifier trainer, it will train again but this time on more data: since t=0 to the trigger time.

how can I build an online BCI just like from the sample scenario, but without cues from XML stimulation? So the user/subject could independently imagining left/right hand movement, and OpenViBE shows the classification outputs in real-time.

Take the motor-imagery-bci-3-online.xml file. The graz visualization needs only one stimulation to show the feedback: OVTK_GDF_Feedback_Continuous (0x30D). For example, you can just link the Stimulations input of the visualization box to a keyboard simulator that provides OVTK_GDF_Feedback_Continuous. Start the scenario, (reading from file or acquisition client), press the right key in the keyboard stimulator panel, and look the real-time result on the graz visualization panel.

Regards,
Laurent-

[ariandy]

yes, you can. If so, the training process will rely on the first-half part of the EEG data.
When the train trigger is detected by the classifier trainer, the file reading is paused, and the trainer box do its job with the data it received so far. When the training is done, the EEG file reading is resumed. If you trigger again the classifier trainer, it will train again but this time on more data: since t=0 to the trigger time.

Okay, thanks for the information, it's very useful.

Take the motor-imagery-bci-3-online.xml file. The graz visualization needs only one stimulation to show the feedback: OVTK_GDF_Feedback_Continuous (0x30D). For example, you can just link the Stimulations input of the visualization box to a keyboard simulator that provides OVTK_GDF_Feedback_Continuous. Start the scenario, (reading from file or acquisition client), press the right key in the keyboard stimulator panel, and look the real-time result on the graz visualization panel.

Wow, that's great! Thanks again for the tip, I will try this soon.

And thank you very much for your great support!

[ariandy]

Dear Yann, Laurent,

I have additional questions regarding this box. Now it's about number of partitions for K-Fold Test. I've test my EEG recording with various numbers of partitions from 3 to 10, and different partition gives different classifier performance. In a simple way, in order to get the best classifier performance, does changing different number of partitions is some kind of manual tuning?

I have read the fine documentation on the main site, but a bit of extra explanation would be great.

Thank you very much.

[yrenard]

Dear ariandy,

indeed, you can test different values for the partition number in the K-fold test. But generally, you will end with an almost optimal value between 3 and 10 partitions. I'm not sure from your feedback whether the documentation of the box was clear or not. However, you should probably tell us what were the results on the different partitions. You could also look at how much artifacts you have in the signals.

Yann

[ariandy]

Dear ariandy,

indeed, you can test different values for the partition number in the K-fold test. But generally, you will end with an almost optimal value between 3 and 10 partitions. I'm not sure from your feedback whether the documentation of the box was clear or not. However, you should probably tell us what were the results on the different partitions. You could also look at how much artifacts you have in the signals.

Yann

Dear Yann,

Sorry for my late reply, I'm being kept busy lately.

For the same GDF file input, these are classifier trainer results:
3 Partitions = 67.0918%
4 Partitions = 67.2959%
5 Partitions = 67.0408%
6 Partitions = 67.6531%
7 Partitions = 67.0408%
8 Partitions = 67.7041%
9 Partitions = 66.9388%
10 Partitions = 66.7857%

The best result are from 8 partitions. So, if I use 8 partitions classification coefficient file, will my online session accuracy increase? And please tell me what do you think about my trainer results, i never get better than 70% performance.

Thank you.

And here's the full output. I've cleaned it a bit, but still a long list, too bad we don't have spoiler tag here.

Code: Select all

partition 1 (0x1) / 3 (0x3) (performance : 77.6417%)
partition 2 (0x2) / 3 (0x3) (performance : 63.8591%)
partition 3 (0x3) / 3 (0x3) (performance : 60.8563%)
Best classifier performance was for partition 1 (0x1)... Traininng on this partition again !
[  INF  ] At time 1418480058368 (0x14a44000000)<Box algorithm::Classifier trainer> Best classifier performance on whole set is
3 Partitions' Accuracy = 67.0918%

partition 1 (0x1) / 4 (0x4) (performance : 77.9592%)
partition 2 (0x2) / 4 (0x4) (performance : 67.551%)
partition 3 (0x3) / 4 (0x4) (performance : 54.2857%)
partition 4 (0x4) / 4 (0x4) (performance : 64.2857%)
Best classifier performance was for partition 1 (0x1)... Traininng on this partition again !
4 Partitions' Accuracy = 67.2959%

partition 1 (0x1) / 5 (0x5) (performance : 81.6327%)
partition 2 (0x2) / 5 (0x5) (performance : 71.9388%)
partition 3 (0x3) / 5 (0x5) (performance : 58.1633%)
partition 4 (0x4) / 5 (0x5) (performance : 60.4592%)
partition 5 (0x5) / 5 (0x5) (performance : 58.9286%)
Best classifier performance was for partition 1 (0x1)... Traininng on this partition again !
5 Partitions' Accuracy = 67.0408%

partition 1 (0x1) / 6 (0x6) (performance : 77.3006%)
partition 2 (0x2) / 6 (0x6) (performance : 75.2294%)
partition 3 (0x3) / 6 (0x6) (performance : 72.1713%)
partition 4 (0x4) / 6 (0x6) (performance : 54.6012%)
partition 5 (0x5) / 6 (0x6) (performance : 63.6086%)
partition 6 (0x6) / 6 (0x6) (performance : 53.8226%)
Best classifier performance was for partition 1 (0x1)... Traininng on this partition again !
6 Partitions' Accuracy = 67.6531%

partition 1 (0x1) / 7 (0x7) (performance : 77.1429%)
partition 2 (0x2) / 7 (0x7) (performance : 82.8571%)
partition 3 (0x3) / 7 (0x7) (performance : 64.6429%)
partition 4 (0x4) / 7 (0x7) (performance : 70%)
partition 5 (0x5) / 7 (0x7) (performance : 47.5%)
partition 6 (0x6) / 7 (0x7) (performance : 69.2857%)
partition 7 (0x7) / 7 (0x7) (performance : 59.2857%)
Best classifier performance was for partition 2 (0x2)... Traininng on this partition again !
7 Partitions' Accuracy = 67.0408%

partition 1 (0x1) / 8 (0x8) (performance : 78.7755%)
partition 2 (0x2) / 8 (0x8) (performance : 80%)
partition 3 (0x3) / 8 (0x8) (performance : 73.0612%)
partition 4 (0x4) / 8 (0x8) (performance : 62.8571%)
partition 5 (0x5) / 8 (0x8) (performance : 47.7551%)
partition 6 (0x6) / 8 (0x8) (performance : 60%)
partition 7 (0x7) / 8 (0x8) (performance : 62.8571%)
partition 8 (0x8) / 8 (0x8) (performance : 64.4898%)
Best classifier performance was for partition 2 (0x2)... Traininng on this partition again !
8 Partitions' Accuracy = 67.7041%

partition 1 (0x1) / 9 (0x9) (performance : 77.4194%)
partition 2 (0x2) / 9 (0x9) (performance : 83.945%)
partition 3 (0x3) / 9 (0x9) (performance : 66.9725%)
partition 4 (0x4) / 9 (0x9) (performance : 70.6422%)
partition 5 (0x5) / 9 (0x9) (performance : 61.2903%)
partition 6 (0x6) / 9 (0x9) (performance : 57.7982%)
partition 7 (0x7) / 9 (0x9) (performance : 58.2569%)
partition 8 (0x8) / 9 (0x9) (performance : 64.2202%)
partition 9 (0x9) / 9 (0x9) (performance : 59.633%)
Best classifier performance was for partition 2 (0x2)... Traininng on this partition again !
9 Partitions' Accuracy = 66.9388%


partition 1 (0x1) / 10 (0xa) (performance : 75%)
partition 2 (0x2) / 10 (0xa) (performance : 88.2653%)
partition 3 (0x3) / 10 (0xa) (performance : 67.8571%)
partition 4 (0x4) / 10 (0xa) (performance : 81.1224%)
partition 5 (0x5) / 10 (0xa) (performance : 66.3265%)
partition 6 (0x6) / 10 (0xa) (performance : 49.4898%)
partition 7 (0x7) / 10 (0xa) (performance : 46.4286%)
partition 8 (0x8) / 10 (0xa) (performance : 76.0204%)
partition 9 (0x9) / 10 (0xa) (performance : 58.6735%)
partition 10 (0xa) / 10 (0xa) (performance : 59.1837%)
Best classifier performance was for partition 2 (0x2)... Traininng on this partition again !
10 Partitions' Accuracy = 66.7857%

[yrenard]

Dear ariandy,

The best result are from 8 partitions. So, if I use 8 partitions classification coefficient file

Well, my feeling is that those performances are almost the same

will my online session accuracy increase?

I can't tell, this just reflects that the classifier had better performances classifying feature vectors which it wasn't trained on.
Statistically, it should be able to to classify the future feature vectors better, but there's no guarantee.

And please tell me what do you think about my trainer results, i never get better than 70% performance.

70% should be ok so that you feel it's working... This is clearly more than chance but I understand this could be frustrating. There are several ways you could make that performance better : use more electrodes, get more attentions to artifacts, train yourself on doing the mental task, update the frequency band values depending on what reacts best for you... I gave several tips in another thread that could help.

I've cleaned it a bit, but still a long list, too bad we don't have spoiler tag here.

Is it a standard phpBB functionality ?
If yes, I should be able to activate it
If not, I just don't have time to maintain mod compatibility over time, sorry !

Hope this helps,
Yann

[ariandy]

Dear ariandy,

Well, my feeling is that those performances are almost the same

Ah, yes you right..

I can't tell, this just reflects that the classifier had better performances classifying feature vectors which it wasn't trained on.
Statistically, it should be able to to classify the future feature vectors better, but there's no guarantee.

70% should be ok so that you feel it's working... This is clearly more than chance but I understand this could be frustrating. There are several ways you could make that performance better : use more electrodes, get more attentions to artifacts, train yourself on doing the mental task, update the frequency band values depending on what reacts best for you... I gave several tips in another thread that could help.

This is what i meant before about Classifier Trainer documentation, "The more partitions you have, the more feature vector you have in your training sets... and the less examples you'll have to test on. This means that the result of the test will probably be less reliable. But you will be able to choose the best classifier among a more consequent list."

What do you mean by "able to choose the best classifier among a more consequent list"? Can we choose several top performing partitions to be used as classifier, or we have to take it as a whole, including low performing partition?

Is it a standard phpBB functionality ?
If yes, I should be able to activate it
If not, I just don't have time to maintain mod compatibility over time, sorry !

Hope this helps,
Yann

Nah, don't worry about it. It isn't important anyway

Thanks for your great support!

[yrenard]

This is what i meant before about Classifier Trainer documentation, "The more partitions you have, the more feature vector you have in your training sets... and the less examples you'll have to test on. This means that the result of the test will probably be less reliable. But you will be able to choose the best classifier among a more consequent list."

What do you mean by "able to choose the best classifier among a more consequent list"? Can we choose several top performing partitions to be used as classifier, or we have to take it as a whole, including low performing partition?

Dear ariandy,

the idea of the k-fold test is to chose a subset of the training set to train the classifier and test it on the rest of the set... This gives an idea of how the classifier manages feature vectors that he never known about while training. You can re iterate the process a few times with different subsets, each giving a different accuracy. The best one of all those trained classifier is kept for online use. The way the k-fold test has been implemented in OpenViBE implies that the more partitions you request, the smaller the test subset will be... So if you request a big enough number of partitions, the test subset will be so small that you will have a very rough idea of the classifier performance... e.g. if the test subset has only 3 feature vectors, the classifier accuracy estimation will be either 0%, 33%, 66% or 100%. You'll have nothing between 66% and 100%. At then end, you will have to choose between multiple classifiers that had a performance of 66% or 100% buy you won't be able to know which one is actually the really best

I hope this was clear enough
Yann

[darlenehill19]

I am just new to classifier..
Treat me as a noob
I am following your discussion..pretty useful

I read word by word this link
http://openvibe.inria.fr/documentation/ ... ainer.html

But right now all strings are unattached for me..
Can any body tell me the sequence..what to read first and what to read at second??
I believe in concepts! Sincere advice can really give me an EDGE!

[yrenard]

Dear darlenehill19,

thank you for your interest in OpenViBE and welcome on this forum !

But right now all strings are unattached for me..
Can any body tell me the sequence..what to read first and what to read at second??
I believe in concepts! Sincere advice can really give me an EDGE!

I am sorry to tell you I don't understand your problem. Can you give us more information about what you are trying to do and what problem you are facing ?

Thanks !
Yann

[ddvlamin]

Hi,

I'm also not sure if this could be an answer to your question, but still...

If you want to know more about the application of classification or machine learning techniques in BCI, I found this article very useful http://ml.cs.tu-berlin.de/publications/ ... rBla04.pdf which also has a nice list of references. I think there's also a presentation from that person/group on videolectures.net : http://videolectures.net/bbci09_blankertz_muller_mlasp/

Because you like concepts: the main idea is to first preprocess your signals by for example temporal or spatial filtering all of which has the same purpose: maximizing signal-to-noise ratio, so amplifying the signal you're interested in, while inhibiting all the rest. Common spatial patterns (CSP) is a nice example of a spatial filters used for motor-imagery based BCI and off course the XDAWN algorithm that is available in OpenViBE for improving performance of the P300-based BCI (or other evoked potential paradigms). In both methods they try to find a way of quantifying the signal you're interested in (for example in CSP by computing the variance of the signals in one specific condition,e.g. left hand movement). Then they find a way for quantiyfing the noise (for example the variance of the whole data set or based on the signals that were not measured during the above condition you were interested in).
After preprocessing one calculates the features based on the signals in a certain time frame (e.g. 1 second). In motor-imagery BCI for example, computing the features can be as simple as computing the variance on a 1 second time window on all channels. We can then move the window and do the same thing. This way we get a set of feature vectors for each of the conditions (e.g. left hand or right hand) which we feed to the classifier that then tries to predict based on such a feature vector which class it belongs to. To know more about classifiers you can take a look a the above links. A book which I also find very useful is the book of C. Bishop (pattern recognition and machine learning), for a start one could read chapter 1 and 4.

Best regards,
Dieter Devlaminck