Classifier trainer

[yannick]

Hi,
I have some questions about classifier trainer on openvibe.

1) I have my .ov's file and i want to train a classifier to generate two classes, i would like to know, how it is possible to see if the classifier trainer have finished to train or class?
2) How it is possible to see the performance of classifier trainer in progress?( classifier accuracy measure is for classifier processor only ?)

Thanks

[yrenard]

Dear yannick,

the classifier trainer sends messages in the console / log file that are pretty clear I guess. You will have a feedback about its acuracy on the different partitions if you use k-fold test... However you should know that the classifier training process is not done online. This is not an iterative process. It is done only once when a specific stimulation tells it to perform the actual training. Have a look to the motor-imagery bci sample for an example.

Hope this helps,
Yann

[yannick]

Yes i know that this is not a online process, but i would like to know how can i know that the process is over?
For example, take motor-imagery-bci-2-classifier-trainer.xml, when i lauched it, i not have an output in a console, the time go and go ......
I think that the stimulation is the EEG signas in .ov's file or start button.
But my question is how can i know that the process is over and how can i see the values of classifier trainer at all moment?

thanks

[yrenard]

yannick,

when running the motor imagery bci training scenario, I get this ouput in my console

[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Received train stimulation - k-fold test could take quite a long time, be patient
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 1 (0x1) / 7 (0x7) (performance : 76.9841%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 2 (0x2) / 7 (0x7) (performance : 74.6032%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 3 (0x3) / 7 (0x7) (performance : 62.6984%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 4 (0x4) / 7 (0x7) (performance : 88.0952%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 5 (0x5) / 7 (0x7) (performance : 82.9365%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 6 (0x6) / 7 (0x7) (performance : 88.8889%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 7 (0x7) / 7 (0x7) (performance : 80.9524%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Best classifier performance was for partition 6 (0x6)... Traininng on this partition again !
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Best classifier performance on whole set is 81.1224%

Looks like the box received the train stimulation...

Do you have this output yourself ?

( 0x1ce00000000 seconds in 32:32 = 462.0 seconds = 7 mns + 42 seconds)

Best regards,
Yann

[yannick]

Hi Yann,
When i lauched motor-imagery_bci_2_classifier-trainer.xml with openvibe designer, i don't have this output in my console.
Do you have an idea please?

Best

[yrenard]

Dear yannick,

maybe the stimulation you use to trigger the training is never received. You could use the Stimulation Listener to check this stimulation is actually sent.

Best regards,
Yann

[yannick]

Hi,
I would like to know where you put stimulation listener in the motor-imagery-bci-classifier-trainer?
Because classifier trainer don't have output for input stimulation listener

Thanks

[lbonnet]

maybe the stimulation you use to trigger the training is never received. You could use the Stimulation Listener to check this stimulation is actually sent.

I think Yann meant to check if the trigger was received by the classifier trainer, i.e on its input. Read with the stimulation listener the incoming stimulations sent to the classifier, on parallel. In your console will be printed all stimulations received by the Listener. Check if you see the train trigger.

Laurent

[yannick]

Hi,
I connect the stimulation listener behind the ouput (stimulation) acquisition client and it printed a lot of information. For me, the stimulation listener received the train stimulation.
In that case, i don't understand why i don't have an output with classification performance.

Thanks

[lbonnet]

I connect the stimulation listener behind the ouput (stimulation) acquisition client and it printed a lot of information.

In my version of there is no Acquisition Client in this scenario, only a Generic stream reader. If you connect the stimulation listener behind the reader box (stimulation output of course) you will see ALL stimulations, including the instructions. It may represent a lot of prints... In this case you may use the Stimulation Filter (under Stimulations/Adaptation) : select the OVTK_StimulationId_EndOfFile and reject everything else.

If the train trigger is unchanged, it should be OVTK_StimulationId_EndOfFile, a stimulation written automatically in the .ov file at the end of the data. If you don't see it with the Listener, maybe there was an error during the writing process of the .ov file, and the stimulation has not been written...

Here is backup solution : add a Keyboard Stimulator to the stimulation flow going to the classifier trainer (you can use a stimulation multiplexer for that purpose). When you know your EEG file has been completely read, launch the training process manually with a stimulation coming from the Keyboard Stimulator.

Laurent

[lbonnet]

If the train trigger is unchanged, it should be OVTK_StimulationId_EndOfFile, a stimulation written automatically in the .ov file at the end of the data. If you don't see it with the Listener, maybe there was an error during the writing process of the .ov file, and the stimulation has not been written...

Yann just corrected me : in fact the stimulation OVTK_StimulationId_EndOfFile is not written automatically in the .ov file. It is indeed present in the .ov file given with the scenario motor-imagery-bci-2-classifier-trainer.xml, because this data file comes from an original GDF file (and the GDF file reader box raises the end-of-file stimulation)...
Anyway, change the train trigger to OVTK_GDF_End_Of_Session. This stimulation should be a better trigger.

I have a question... How did you record your training data ? What openvibe scenario and stimulation scenario are you using ?

You should use motor-imagery-bci-3-online.xml, where the file reading is just replaced by an Acquisition Client and a XML stimulation scenario player. The corresponding stimulation scenario should be share/openvibe-plugins/stimulation/graz_stimulation_scenario.xml. In this stimulation scenario, the last stim is OVTK_GDF_End_Of_Session.

Laurent

[yannick]

Ok, thanks for your help.
In fact, i would like to play motor-imagery-bci-classifier scenario to see the performance of classifier trainer.
I f i understand what you said, it's necessary to use stimulation listener to print all information.
But if i want to see the classifier's performance like that :

[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Received train stimulation - k-fold test could take quite a long time, be patient
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 1 (0x1) / 7 (0x7) (performance : 76.9841%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 2 (0x2) / 7 (0x7) (performance : 74.6032%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 3 (0x3) / 7 (0x7) (performance : 62.6984%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 4 (0x4) / 7 (0x7) (performance : 88.0952%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 5 (0x5) / 7 (0x7) (performance : 82.9365%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 6 (0x6) / 7 (0x7) (performance : 88.8889%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Finished with partition 7 (0x7) / 7 (0x7) (performance : 80.9524%)
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Best classifier performance was for partition 6 (0x6)... Traininng on this partition again !
[ INF ] At time 1984274890752 (0x1ce00000000)<Box algorithm::Classifier trainer> Best classifier performance on whole set is 81.1224%
(Yann console output).

How i can do it with this (see snapshot)?

Information: motor-imagery-bci-classifier-trainer.xml (default in openvibe), i want to see just the performance.

[lbonnet]

If you take the given scenario motor-imagery-bci-2-classifier-trainer.xml, and you play it (fast forward); around 7min30 the classifier performances should be printed in the console. Nothing more to do really... It works fine on my freshly installed openvibe.

If you want to use the stimulation listener, just read the incoming stimulations. It's just a tool for debug purpose.



If you set the print level to something visible, eg "information", you should see something like :

Code: Select all

.....
[  INF  ] At time 1984308445184 (0x1ce02000000)<Box algorithm::Stimulation listener> For input 0 (0x0) with name Stimulation stream 1 got stimulation 1010 (0x3f2)[OVTK_GDF_End_Of_Session] at date 1984165838848 (0x1cdf9800000) and duration 0 (0x0)
[  INF  ] At time 1984308445184 (0x1ce02000000)<Box algorithm::Stimulation listener> For input 0 (0x0) with name Stimulation stream 1 got stimulation 33284 (0x8204)[OVTK_StimulationId_EndOfFile] at date 1984274890752 (0x1ce00000000) and duration 0 (0x0)

See the train trigger OVTK_StimulationId_EndOfFile.

And right after this, the classifier trainer prints the performances (I used 5 partitions for k-fold test):

Code: Select all

[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Received train stimulation - k-fold test could take quite a long time, be patient
[ TRACE ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Training on partition 0 (0x0) (feature vectors 0 (0x0) to 351 (0x15f))...
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Finished with partition 1 (0x1) / 5 (0x5) (performance : 76.4205%)
[ TRACE ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Training on partition 1 (0x1) (feature vectors 352 (0x160) to 704 (0x2c0))...
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Finished with partition 2 (0x2) / 5 (0x5) (performance : 61.1898%)
[ TRACE ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Training on partition 2 (0x2) (feature vectors 705 (0x2c1) to 1057 (0x421))...
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Finished with partition 3 (0x3) / 5 (0x5) (performance : 86.4023%)
[ TRACE ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Training on partition 3 (0x3) (feature vectors 1058 (0x422) to 1410 (0x582))...
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Finished with partition 4 (0x4) / 5 (0x5) (performance : 88.6686%)
[ TRACE ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Training on partition 4 (0x4) (feature vectors 1411 (0x583) to 1763 (0x6e3))...
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Finished with partition 5 (0x5) / 5 (0x5) (performance : 80.4533%)
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Best classifier performance was for partition 4 (0x4)... Training on this partition again !
[  INF  ] At time 1984341999616 (0x1ce04000000)<Box algorithm::Classifier trainer> Best classifier performance on whole set is 80.6689%

All I did was adding the Stimulation listener to the given scenario... Did you change something else on your scenario ?

PS : The openvibe I used for the screenshot is a development version, classifier trainer has an output stimulation. Don't pay attention to it.

[yannick]

Ok, all informations are displayed on my console, but after 40 seconds openvibe-designer.exe crashed.
I have install openvibe on 4 pc on windows xp and i have some problems like that, please see the snapshot.

Thanks.

[lbonnet]

Ok I see !
This problem is known and corrected : the stimulation encoding/decoding with EBML had a (vicious) bug, that became a top-priority... a patch has been promptly developed to fix this, but I'm afraid the 0.5.0 release is bugged.

If you use the latest SVN it will be ok ! The upcoming release will include this fix of course.

Laurent