Classifier Processor Usage with more than 2 classes.

Concerning processing components: filters, file load/save, visualizations, communication ...
eeg
Posts: 5
Joined: Tue Mar 08, 2011 6:34 am

Classifier Processor Usage with more than 2 classes.

Post by eeg »

Hi there,

I have created a scenario which uses SVM (support vector machine) algorithm to create a classification file with 6 different classes (yes, this is probably very inaccurate due to high number of differentiations). The training process has worked successfully and created this file, but I cannot figure out how to use the Classifier Processor box to classify more than 2 classes.

Feedback would be greatly appreciated :)

Thanks

EDIT: The classifier processor is in its own scenario. I just cannot figure out how to set it to output and detect more than 2 different classes.

bpayan
Posts: 46
Joined: Fri Jan 08, 2010 4:02 pm

Re: Classifier Processor Usage with more than 2 classes.

Post by bpayan »

Dear eeg,

Thank you for your interest in OpenViBE and welcome on this forum

Actually, the classifier processor can't classify more than 2 classes. At the moment, I'm working on SVM to add the possibility to modify parameters of the SVM algorithm rather than always use the default parameters. Currently, the SVM algorithm can learns and classify 2 classes or more, but the probability is given only for the first class(Matrix output of the classifier processor box).

The real problem comes to the classifier processor. To classify more than 2 classes, you need to add new "class label" setting, but this function is not implemented yet.

I'll look the classifier processor, to allowed classify more than 2 classes and I'll try to modify the output Matrix to give the probability of the best class without break the scenarios using the actual SVM classifier.

Best Regards,

bpayan

eeg
Posts: 5
Joined: Tue Mar 08, 2011 6:34 am

Re: Classifier Processor Usage with more than 2 classes.

Post by eeg »

Hi again bpayan,

Thank you very much for the response :)
Would a temporary solution be to use 3 (2class) classifiers and then use the voting classifier to determine the most active state?

Regards

ddvlamin
Posts: 160
Joined: Thu Aug 13, 2009 8:39 am
Location: Ghent University
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by ddvlamin »

Yes, I think it is possible to create a one-versus-all classifier by using the voting classifier.

You could make for example 6 different classifiers: class 1 verus rest, class 2 versus rest, class 3 versus rest, class 4 versus rest, class 5 versus rest and class 6 versus rest. If you feed an object of class 3, all classifiers should classify it as non-target except for the third classifier which gets the final vote. However there can be some problems with this scheme: what if two classifiers detect it as target class? then maybe one should thrust the classifier where the object lies furthest from the decision boundary?

I doubt if this scheme will work well unless your classes are really easy to separate.

Best regards,
Dieter

eeg
Posts: 5
Joined: Tue Mar 08, 2011 6:34 am

Re: Classifier Processor Usage with more than 2 classes.

Post by eeg »

Thank you Dieter :)

I still haven't tested the accuracy because I have to work on various preprocessing models, but the actual classification works well- each of the 6 class trainers have the non-target set to an interval between recording, and each of the 6 trainers uses a different file.

One thing I have noticed however is that the stimulations from the .ov recording are difficult to sync with the stimulations from the voting classifier (the Classifier accuracy box doesn't seem to work otherwise). The only solution I have found to this is to use the deprecated resynced stimulations output on the stimulation based epoching box.

I have looked around in the documentation but cannot find the sync which replaces this output.
If you know how this would be adequately implemented, it be of great benefit.

ddvlamin
Posts: 160
Joined: Thu Aug 13, 2009 8:39 am
Location: Ghent University
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by ddvlamin »

eeg wrote:Thank you Dieter :)
One thing I have noticed however is that the stimulations from the .ov recording are difficult to sync with the stimulations from the voting classifier (the Classifier accuracy box doesn't seem to work otherwise). The only solution I have found to this is to use the deprecated resynced stimulations output on the stimulation based epoching box.

I have looked around in the documentation but cannot find the sync which replaces this output.
If you know how this would be adequately implemented, it be of great benefit.
I'm not sure if I understand your problem, can you give us the scenario you have built?

I tried to use the confusion matrix box once (it works similar to the classification accuracy box I guess) for evaluating a motor imagery experiment. Because you only have one stimulation per trial you can also only extract one feature object per trial and give it for classification to the classifier (otherwise both are indeed not synchronized). So I used the stimulation epoching box so that I could extract per trial the variance for a two second window that followed that stimulation. However, the problem is that in the stimulation based epoching box I couldn't enter two stimulation types for epoching. So I created a workaround as in the scenario included in the picture https://picasaweb.google.com/1008270689 ... 5142620418. The left path is for left motor imagery trials and the right path for right hand motor imagery trials.

This resulted in this ugly confusion matrix:
https://picasaweb.google.com/1008270689 ... 6616726482

There's probably a much better way to do it, but at the moment I don't know either.

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by yrenard »

hello everyone,

the code of the Classifier Processor box actually is ready for multi class output. Just that I forgot to add the setting addition capability to the box. I'll update this immediately.

Best regards,
Yann

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: Classifier Processor Usage with more than 2 classes.

Post by lbonnet »

Hi everyone,
I tried to use the confusion matrix box once (it works similar to the classification accuracy box I guess) for evaluating a motor imagery experiment. Because you only have one stimulation per trial you can also only extract one feature object per trial and give it for classification to the classifier (otherwise both are indeed not synchronized). So I used the stimulation epoching box so that I could extract per trial the variance for a two second window that followed that stimulation. However, the problem is that in the stimulation based epoching box I couldn't enter two stimulation types for epoching. So I created a workaround as in the scenario included in the picture https://picasaweb.google.com/1008270689 ... 5142620418. The left path is for left motor imagery trials and the right path for right hand motor imagery trials.

This resulted in this ugly confusion matrix:
https://picasaweb.google.com/1008270689 ... 6616726482

There's probably a much better way to do it, but at the moment I don't know either.
I attach to this post a commented screenshot of a scenario. It's some motor imagery replay.
I first select only the epochs on which the feedback is provided (a 4 sec window after the OVTK_GDF_Left/OVTK_GDF_Right instructions, starting 0.5s after the instruction stimulation) with the Stimulation based Epoching box.
Then I use a streamed matrix multiplexer to merge the 2 kinds of epoch in one stream. Please note I am not using a Signal Merger.
After that part, the classifier only works on the part of signal where I want the feedback (according to graz protocol).

I add the 2 visualizations : classifier accuracy measure, and confusion matrix +matrix display.
In order to select only instructions and classifier results, I filter the stimulation stream (select only OVTK_GDF_Left/OVTK_GDF_Right) with the Stimulation Filter box.

After some time, I got the visualization you see on the snapshot. At time 1:39, I have an overall performance of 59.86%, but the confusion matrix tells me that the Right hand is working much better than the Left one : 70% compared to 55% (maybe these values need more precision in the computation...).

Notes:
The Stimulation Based Epoching wait for a full epoch before sending the block. Thus the Graz visualization (or signal display) displays something only after 4 sec, and not regularly as usual.

Hope this helps !

Laurent
Attachments
classifier accuracy measure and confusion matrix
classifier accuracy measure and confusion matrix
snap-classifier-vizu.png (305.52 KiB) Viewed 21390 times
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

ddvlamin
Posts: 160
Joined: Thu Aug 13, 2009 8:39 am
Location: Ghent University
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by ddvlamin »

lbonnet wrote:
Then I use a streamed matrix multiplexer to merge the 2 kinds of epoch in one stream. Please note I am not using a Signal Merger.
Great, that's indeed the thing I was looking for, didn't know about its existence, I should have taken a better look at the documentation.

lbonnet wrote: After some time, I got the visualization you see on the snapshot. At time 1:39, I have an overall performance of 59.86%, but the confusion matrix tells me that the Right hand is working much better than the Left one : 70% compared to 55% (maybe these values need more precision in the computation...).

Notes:
The Stimulation Based Epoching wait for a full epoch before sending the block. Thus the Graz visualization (or signal display) displays something only after 4 sec, and not regularly as usual.

Hope this helps !

Laurent
Normally I use windows of 1 or maximum 2 seconds long, with half a second delay after the stimulation of the start of the trial.

Thanks,
Dieter Devlaminck

eeg
Posts: 5
Joined: Tue Mar 08, 2011 6:34 am

Re: Classifier Processor Usage with more than 2 classes.

Post by eeg »

Hi Yann,

When the classifier processor box is updated, will the code be added in the SVN depository, and will there be any kind of update announcement? I'd really like to contribute by fixing this myself, but I don't have enough programming experience :(.

@Dieter,

The issue I am encountering involves the classifier accuracy box- it has to receive a stimulation from the OV file reader at the same time as a stimulation from a classifier, and then a "hit" is registered. What I have noticed is that the stimulation from the classifier does not register at the same timing as the stimulation from the file reader- this causes no score to show (not even 0/20, just blank).

The way I solved this issue was by sending a stimulation from the Resynched stimulations output of the epoching box (which the documentation names as deprecated) instead of a stimulation from the file reader.

As a result, the accuracy box receives 2 stimulations at the same time- one from a classifier, and one from the associated Resynced epoching box.

Attached is a scenario which demonstrates what I am doing in a simplified manner
(the unsimplified scenario uses multiple instances of the same tree of processing).
Example1.xml
(59.8 KiB) Downloaded 335 times
Thanks

EDIT:
@lbonnet, thank you for explaining the Confusion Matrix box. It suits my purpose very well as I can see which class was chosen instead of the correct one.

I am trying to distinguish 6 English phonemes (W, Ah, T, S, Uh, N) from Broca's area in order to derive 'silent speech'. This new accuracy box helps as it allows me to determine which of the phonemes was actually detected, and thus I can learn which is the most easily differentiated (hopefully I can achieve at least 25% accuracy for statistical significance, but as mentioned in other threads, it will be very hard to distinguish more than 3 different thoughts).

ddvlamin
Posts: 160
Joined: Thu Aug 13, 2009 8:39 am
Location: Ghent University
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by ddvlamin »

eeg wrote: The issue I am encountering involves the classifier accuracy box- it has to receive a stimulation from the OV file reader at the same time as a stimulation from a classifier, and then a "hit" is registered. What I have noticed is that the stimulation from the classifier does not register at the same timing as the stimulation from the file reader- this causes no score to show (not even 0/20, just blank).

The way I solved this issue was by sending a stimulation from the Resynched stimulations output of the epoching box (which the documentation names as deprecated) instead of a stimulation from the file reader.

As a result, the accuracy box receives 2 stimulations at the same time- one from a classifier, and one from the associated Resynced epoching box.
I, myself, never had to use the resynched output (as you can see in my "ugly" scenario above). I can not really think of any reason why it does not work in your case. However, maybe you can make use of the stimulation filter box and the streamed matrix multiplexer box as explained in the scenario lbonnet provided. I also tried to change your scenario that way (as attached to this post), but could not test it. The use of these boxes could maybe reduce the complexity of the scenario.

eeg wrote: I am trying to distinguish 6 English phonemes (W, Ah, T, S, Uh, N) from Broca's area in order to derive 'silent speech'. This new accuracy box helps as it allows me to determine which of the phonemes was actually detected, and thus I can learn which is the most easily differentiated (hopefully I can achieve at least 25% accuracy for statistical significance, but as mentioned in other threads, it will be very hard to distinguish more than 3 different thoughts).
This is quite interesting research, I saw a demostration of an EMG based silent speech device on the BIOSTEC conference by the lab of Professor Schultz (http://csl.ira.uka.de/index.php?id=169&L=1). So I'm curious about your results as you're doing it based on EEG which is probably a lot more difficult.

Dieter
Attachments
Example1.xml
(67.25 KiB) Downloaded 300 times

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by yrenard »

Dear eeg,

I actually already updated the classifier processor box in the SVN repository. By the way, the classifiers themselves are not yet ready for multi class. Baptiste is currently working on a multi class SVM so stay tuned. It will indeed be announced, at least in the release message where this functionality is included.

I am also very interested in what you are trying to do. Keep us posted !
Best regards,
Yann

ddvlamin
Posts: 160
Joined: Thu Aug 13, 2009 8:39 am
Location: Ghent University
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by ddvlamin »

Hi,

I thought the classifier processor box supported multiple classes since the last release, but for some reason I can't add any settings for a third class. Could it be that it is not yet updated?

Best regards,
Dieter

lbonnet
Site Admin
Posts: 417
Joined: Wed Oct 07, 2009 12:11 pm

Re: Classifier Processor Usage with more than 2 classes.

Post by lbonnet »

Hi Dieter,
I thought the classifier processor box supported multiple classes since the last release, but for some reason I can't add any settings for a third class. Could it be that it is not yet updated?
The Classifier Trainer supports N classes now so you can have an estimation of the accuracy through k-fold test.
However, the classifier processor is not multiclass-ready yet.
Our main problem is related to the classifier processor outputs. The class labels (stimulations) are ok, but we have not decided yet what to do with the "classification state" (Streamed Matrix) output.

For example, for LDA it was the distance to the hyperplan. But what can we use for 3-class SVM or more ?
If you want to share your ideas, you're welcome :)

Laurent-
Follow us on twitter >> openvibebci

Checkout my (old) blog for some OpenViBE tips & tricks : here !

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: Classifier Processor Usage with more than 2 classes.

Post by yrenard »

Dear Dieter, Dear Laurent,

let me rephrase a couple of things so everything is clear.

The Classifier Trainer and Classifier Processor boxes are ready for multi-class handling. (Actually, I just checked and maybe we messed somethings in latest commits, it is not possible to add class labels to the "Classifier Processor" box ; still it you add them manually in the XML scenario file, the box will handle them correctly).

But, these two boxes don't perform the classification itself. They only forward the feature vectors in a standard representation to dedicated algorithms (see this article for brief overview of how to add classifier algorithms suitable for the Classifier Trainer and Classifier Processor boxes).

Existing classifier algorithms include SVM and LDA and they don't yet support multi class. Still if you want to include a multi class classifier, you can, if you want to derive the existing 2-classes classifier, you also can :)

Now, while discussing this multi-class issue internally, we also had a discussion on what should be sent out of the Classifier Processor. This box currently exposes the "Classifier State" which a weak data and can vary a lot from a classifier to another. In the case of LDA, this is the hyperplane distance (negative values for one class, positive values for the second one). In the case of SVM, this is the likelyhood to be in the first class (comprised in 0-1 range). We all agreed that having a standardized likelyhood to be in each class would be interesting in addition to the current "Classifier State" which makes it hard to be reused by later boxes. Still, this data is interesting as it is totally unchanged and can be used when the classifier is know not to change.

Finally, extending the Classifier Trainer and Classifier Processor boxes to multiclass causes some question about the classifier state that gets out of the classifier processor. Indeed, these parameters have been handled as a single value in the particular case of LDA and SVM until now, even if two classes were present. Changing this output to two values in case of 2-class classifiers may break the compatibility with boxes following the classifier processor. So this should be considered with care, thus the discussions that we currently have to make the design clear.

As a conclusion, if you are only concerned with classes (the simulation output), just add the class labels manually to the XML scenario file, and code a multi-class classifier algorithm :)

Baptiste has begun changes in the SVM so that it can handle multiple classes. Not sure how far this development is right now.

I hope this helps,
Yann

Post Reply