Poor performance of Support Vector Machine on Openvibe ?

Come here to discuss about OpenViBE in general!
Post Reply
Jeff_B
Posts: 37
Joined: Wed Apr 11, 2012 1:31 pm
Location: Nice - Alpes Maritimes

Poor performance of Support Vector Machine on Openvibe ?

Post by Jeff_B »

Hi Openvibers,

I am trying to use the SVM classifier in the motor imagery scenario bundled with OV (the one using CSP and a LDA classifier).

I have extended CSP to CSSSP using a delay lines box (generating delayed versions of the normal signal flow) and I use the regularized CSP training. Using the Emotiv headset, the sampling frequency is 128 Hz, and I have set the delay box to generate 8 delayed signals (per channel) with a base delay of 2 ms (my 1st guess/try). It enables to estimate specific 8th order FIR filters along with the spatial filtering, effectively enabling to implement a spatio spectral filtering: the training will select the most appropriate spatial filters along with the most discriminant frequency filters. With a LDA classifier, this provides a 20% on the classification (73% for the CSP, 92% for the CSSSP). The cost of this way to implement a CSSSP: the size of the covariance matrix (of the sensors) is directly multiplied by the order of the FIR (the number of delayed signals) but at the 8th order with 14 channels (~112 channels), it seems to be able to run in real time without problem.

My problem is when I try to use the SVM classifier with a radial basis kernel. I have already used these classifiers and if a simple LDA correctly classifies 92% of the features (48 spatiofrequential filters or features are produced by the CSSSP), I am not expecting SVM to perform around 50% whatever the (gamma, cost) settings of the radial basis SVM. This is what I get though.

I then extracted the feature vectors used by the SVM within OV to process them offline on Matlab with exactly the same libraries than the one used by OV (libsvm). While I get around 50% accuracy on OV for around 20 (gamma, cost) settings, I get 100% accuracy offline on Matlab for a large area of the (gamma, cost) plan. Again, same librairies (libsvm) used, same feature matrix used.

I had to filter out a few NaN features among the 48 (I do this both on OV and Matlab), but on Matlab, the SVM with RBF performs equally well whatever the (gamma;cost) settings and whether I normalize the features or not: I can feed the SVM classifier with features whose distribution is largely outside [-1;+1] or [0;1] without degrading the performance. The normalization of the features (and the filtering of NaNs) did not solve the very poor performances of the SVM under OV.

I have not compared the support vectors (SVs) and associated coefficients produced by OV and Matlab yet, 1) but they are approximately in the same number, and 2) the SVM on OV seems stuck on one class meaning it always classify the same way (the estimated labels are either always class 1 or always class 2). Basically, two same functions/methods from the libsvm package ares used by OV and Matlab: svm_train and svm_predict, and the accuracy (confusion matrix) of the classifier is estimated using svm_predict. It could very well be that the training of the SVM is ok but the prediction phase (stuck on one class) is a problem... still digging... I think that the SVs produced by the SVM on OV are stored in myclassifier.cfg which is next used by the online session: I am going to extract them from Matlab to be able to compare OV and Matlab but I was wondering if I could copy paste the OVs and coeff produced by Matlab (100% accuracy) into this xml file myclassifier.cfg so that it (the SVM estimated by Matlab) can be reused on OV ?

Has any of you ever succeeded to get good classification performances with the SVM on OV ?
Best wishes for all of you !
Jeff

Jeff_B
Posts: 37
Joined: Wed Apr 11, 2012 1:31 pm
Location: Nice - Alpes Maritimes

Re: Poor performance of Support Vector Machine on Openvibe ?

Post by Jeff_B »

1st SVM comparison between OV and Matlab:
- the support vectors (up: OV, down: Matlab):
Image
- the associated coefficients (red: Matlab, blue: OV)
Image
The outputs of svm_train (support vectors and coeffs) in Matlab or OV seems very similar even if not strictly equals.
Will compare the predictions (svm-predict) next.
Jeff

Jeff_B
Posts: 37
Joined: Wed Apr 11, 2012 1:31 pm
Location: Nice - Alpes Maritimes

Re: Poor performance of Support Vector Machine on Openvibe ?

Post by Jeff_B »

Hi there,
Trying to understand why the training phase (making use of the svm_train function of the LIBSVM library) of the SVM classifier outputs slightly different support vectors & associated coefficients, I recompiled OV with the last LIBSVM libraries (version 321 instead of 291 in the OV repository), the same I use in my Matlab mex files: these different LIBSVM versions do not explain these slight differences in the training phase. Having thought to this, I believe that the set of training runs is not exactly the same in OV (which seems to train on a subset/partition) and Matlab (who trains on the whole data set).
Then I looked to the main problem, the classifying phase (making use of the svm_predict_probability or svm_predict_values depending on the classifier configuration) of the SVM classifier. Remember it was stuck on one of the two classes whatever the (gamma; cost) setting, giving a performance close to random guessing. Exploring the data flow within the classifier, I finally found a redhibitory stuff: the gamma parameter of the model is not correctly transfered to the classification phase (it is zeroed), you have to modify this:

Code: Select all

l_pTempNode = pParamNode->getChildByName(c_sDegreeNodeName);  // Jeff modif
if(l_pTempNode != NULL)
{
	std::stringstream l_sData(l_pTempNode->getPCData());
	l_sData >> m_pModel->param.gamma;
}
into this:

Code: Select all

l_pTempNode = pParamNode->getChildByName(c_sGammaNodeName);  // Jeff modif
if(l_pTempNode != NULL)
{
	std::stringstream l_sData(l_pTempNode->getPCData());
	l_sData >> m_pModel->param.gamma;
}
In case, I added in the classifier this protection (it is present for the training phase but not for the classifying/predicting phase):
if(m_pModel->param.gamma == 0 && rFeatureVector.getSize() > 0)
{
m_pModel->param.gamma = 1.0/rFeatureVector.getSize();
this->getLogManager() << LogLevel_Trace << "SVM model gamma found equal to zero is set to " << m_pModel->param.gamma <<".\n";
}
Another thing: LIBSVM seems to expect the indexes of the features to be between 1 and Nfeatures (not 0 and Nfeatures-1), so I also suggest to modifify these two lines:
l_pX.index=i; should be modified as l_pX.index=i+1;

and
m_oProb.x[j].index=j; shoudl be changed with m_oProb.x[j].index=j+1;

One last thing: when I rcompiled OV with the last LIBSVM librairies, the compiler complained about two undefined functions:
get_svm_type()
get_kernel_type()
There are two similar functions in the LIBSVM package (with the prefix svm_), but these are not the missing functions: I was not able to find them in the whole OV code and I had to implement them in svm.cpp:

Code: Select all

const char *get_svm_type(const int svm_type)
{
	return svm_type_table[svm_type];
}
const char *get_kernel_type(const int kernel_type) // Jeff modif
{
	return kernel_type_table[kernel_type];
}
with their associated declaration in svm.h... for OV to compile again.
But the most important thing of all of this is the correct transfer of the gamma parameter to the classification routine.
This being corrected, as expected, the SVM outperforms the LDA (100% versus 92%) on a large portion of the (Gamma;Cost) plan.
Best,
Jeff

Jeff_B
Posts: 37
Joined: Wed Apr 11, 2012 1:31 pm
Location: Nice - Alpes Maritimes

Re: Poor performance of Support Vector Machine on Openvibe ?

Post by Jeff_B »

And here is the SVM performance with a RBF kernel in the (gamma;cost) plan:
Image
Remember that the inputed features are not normalized or scaled as they should (according to the LIBSVM practical use) and that we still have to figure out how to do this. The combination of "univariate statistics" and "simple DSP" boxes may not help to normalise features since it works on a data chunk to build statistics whereas we need these statistics on the whole feature dataset. And it is feasible on the training phase, but how do we do this in real time, in the testing phase ? (where we do not have the whole dataset but where we get the data sample by sample).
Jeff

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: Poor performance of Support Vector Machine on Openvibe ?

Post by jtlindgren »

Hi Jeff,

very good catch about the gamma. We have some automatic tests for the SVM, but they only cover the linear kernel, so they've missed this problem. Thank you for digging all the way into it, looks like a simple fix but something that is not at all easy to spot. I'll crank a fix in.

About the indexes, if you added the +1, could you get similar support vectors and coefficients in OV and Matlab? Your plot about 'svcoeff' suggest a kind of 'off by one' issue, but curiously it doesn't seem to be there always. Coeffs from 25 to 50 seem aligned. Did you notice operational differences by changing the indexing?

Finally, the OV classifier trainer box does cross validation to estimate the accuracy, but the final, saved model is learnt on the whole dataset. However, the cross validation in this case might be a bit optimistic. For example, your enhanced CSP may be overfitting the data, even more so than regular CSP since there are more dimensions. I've written a short article about this problem here.


Happy hacking,
Jussi

Jeff_B
Posts: 37
Joined: Wed Apr 11, 2012 1:31 pm
Location: Nice - Alpes Maritimes

Re: Poor performance of Support Vector Machine on Openvibe ?

Post by Jeff_B »

Jussi, adding +1 to the indexes did not appear to resynchronize the SV coeffs as I was hoping too... but there remains the possibility that the SV are simply permutated/shifted (I did not compare the SVs of two equals but shifted coeffs). I am surprised that the same methods, now from the same version of the library, with the same inputs does not produce exactly the same results, but I did not dig further.
And you were completely right, adding so many degrees of liberty (lagged versions) to the CSP (along with a SVM also known for that) is a real good receipe to overfit data !

Post Reply