Troubleshooting motor imagery spatial filter (mi-csp-2-train-CSP.xml)

[membrane286]

Hello,

I've been trying to use my OpenBCI equipment (Cyton+Daisy 16 Channels, Ultracortex Mark IV) in conjunction with openvibe to detect motor imagery. When I get to the spatial filter step, the mi-csp-2-train-CSP.xml always runs indefinitely after displaying a warning "...CSP Spatial Filter Trainer> Eigen vector decomposition failed...". In stream reader, I made sure to select the .ov file generated by the graz visualization in mi-csp-1-acquisition.xml, but the filter runs indefinitely regardless. I've tried uninstalling and reinstalling, using different versions of openvibe (3.3.0 and 3.5.0), putting the .ov file in the same folder as the xml files, but nothing seems to fix this issue. I should also note that the mi-csp-0-signal-monitoring.xml file appears to be generating accurate waveforms and I always allow the graz visualization to close by itself (I've heard that closing the window manually will cause problems). What would you suggest I do to run the spatial filter successfully?

This video walkthrough is what I'm trying to accomplish (although instead of an oscillator I use LSL): https://www.youtube.com/watch?v=prCIWIqyJus

Thank you for your help.

[Thomas]

Hi membrane286,

Did you make any progress on this ?

The CSP Spatial Filter Trainer box is trying to get eigen vectors and eigen values from covariance matrices from its two signal inputs.
The error message indicates that it failed at doing so.

This could be for two reasons:
- The matrix inversion on the covariance matrix for the right arm trials may have faileld. I think this can happen if the CSP spatial Filter Trainer box was not provided with enough data.
- The calculation of the eigen vectors itself could have failed, but I am not sure what could have caused this yet.

If you can share the OV file that you have generated I could test a couple of things to try to narrow down the issue.

Kind regards,
Thomas

[membrane286]

Hi Thomas,

Thank you for your reply. Interestingly, I tested the spatial filter on an ov file generated from running the graz visualization for no longer than 30 seconds without putting on the EEG headset and it appeared to run correctly. I have not gotten the spatial filter to work on 7 minute trials where I have the headset on. I've attached screenshots of the dialog boxes that were produced, the long trial being the "eigen vector decomposition failed" and the short trial being the successful one. I tried to attach the ov files but it wouldn't let me due to an "invalid file extension" error. How should I go about attaching an ov file?

Thanks again.

[Thomas]

Hi,

That's strange indeed.

You could send me the files using any file transfer platform and I'll try to dig a bit more.

Thanks,
Thomas

[membrane286]

Hopefully this link works (it expires in 7 days): https://www.transfernow.net/dl/202401077gqgcq85

Thanks again

[membrane286]

Here's a new link in case you weren't able to access the files: https://www.transfernow.net/dl/20240114j77IICXT

[membrane286]

Hello,

Were you able to download the ov files? If not I can send another link. In the meantime I'll start doing more testing and see what works.

[membrane286]

I did some testing with a shorter acquisition session (number of trials for each class was set to 2) and was getting more or less the same results. I put a link to the ov file for this session (expires February 2nd):
https://www.transfernow.net/dl/20240126KDGYUgR3

Here also are the exact messages produced for certain train triggers on the CSP spatial filter trainer block (I tried different ones):

-with OVTK_GDF_End_Of_Session (default) filter dimension 2
[ INF ] At time 69.125 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> Received train stimulation - be patient
[ INF ] At time 69.125 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> Data covariance dims are [16x16]. Number of samples per condition :
[ INF ] At time 69.125 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> cond1 = 2 chunks, sized 500 -> 1000 samples
[ INF ] At time 69.125 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> cond2 = 2 chunks, sized 500 -> 1000 samples
[WARNING] At time 69.125 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> Eigen vector decomposition failed...

-with OVTK_StimulationId_Train filter dimension 2
[ INF ] At time 74.180 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> Received train stimulation - be patient
[ INF ] At time 74.180 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> Data covariance dims are [16x16]. Number of samples per condition :
[ INF ] At time 74.180 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> cond1 = 2 chunks, sized 500 -> 1000 samples
[ INF ] At time 74.180 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> cond2 = 2 chunks, sized 500 -> 1000 samples
[WARNING] At time 74.180 sec <Box algorithm::(0x00004125, 0x0000536f) aka CSP Spatial Filter Trainer> Eigen vector decomposition failed...

-with OVTK_StimulationId_TrainCompleted the program ran indefinitely without warnings

it seems like when I run this program with an erroneous ov file it terminates as expected, because for one of my tests the battery on my headset died mid trial but the ov file was successfully run through the spatial filter. However, upon running that through the successive programs the resulting graz visualization did not display the blue bar representing the classifications.
Let me know if you find anything. Thank you.

[kyunghowon]

Hello.

I looked at your latest attachment, and It seems that channel 14 is dead.
Your covariance matrix will not have full rank, yielding one eigenvalue of nearly 0.

Depending on how internal scripts deal with less rank matrix, I think it could print an error.
(e.g., Reducing dimension or enforcing full rank input with error messages, etc.)

Best regards,

[membrane286]

Hi,

Wow that dead channel was really messing it up. I had known that a couple channels weren't working properly (namely channel 12 and channel 14) but I naively assumed that wouldn't be an issue since there were 14 other working channels. Maybe I could have selected specific channels or even run the OpenBCI GUI with 8 channels but I wanted to leave everything at default for the time being. Regardless, I replaced the faulty electrodes today and sure enough all the programs worked properly! The classifier was even 90% accurate if I'm reading the messages correctly, which I've copy/pasted below:

[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Received train stimulation. Data dim is [540x6]
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> For information, we have 270 feature vector(s) for input 1
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> For information, we have 270 feature vector(s) for input 2
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> k-fold test could take quite a long time, be patient
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Finished with partition 1 / 5 (performance : 81.4815%)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Finished with partition 2 / 5 (performance : 48.1481%)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Finished with partition 3 / 5 (performance : 82.4074%)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Finished with partition 4 / 5 (performance : 65.7407%)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Finished with partition 5 / 5 (performance : 83.3333%)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Cross-validation test accuracy is 72.2222% (sigma = 13.6711%)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Cls vs cls 1 2[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Target 1: 64.8 35.2 %, 270 examples
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Target 2: 20.4 79.6 %, 270 examples
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Training set accuracy is 90.7407% (optimistic)
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Cls vs cls 1 2[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Target 1: 85.9 14.1 %, 270 examples
[ INF ] At time 137.219 sec <Box algorithm::(0x0a5a6a4a, 0x1d92a778) aka Classifier trainer> Target 2: 4.4 95.6 %, 270 examples

I'll definitely need to research the machine learning concepts implemented in the boxes to better understand what's actually happening, but I'm glad it's working now!

Thank you for your help.

[kyunghowon]

Hello.

I am glad to hear that you resolved the issue.

Let me explain reading the output.
It may be redundant if you know already, but I hope this helps, just in case.

For now, your classifier is optimized based on all your input, yielding optimal performance.
Training accuracy is calculated by the same data that used for training the classifier.

However, when you apply your classifier in a different scenario, which is a test, your classifier should classify unseen data.

In general, test accuracy is lower than training accuracy if the classifier is optimized well only for training data, especially when the training data is not enough (overfitting). Also, if training data and test data are very different, for example, testing different participants from the training will result in very low accuracy through the vanilla CSP.

You can somehow predict your test accuracy from the average k-fold cross-validation accuracy of 72.22%.

It splits your data into five folds (540 feature vectors -> 108 feature vectors for each fold), then trains a classifier using 4 folds and tests the trained classifier using the rest unseen 1 fold data that is not used for training the classifier, and repeats five times while changing the test fold. (You can see the k-fold validation includes near random to 83% accuracy depending on the fold)

Best regards,