Epoch Data Extraction Over Concatenated Files using Python Script

Working with OpenViBE signal processing scenarios and doing scenario/BCI design
Post Reply
openvibeexplorer
Posts: 19
Joined: Wed Mar 18, 2020 2:48 pm

Epoch Data Extraction Over Concatenated Files using Python Script

Post by openvibeexplorer »

Hello there,

I am stating my issue below:

I am dumping the samples in 600 msec epoch after every stimulation, into an excel sheet using python.

I have a sampling frequency of 256 Hz. I am decimating the signal by 4. And I am considering 0.6 msec epoch. This gives me 38 samples per epoch per channel. I have 4 such channels. Which gives me 152 samples per epoch. Below is the python script and the scenario to extract the 152 samples in each epoch.
epoch-data.py.txt
(3.12 KiB) Downloaded 109 times
https://drive.google.com/file/d/11lTS1V ... sp=sharing

I have two ov files. At first, I am dumping the samples in every epoch from each of the ov files, in two separate excel sheets. Then I concatenate the two ov files and I dump the samples in epochs (from the concatenated ov file) in a single excel sheet.

I am expecting the samples in concatenated excel sheet should match with two separate excel sheets. That is, the epoch data after concatenation should be the same as before concatenation. However, they do not match. It gives me an impression that there is some delay involved. Below is my concatenation configuration.

https://drive.google.com/file/d/1jTGP6i ... sp=sharing

There are chances that we will miss some important data due to concatenation delay. How should I fix this concatenation issue?

Thanks and Regards,
Swati.

Thomas
Posts: 210
Joined: Wed Mar 04, 2020 3:38 pm

Re: Epoch Data Extraction Over Concatenated Files using Python Script

Post by Thomas »

Hi Swati,

Thanks for the detailed explanations and the python script.

I tested it with some generated signals, and indeed I could see what seems to be a delay, giving slightly different values in the concatenated file (difference in the order of 10^-11).

Also what I noticed is that this difference is only happening for the samples of the first file. For the second file, there is no difference between the epoched samples of the file alone and the epoched samples in the concatenated file?
Do you see this as well ?

To understand better what the concatenation issue might be, I tried the same whole process with CSV files instead of OV files for the signal data. This way I was able to visually compare the two files with the concatenated file.
And it appears that the concatenated file contained exactly the first two files, with the same values and times and stimulations in the correct place, so the concatenation process seems to be working ok.
However, I still see a discrepancy between the epoching of the first file, and the first part of the epoching of the concatenated data. Again the second part of the epoching of the concatenated data is the same as the epoching of the second file alone.

This leaves pretty unclear what the source of the problem might be.

Next steps I would go for would be to remove boxes (one at a time) from scenario (the temporal filter, and the signal decimation) in order to keep the scenario to the minimum and narrow down where the issue might come from.

This is all I have for now.
Have you made any progress on your side?

Cheers,
Thomas

openvibeexplorer
Posts: 19
Joined: Wed Mar 18, 2020 2:48 pm

Re: Epoch Data Extraction Over Concatenated Files using Python Script

Post by openvibeexplorer »

Hello Thomas,

Thanks a lot for understanding my issue and confirming it from your side. I am working on a P300 Speller and I am gathering 360 characters data for training a CNN-1D network model. For this I need to concatenate 36 OV files (Each file containing data for 10 characters) Out of 36 files, the epochs of the FIRST file match in the concatenated excel. The epoch data of next files does not match. It looks like the delay propagates along all the files during concatenation.

After realizing the above issue, I tried a brute force approach. I am creating each individual excel separately which gives me 36 excels. Then I combine the excel sheets with the following command on windows.
copy *.csv combined.csv

This process of creating 36 separate excel sheets is very cumbersome. However, I do not find sufficient time and skills (c++ understanding) at my hand to fix this issue. It would be of great help if this issue is fixed.

Thanks and regards,
Swati.

Post Reply