Statistics Debug Scenario

Concerning processing components: filters, file load/save, visualizations, communication ...
Post Reply
LeeFX
Posts: 24
Joined: Wed Dec 17, 2014 2:52 pm

Statistics Debug Scenario

Post by LeeFX »

Hi,

I made a debug scenario to compare different (cumulative / real-time) Descriptive Statistics (Mean, Variance,...) Calculation Methods in openvibe and to crossvalidate the results with other programs (SPSS and EEGLab). Mainly I wanted to compare the DSP Raw Signal Results with the FFT Results.

The reference statistic results were calculated with SPSS - the EEGLab Channel Statistics seem to match the SPSS results exactly.

Scenario Screenshots

Statistics
Image
better resolution http://i.imgur.com/rmm6RRo.png

Statistics Result Displays
Image


SPSS Descriptive Statistics Output

Image
C3 = Raw Data
C3centered = Raw Data (Baseline / DC-Offset removed)
* C3absolute = Magnitude / Amplitude
C3power = Magnitude²

Reference Data:
The reference data is the default ${Path_Samples}/signals/bci-motor-imagery.ov file, so no additional data needed
to be downloaded.
(However this set turned out to be not the best suited dataset, due to a signal offset and zeroes at the end of the file)
Simple Preprocessing was necessary:
http://i.imgur.com/cL6Gsr2.png

original and transformed Signal Data Histogram (just for fun)
http://i.imgur.com/XVcLB8X.png


Scenario File
Statistics Comparison Debug v0.02.xml
(159.37 KiB) Downloaded 233 times

My conclusions in short:

*I could not get comparable results of the Signal Amplitude calculation and the FFT Amplitude calculation.
(Here I assume a mistake in the FFT calculation chain)

*The Univariate Statistics Box did not show useful results besides of the mean
(wrong usage)?


*Overall Epoch Variance Box statistics calculation seems to be the most robust method and
showed very similar results in comparison to the SPSS output

Now it would be great to be able to capture the final (cumulated) calculation step to export as a single row or case.

If you have any suggestions, feel free to share it. Hints to bugs and errors are also very appreciated ;)

Best,

Lee

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: Statistics Debug Scenario

Post by jtlindgren »

Hi Lee, thanks for the detailed report!

Here's a few thoughts about it,

- you expect statistics computed from raw signal and from fft signal to be similar? Do you have some reference listing the expected equivalences?

- univariate statistics and the epoch variance box are different in the following sense. Assume the incoming matrix chunks in the stream are A1,A2,A3,... The first box outputs one vector for one chunk. So the statistics are computed inside each A separately. In the Epoch Variance box, it will compute the statistics across the chunks, depending on how many you choose. Lets consider the mean. The univariate statistics computes mean of every channel over the matrix colums. So if B1 is the output for A1, then it will be a vector and B1(1) = 1/k * ( A(1,1) + A(1,2) + ... + A(1,k) ) where k is the chunk size or number of samples. In the Epoch Variance box with the 'Epoch Block Average' option, what is done instead is that matrix B1 = 1/c * (A1 + A2 ... + Ac), where c is the epoch count scalar and the sums are matrix sums. At least this is how they should behave. Then the other statistics are computed afaik in a similar fashion.

- If you do epoch variance / epoch average and univariate statistics after each other, you'll be computing some sort of statistics of statistics. This is actually sometimes done but only when you want to examine how the statistics themselves behave ('how much the variance varies', for example)

- One way to get the last output of some box, you can try to rely on the 'end of file' stimulus that your data can contain. If it doesn't contain this, you can use the timeout box after the file reader box to for example send the timeout after a few seconds of no more data. Then, you write everything the box of interest outputs to a CSV file, and then make 'Run Command' box to react to the end of file stimulus, and use some windows/linux shell command to extract the last line from the stored file. For example on cygwin or linux you could use the command 'tail'.

- Just a general idea about debug: instead of a real file, try using Noise Generator as signal. This way you know exactly what properties are expected from the signal.

If you feel there's a bug in the openvibe stats computations, could you make a bit more simplified scenario that does just the computation which is different from some expected reference?

Edit: removed one bad suggestion ;)

Hope this helps,
Jussi

LeeFX
Posts: 24
Joined: Wed Dec 17, 2014 2:52 pm

Re: Statistics Debug Scenario

Post by LeeFX »

Hello Jussi,

thanks for your quick review.

(1) I am not sure, but I had suggested that computing the Amplitude from a raw discrete signal [ (Magnitude/)Amplitude = (|x| / 2) ? ] should in some way be similar to calculating the overall amplitude of the whole frequency spectrum with the FFT [ Amplitude = sqrt[real-part(FFT(x))² + imaginary-part(FFT(x))²] ]

So my assumption was that
Signal -> Temporal Filter -> Power Calculation -> Average over time
=!
Signal -> FFT Amplitude -> Power Calculation -> Spectral Average -> Average over time

And for being the Phase information that is additionally retrievable with FFT:
Under Computations Using the FFT (p.4)
The power spectrum shows power as the mean squared amplitude at each frequency line but includes no phase
information. Because the power spectrum loses phase information, you may want to use the FFT to view both the
frequency and the phase information of a signal.
Cerna & Harvel (2000). The Fundamentals of FFT-Based Signal Analysis and Measurement. National Instruments.
http://www.iaa.ncku.edu.tw/~jjmiau/down ... alysis.pdf

I mean, I want to make sure that my FFT Settings are correct and comparable to other Software results, and I noticed, that calculating the
amplitude from FFT seems to be very sensitive to certain Settings (Epoching, low-cut Filter (0-2 Hz), etc.). Overall I want to make sure
that my processing is reliable.
(to be honest, I didn't calculate FFT with another program yet for this dataset for comparison, so my question might have been obsolete by now)



(2) Yes I expected that the univariate statistics box is calculating 'horizontally' over all the samples coming each point from the buffer. Like the Signal Average box!? Fair enough, I don't know exactly for what application it is best used for - probably for analysing time-locked parallel ERPs?!


(3) tgaugry said that the end-of-file query will be a feature of the csv-writer box in openvibe 2.0, this sounds like a reasonable plan that might facilitate result and report creation, I guess


I did not find something, what I would refer to as bug yet, except of the Average Box that seemed to produce a rounding error for very long (1h) datasets, as the Univariate Statistics Box underlying Mean calculation did not show this increasing deviation, but I want to try this again to make sure it was not my mistake.

:)
Lee

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: Statistics Debug Scenario

Post by jtlindgren »

You can use a free package like R to test FFT equivalences. Likely its also in Octave. Simply create some long random vector, take that as a signal and then see if your expected equivalences hold (e.g. something from fft(x) vs something from x). You can also import CSV files to both of them to compare ov results if you like.

In any case I'm not surprised if fft is sensitive to window size etc. In principle you should be able to extract band power features both via FFT+power and via temporal filter + squaring (remember to pick the frequencies of interest from the fft, dont take all of it!), but it might be difficult to get exactly the same numeric values out of the two, for example if you run temporal filter on a continuous signal and fft for small chunks, the latter might have some aliasing effects.

Here's some discussion on the subject,

https://electronics.stackexchange.com/q ... nipulate-s


Cheers,
Jussi

Post Reply