LSL Export (Gipsa) - Weird issue with lost markers

Concerning processing components: filters, file load/save, visualizations, communication ...
Post Reply
faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

Dear Friends,

Scenario: P300 Speller (based on p300-xdawn-4-online.xml)
Version: 2.1.0

We found that LSL Export (Gipsa) is loosing the first Marker for the P300 Speller when it is sent at the same time as "Segment Start" (32771)

<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Trial start
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Clear votes
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 0000008003,32771 at 15.048268 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 0000008206,33286 at 15.048289 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 000000810a,33034 at 15.048311 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Label
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 000000800b,32779 at 15.048331 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 000000800c,32780 at 15.322899 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 0000008206,33286 at 15.355623 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Stimulation 0000008103,33027 at 15.355700 (now = 0.000000)
<Box algorithm::(0x0000445b, 0x000068e3) aka P300 accumulator> <In Script> Label

Steps to reproduce
- Add LSL Export (Gipsa) and connect the input Signal to "Acquisition Server" and Stimulator input to "P300 Speller Stimulator".
- Receive the markers with the LSL stream name. The first channel contains the timestamp, and the last channel contains the Markers.
- The first received flashing marker for the Segment is lost.

Thanks for any help !

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

Hi, I can investigate if there's a bug in the box, but what you are doing should not be done in general. Please read

http://openvibe.inria.fr/tcp-tagging/

In short, if you have stimulations generated by a box and have some rendered events corresponding to them, and expect to somehow analyze the EEG corresponding to these events, you will have trouble. This is because the timestamps of the stimulations will be wishful thinking and will not be correctly aligned with the EEG. The way to address this is for the renderer to forward the stimuli with a corrected timestamp to the Acquisition Server after there is precise knowledge when the rendering actually happened. Hence, stims recorded to file or sent outside openvibe if related to EEG analysis must be recorded from the Acquisition Client box.The main exception are classification results used to control applications, these can be safely taken from the classifier processor box.


Cheers,
Jussi

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

Ok, thinking about your report, is the problem because the stimulations are exactly at the same time? Since the gipsa box is using a signal channel to transmit the stimulations, maybe its just not possible to put two numbers in one slot. To get around this, you can either use the non-gipsa box or adjust the timings to be slightly different.


Cheers,
Jussi

faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

Excellent Jussi thanks a lot !!

The reason we are doing this merging of stimulations, is because of this thread: viewtopic.php?f=6&t=9421&p=15825#p15825. The markers that the LSL Export (Gipsa) is sending when its input is connected to the "Acquisition Client" output of Stimulations, are all zero when they are read in the first column of the LSL stream. If they are connected to an LSL Export Box, on the other hand, markers are perfect (but they come in a different stream).

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

If you really want to use the gipsa box, I imagine the box will work correctly if it gets proper signal in one input and stimulations in the other. Can you try that? The design philosophy of the gipsa box likely is to have the stimulations aside the signal. If you use the other box, yes you'll have different streams but the timestamps are compatible in both streams which can then be used for alignment. Separete streams afaik is the 'more lsl' way to do it.


J.

faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

Thanks J !!

Yes we tried this on an p300-xdawn-4-online.xml scenario:

*) Acquisition Client: Signal Stream --> Input Signal, Stimulations --> Input Stimulations LSL Export Gipsa
*) Acquisition Client: Signal Stream --> Input Signal, Stimulations --> Input Stimulations LSL Export
*) Acquisition Client: Experiment Information -> Input Stream 1, Signal Stream --> Input Stream 2, Stimulations --> Input Stream 3, Generic Stream Writer

The LSL Export under the names openvibeSignal and openvibeMarkers is correct (markers are there).
The exported file is also correct (ov file)

However the LSL Export Gipsa under the name "openvibeSg" as "Markers" and "Signals" (types of stream, I tried both) do not contain any marker at all.

Really, thanks a lot !

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

Hi, I'm trying to figure out whats the problem here. Do you in some circumstances have two acquisition servers running at the same time? These should have at least two ports different from each other: the client port (normally 1024) and the tcp tagging port (normally 15361). However, the latter will be a problem as usually that port cannot be changed from the OV box side.

If there's two acquisition servers trying to put tcp tagging to the same port, it seems the stimulations can vanish. I'll see if I can change the code to produce an error if the port is already taken + add port config to the designer side.


J.

faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

Do you in some circumstances have two acquisition servers running at the same time?

Not at all! I am just having one acquisition server (generic oscillator) and I have an Acquisition Client Box on the Designer which gets the data.
From the acquisition client I am sending the stimulations and the streams to: Generic Stream Writer, LSL Export and LSL Export (gipsa)
The designer also has the P300-stuff (which is derived from the sample scenario): Launch -- P300 Speller Stimulator -- Target Letter Generation -- P300 Speller Visualisation

I am not sure if it is a problem or it is something I am doing wrong, but the markers are perfect in the OV File and at the LSL Export (I have a C++ program running LSL which gets the data). However markers are completely empty from the LSL Export (gipsa).

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

If I connect "Sinus Oscillator" and "Clock Stimulator" to LSL Export GIPSA, the result is that the LSL signal stream will have +1 signal channel, and I see the stimulations that the clock stimulator generates there as float numbers on that extra channel. When there is no stimulation, the value is 0. Do you get something different?


J.

faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

I just tried that and I get exactly the same as you. I can see the tick stimulator from the clock out of the LSL Export Gipsa.

The issue is found when you connect the Stimulation input of the LSL Export (Gipsa) to the Acquisition Client (which should be the way to do it). For some reason, which is also reported on this thread viewtopic.php?f=6&t=9421&p=15825#p15825 it seems that markers are lost and the channel +1 that you get on the LSL stream only contains zeros, no marker at all.

I really thank a lot for your help !

faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

I coded an LSL Gateway in C++ that receives both streams from the LSL Export Box, paste them together and send them as it should have been sent by the "LSL Export Gipsa" (with +1 channel) and I can see all the markers. I bet that there must be some problem with this box when it's connected to the acquisition client.

I couldn't find the box code in the source code. Is it used as compiled binary library ?

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

Hi, the gipsa lsl box source code is under extras/contrib/.

Thanks to your report, I found out that the box can indeed have some problems synchronizing the signal and stimulation streams, and that sometimes it'll never pass out the stimulations. This morning I made a fix attempt for the problem. Can you give the attached file a try and report if it helps in your situation?

How to do it: Simply extract the zip contents to replace the original .dll of the OpenViBE installation in the bin/ directory.

Cheers,
Jussi
Attachments
openvibe-plugins-contrib-network-io-lslexportgipsa-fix-candidate.zip
(24.64 KiB) Downloaded 198 times

faturita
Posts: 8
Joined: Fri Jun 08, 2018 3:15 pm

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by faturita »

Jussi !

Absolutely thank you !! You rule !!! Stims are now there !!

However, one more issue I would like to mention is the topic of this thread (after all).

At time 0.000 sec <Box algorithm::(0x000000ea, 0x000057a1) aka Launch> Lua script terminated
At time 35.875 sec <Box algorithm::(0x00007eb2, 0x000016f7) aka LSL Export (Gipsa)> Received two stims with exactly the same timestamp. Keeping the first one.
At time 35.875 sec <Box algorithm::(0x00007eb2, 0x000016f7) aka LSL Export (Gipsa)> Received two stims with exactly the same timestamp. Keeping the first one.
At time 35.875 sec <Box algorithm::(0x00007eb2, 0x000016f7) aka LSL Export (Gipsa)> Received two stims with exactly the same timestamp. Keeping the first one.
At time 283.750 sec <Box algorithm::(0x00007eb2, 0x000016f7) aka LSL Export (Gipsa)> Received two stims with exactly the same timestamp. Keeping the first one.

There are valid stims that happen at the same time. Like "Segment Start" and the stimulation that correspond to some row or column flashing. I believe that if one of them are discarded it may loose the marker of a valid segment. I just rerun the scenario again and found that now there are stims, but not all of them.

For instance I run the P300 scenario, flashing all the rows and columns of the speller matrix twice (which will give you for sure at least the 12 stimulations for rows and columns) and I only got two:
33034
33035

I believe that perhaps instead of discarding one of the stims it should be better to just tweak the timestamp a little bit (adding a microsecond or something) to make different timestamps.

jtlindgren
Posts: 775
Joined: Tue Dec 04, 2012 3:53 pm
Location: INRIA Rennes, FRANCE

Re: LSL Export (Gipsa) - Weird issue with lost markers

Post by jtlindgren »

Thanks for the report! Its difficult to support multiple simultaneous timestamps due to the boxes design where there can be one stim per one sample. I suggest simply modifying the stimulus generator to have sufficient delays.

But actually this makes me realize that instead of checking for timestamp equality, the box should check the stims are at least 1/samplingrate distance from each other. I'll fix that.


Cheers,
Jussi

Post Reply