Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000197openvibe-applicationsbug reportpublic2018-08-13 23:222018-08-27 14:43
Reporterjfrey 
Assigned Tojtlindgr 
PrioritynormalSeverityminorReproducibilityalways
StatusassignedResolutionopen 
PlatformOSOS Version
Summary0000197: Use LSL default timestamps to enable proper synchronization
DescriptionHello,

I have been using LSL quite extensively to gather data from different sources. A problem arises with OpenViBE: it pushes a custom timestamp alongside each sample, which disrupts the built-in synchronization mechanism and prevent time correction across multiple machines / sources.

For example if I record multiple streams using LSL LabRecorder, with physiological signals coming from OpenViBE acquisition server on one machine and stimulations / markers coming from Unity on another, then it is next to impossible sync both signals and do a proper epoching when the resulting XDF file is loaded later on for analyses.

A simple fix consists in removing the second parameter in every occurrence of push_sample(), e.g. in plugins/server-extensions/lsl-output/ovasCPluginLSLOutput.cpp for the acquisition server and in plugins/processing/network-io/src/box-algorithms/ovpCBoxAlgorithmLSLExport.cpp for the designer,

Unless OpenViBE uses the timestamp in some fancy way (if ever it reads its own LSL streams??), then it would make sense to rely on LSL, since in its implementation LSL (supposedly) uses the most precise clock available on the system (time since the machine started, using libboost chrono, see lsl_local_clock() in https://github.com/sccn/labstreaminglayer/blob/master/LSL/liblsl/src/lsl_freefuncs_c.cpp [^] ).

Note that a after a look at the source code of the gipsa LSL box, it seems that there the default clock is used, i.e. "m_outlet->push_chunk(mychunk);".

The proposed change is trivial, but if it facilitates the workflow I can edit the code and make a pull request on gitlab.

TagsNo tags attached.
Attached Fileszip file icon openvibe-lsl-timestamps-fix-candidate-20180827.zip [^] (724,644 bytes) 2018-08-27 14:39

- Relationships

-  Notes
(0001263)
dmedine (reporter)
2018-08-20 10:08

One can force LSL to override requests from clients to impose their own timestamps. By creating a configuration file ~/lsl_api/lsl_api.cfg with the lines:
[tuning]
ForceDefaultTimestamps = 1
LSL will force its default timestamps on all outlets. However, the version of liblsl that ships with OpenViBE is an older one that doesn't yet have this feature implemented. OpenViBE should update the version of liblsl that it uses to the latest version: ftp://sccn.ucsd.edu/pub/software/LSL/SDK/liblsl-C-C++-1.12.zip [^] and, if necessary, the redistribution of the LSL source code to the source code therein.

This is a better solution, in my opinion, than changing the source code to OpenViBE. Sometimes people prefer to have the OpenViBE timestamps over the LSL timestamps.
(0001264)
jtlindgr (administrator)
2018-08-20 10:11

Thanks for the report!

I agree we have a problem, as currently OpenViBE seems to be misusing the push_sample(). This is because the pushed timestamp should be relative to the local_clock() according to the LSL API -- which it is not as stamped by the current OV code.

However, removing the stamping altogether may also be problematic: For example, in the case of stimulations, these may be received by the LSL Export box in a chunk with a time range [t1,t2] at some time t2+eps. In principle there is no absolute guarantee that t2-t1 is small. Hence, if we removed the timestamping, a stim at t1 would be tagged by LSL with a stamp "now", even though t1 may be quite quite far in the past.

The challenge here is that we don't really have tests for OV/LSL integration, and we do not know what the typical use-cases are. But nevertheless, I'd like the fix to be one that is 'generally correct' and doesn't introduce new problems.

Would it solve your problem if I tried to make the timestamps relative to the local clock?

Any thoughts?

ps. thanks for the heads up David -- I was just writing this when you posted your comment! Updating the LSL lib might be the best way to address this. However, is it possible to send "stim in the past" if we go that route?
(0001265)
jtlindgr (administrator)
2018-08-23 09:41

I've made a branch doing the LSL 1.12 update which I expect can be merged without issues. However, I'd like your opinion if it'd be additionally useful to change the timestamps to be relative to local_clock(); see my previous comment.
(0001266)
dmedine (reporter)
2018-08-23 09:50

My opinion for now is that OpenViBE should not change apart from the 'hidden' override timestamp option in LSL be implemented. This actually touches on a side discussion in the LSL universe about timestamping. Some people have suggested (and some apps, i.e. the Eyelink app already do this) that it is appropriate to create an additional timestamp channel as part of the data itself. This has pretty significant ramifications for the handling of data loading in offline analysis because for now the synchronization routines rely on the implicit timestamping.

But, for the time being, I think it is ok to update LSL in OpenViBE and leave it be. For the most part, I think the main use case is when people want to record Emotiv data in LSL via the OpenViBE client, in which case the ramifications that you state aren't relevant anyway.

A larger issue is that OV/LSL integration is not well tested. I'm not quite sure what to do about that. Ideally, some grad student somewhere needs a project to do and this can be it, but that is pretty wishful thinking. To be continued, I guess...
(0001267)
jfrey (reporter)
2018-08-23 22:21
edited on: 2018-08-23 22:37

Hello,

Thanks Jussi for the reactivity, and David for this solution :)

Personally as long as there is a way to get the timestamps I want without touching the code and compiling, I am okay with it. However I am not sure that the default behavior should be the "custom timestamp". A naive user would expect OpenViBE to behave as a "regular" LSL server to integrate it in their pipeline smoothly. At the moment there is no obvious reason why this particular clock is used. I do not know anyone using OpenViBE timestamps, but on the other hand comes immediately in mind three instances where the LSL timestamps are desired, myself, a lab I have been visiting recently where ohter physio/VR XP are running(no emotiv involved), the person who posted a bug report on the LSL repo.

What bothers me slightly with the cfg file solution is that if I understand correctly there is no way to specifically force OpenViBE to use default timestamp while retaining the ability to tune timestamp elsewhere (e.g. python scripts). To me the most flexible solution, if really there are people out there relying on OpenViBE's timestamp, would consist in adding an option of the OpenViBE side (possibly... using one of its config file, where one can already tune some under-the-hood parameters).

On a side note, in some other experiment I have been playing with the meta-data of the LSL outlet in order to embed an Unix epoch time in it and get an "absolute timestamp" across various devices for reference (the client reads the epoch upon init and updates it with the LSL clock afterward, best solution I could find for my specific use-case, could be yet another idea for managing time).

David, what do you have in mind about the lack of OV/LSL integration? "Just" have some sort of unit tests, or did you spot specific issues? In the coming year I might have ressources to dedicate to OV dev, and as a matter of fact having robust LSL communications is among my top priorities.

(0001273)
dmedine (reporter)
2018-08-24 16:31

Jeremy, great to hear that you would like to work on this some. I think that unit tests are a good place to start. I'd be curious to know what Jussi's thoughts about what specifically needs testing.

As for this stimulation marker timestamp issue in LSL (that Jussi mentions above), the marker will be at some some sample n in the [t1,t2] chunk of data. The naive way to handle this is simply mark the time at the start of the chunk (now = lsl::local_clock), then add n*samplingrate to that time and assign the stimulation marker that time. However, this would be negated by the timestamp override option...

The other problem with this method is one of micro-timing. You can read about it here: https://github.com/sccn/labstreaminglayer/blob/master/Apps/BrainProducts/LiveAmp/explanation_of_trigger_marker_types.pdf [^]
(0001274)
jtlindgr (administrator)
2018-08-24 16:42

Afaik, we could do a bit better than the naive solution: we have some sample time or stimulation time in OpenViBE which is by default relative to the acquisition starting (0). Its the same both on server and designer side, on start, the time starts at 0. But we can take a stamp of the clock time at the start, and this way get the stamps relative to "now". The stamps can then again be made relative to the lsl's clock. Actually I was thinking of quickly putting this together and asking if either of you could give it a quick spin.

In pseudo, I thought about getting the adjusted stamps as "diffSec = ov_clock_at_0 + stamp_relative_to_0 - ov_clock_now; lslStamp = lsl_local_clock() + diffSec".

I'll take a look at the micro-timing next week.
(0001275)
jfrey (reporter)
2018-08-24 17:28

That sounds the best solution :) I am trying to think of reasons why one would be interested in getting the exact "OpenViBE time"... Not sure if it is worth it, but you could still embed this info in stream metadata (i.e. one value with "OpenViBE 0 time" relative the stream "created at").

About micro-timing, very interesting paper, I never thought about the problem associated with de-jittering. Actually if there is one of the two implementations of LSL within OpenViBE designer (the gipsa one) that translates openvibe stimuli to floats and that then embeds them in a signal stream as a supplementary channel, so in a way this is already covered. (random off-topic thought: if there is a way to signal to the de-jittering algorithm that the unsampled stream is tight to another sampled stream, then it could apply the computations made on the latter to the timestamps of the former?).
(0001276)
dmedine (reporter)
2018-08-27 09:38

Yes to Jussi's solution, and yes to embedding the OpenViBE 0 time in the metadata. Those are excellent ideas.

As for the de-jittering question, that is an interesting idea, but I can't think of a way to actually implement it that makes much sense. I think the best you could do would be another approximation that would also be slightly wrong. I think the easiest solution is simply to make the marker stream have a sampling rate and be mostly empty. Not very elegant, but simple and reliable.
(0001277)
jtlindgr (administrator)
2018-08-27 10:32
edited on: 2018-08-27 10:39

I read David's paper about the micro timing. In regard to OpenViBE, the challenge is that after very early stages (in the Acquisition Server), the 'true' timing information is no longer available due to how OpenViBE was designed. Basically if we forget about drift correction option (which doesn't do de-jittering in any case), how it goes in openvibe is that the amplifier driver packs samples into chunks, or sends them individually, and the AS does the packing. These chunks are interpreted to have a sequence of timestamps [0,t1][t1,t2][t2,t3]... where the t's are derived *deterministically* from a fixed sampling rate and the number of samples received. To be pedantic, these stamps are not even transferred out from the AS, it simply sends out the sampling rate and the Acquisition Client starts the stamping from 0, according to the sampling rate. The stimulation timestamps are adjusted to be compatible with the chunk timestamps (and can have sub-sample precision in theory). If there was a hickup somewhere or the device is jittering, and the samples actually are not equidistant in time, this knowledge is currently in no way available to either Designer or AS plugins that get the data. Its a "too bad, no can do" situation. Due to this, even as Gipsa LSL output box does attach the stimulations as an EEG channel, there's no way that it could de-jitter them, as when it gets the signal and the stims, all that data has already been pressed into a platonic ideal world. No knowledge of the original sample times remain.

All openvibe boxes handling signal are afaik designed with the assumption that the data is equidistant. The assumption probably is that the source (acquisition server, driver, amp) should mangle the received data as best as it can to make it confirm to the theoretical sampling rate with minimal error. Whether this was a good idea or if its politically feasible to change it after 10 years is another question.

Finally, regarding the question about unit tests, I'm not an LSL expert and aside hacking some of the OV components together, I haven't really used the tech. Its of course possible to derive some tests from common sense grounds and knowing what people are usually concerned about (e.g. marker and signal alignment), but I feel the best would be if these could be made from concrete use cases, so whover writes the tests should first extract some knowledge about what are the most important tasks done with an OV+LSL combo.

(0001278)
jtlindgr (administrator)
2018-08-27 14:43
edited on: 2018-08-27 14:44

I've attached a fix candidate that tries to make the timestamps relative to the local clock. Jeremy, if you have some lsl/ov related business ongoing, could you give this a try and let us know if it works better than the previous approach?

Of course anybody is welcome to try it.

The corresponding git entry is commit b3b0a9c8dd32bb1e7bc48b4836e70fdab85c3e2a (HEAD -> wip-jlindgre-lsl-timestamps, origin/wip-jlindgre-lsl-timestamps)

Thanks,
Jussi


- Issue History
Date Modified Username Field Change
2018-08-13 23:22 jfrey New Issue
2018-08-20 09:58 jtlindgr Assigned To => jtlindgr
2018-08-20 09:58 jtlindgr Status new => assigned
2018-08-20 10:08 dmedine Note Added: 0001263
2018-08-20 10:11 jtlindgr Note Added: 0001264
2018-08-20 10:11 jtlindgr Status assigned => acknowledged
2018-08-23 09:41 jtlindgr Note Added: 0001265
2018-08-23 09:41 jtlindgr Status acknowledged => feedback
2018-08-23 09:50 dmedine Note Added: 0001266
2018-08-23 22:21 jfrey Note Added: 0001267
2018-08-23 22:21 jfrey Status feedback => assigned
2018-08-23 22:21 jfrey Note Edited: 0001267 View Revisions
2018-08-23 22:37 jfrey Note Edited: 0001267 View Revisions
2018-08-24 16:31 dmedine Note Added: 0001273
2018-08-24 16:42 jtlindgr Note Added: 0001274
2018-08-24 17:28 jfrey Note Added: 0001275
2018-08-27 09:38 dmedine Note Added: 0001276
2018-08-27 10:32 jtlindgr Note Added: 0001277
2018-08-27 10:32 jtlindgr Note Edited: 0001277 View Revisions
2018-08-27 10:37 jtlindgr Note Edited: 0001277 View Revisions
2018-08-27 10:39 jtlindgr Note Edited: 0001277 View Revisions
2018-08-27 14:39 jtlindgr File Added: openvibe-lsl-timestamps-fix-candidate-20180827.zip
2018-08-27 14:43 jtlindgr Note Added: 0001278
2018-08-27 14:44 jtlindgr Note Edited: 0001278 View Revisions


Copyright © 2000 - 2010 MantisBT Group
Powered by Mantis Bugtracker