Extensions: TCP Tagging (Software Tagging)

  • NB: Document updated for OpenViBE 2.0.1 (doc updated 14.Feb.2018).

Background

This text describes TCP Tagging, a software tagging mechanism for time-accurate placement of event markers (called stimulations in OpenViBE) in the EEG signal recordings. In OpenViBE, this mechanism is used by many scenarios to get the events properly aligned with the EEG data. Typically the boxes that do the rendering will use TCP Tagging to send the event identifier (stimulation) to the Acquisition Server right after rendering the event. This is called software tagging because it requires no support from the hardware.

Warning: If you are doing any EEG analysis where you analyze EEG properties related to events in time (as represented by stimulations or markers in time), it is NOT correct to merge these stimulations to the EEG directly in Designer from any non-rendering source such as LUA Stimulator or P300 Stimulator. If you do that, the markers will not be correctly aligned to the EEG. Instead, the renderer of the events must forward the stimulations to the Acquisition Server over TCP Tagging when there is certainty when the rendering happened. In this practice, the stimulations should be recorded from the Acquisition Client box together with the signal. This document gives the details. The main exception to this practice are the results from the classifier processor box. If these stimulations are used to control applications and not in event-related EEG analysis, they can be sent to the app directly.

Accurate tagging is especially important in order to process Event-Related Potentials (ERP) such as P300 that are very precisely event time-locked. ERP is a neurophysiological response to a specific stimulus. In order to obtain signal segment corresponding to the ERP, one needs to have an accurately aligned event marker in the signal, corresponding to the stimulus that elicited the ERP. The signal processing relying on this segmentation can be very sensitive to acquisition drift or time jitter: the delay between the exact moment when the stimulus is presented to the user and the moment corresponding to the event marker in the signal acquisition. A constant jitter (drift) generally has little impact on ERP processing, provided that the jitter is either measurable or not too large. A varying jitter on the other hand may have a huge impact on classification score.

This chart represents the cross-validation score of XDAWN+LDA classifier with an artificial varying jitter. The simulated jitter was uniformly generated over an increasing interval (x-axis). This chart represents the cross-validation score of XDAWN+LDA classifier with an artificial varying jitter. The simulated jitter was uniformly generated over an increasing interval (x-axis).

Some acquisition devices allow inserting event markers directly to the signal during acquisition with an approach called hardware tagging. For example, some TMSI acquisition devices have a dedicated parallel port that can be connected to the computer running the display. Other amplifiers, like gtec gusbamp, have other types of digital inputs that can be used for this purpose. The hardware method is generally very accurate but it is not supported by all devices. What is also a challenge is that there is no standardized way to send the events from the software side to different amplifiers, making it difficult to support hardware tagging in OpenViBE.

Therefore a software solution to this problem is often needed. However this is not an easy task because it requires to have a precise measure of time for the signal, which we don’t always have with the hardware at hand.

TCP Tagging has been designed to achieve a good accuracy for the correct placement of markers, while keeping the design simple. Starting from OpenViBE 1.2.0, TCP Tagging is intended to replace the alternative External Stimulations approach, which has been deprecated.

Design and operation

Basically, TCP Tagging is a plugin in Acquisition Server that opens a TCP/IP port. The stimulator rendering the events causing the ERP are supposed to open a connection to this port, and send the stimulation (event marker) there, possibly immediately after rendering of the event. The TCP Tagging plugin running on the server will then attach the received marker to the EEG stream as closely as possible. Subsequent processing in Designer can then use these markers to segment the signal.

Note that the port the client must connect to is 15361 by default. However you can set up a different port with the acquisition server, by clicking Preferences and change the TCP Tagging port setting.

preferences_panel-highlighted

Once the port is open, you can send triplets of uint64 numbers (24 bytes in total) to the AS while the TCP Tagging connection is open. The interpretation of these three numbers depends on the OpenViBE version as follows.

OV 2.0.1 and later tag format to send to the server:

[uint64 flags ; uint64 stimulation_identifier ; uint64 timestamp]

the timestamp is expected to either be 0 or the time since the boot of the computer in OpenViBE 32:32 fixed point time. You can get this time from the System::Time::zgetTimeRaw() function in OpenViBE. The flags can be ORred (where meaningful). They are

1 == The time is in OV fixed point time (required, or will use timestamp=0)
2 == The client should bake the timestamp (e.g. StimulusSender class)
4 == The server should bake the timestamp on receiving

If the timestamp is 0, the stimulus is interpreted to be immediate and equals setting flag 4: the stamp is put on the server side. The boxes shipped with OV2.0.1 tag with the flags 1||2: immediately after rendering, the client side raw time is used to timestamp the tag.

OV 2.0.0 and earlier tag format to send to the server:

[uint64 padding ; uint64 stimulation_identifier ; uint64 timestamp]

If the timestamp is 0, the stimulus is interpreted to be immediate. The padding is used to keep compatibility with the External Stimulations format. In these earlier versions of OpenViBE, any non-zero timestamp was expected to be ‘Posix time’, i.e. milliseconds since epoch, computed from the result of the C call ftime().

Limitations

Careful: there does not appear to be big endian / little endian byte order conversions in the TCP Tagging at the moment, i.e. AS expects the uint64s to be in the same byte order as the system the server is running on.

The stamps sent in ‘posix time’ as earlier are no longer supported in OV 2.0.1. The reason is that we do not apparently have a steady clock running in ‘wall time’ in C++11. Note that this presently restricts the stimulus sender and the Acquisition Server to reside on the same computer for two reasons: openvibe doesn’t guarantee two computers to have synchronized clocks, and the protocol currently has no mechanism to estimate the amount of difference. See section ‘Future Developments’ in the end.

To be clear, TCP Tagging is not currently taking sending latency into account except when these conditions apply: both sender and server are application is on the same computer, and the sender stamps the tag with OpenViBE’s zgetTimeRaw() call. The boxes using TCP Tagging shipped with OpenViBE currently do this (since 2.0.1).

N.b. : In Designer, in order to be able to configure processing related to the stimulation identifiers, all the numeric stimulus identifiers should be present in this list. Other stimuli may not be selectable in the Designer GUI. If you wish to work with other numbers, you need to modify the OpenViBE source code and register them.

How to send tags from an external application

Basically all the clients just open a TCP socket to the server and send stimulations to the socket. If your Acquisition Server and the application are on different computers, you need to replace ‘localhost’ with the host computers address and use the port you configured in AS. In case of problems, remember to check that a firewall is not blocking the connection. Note however the limitations (see above).

Here are TCP Tagging client examples in different programming languages. They are currently pre-2.0.1 but the modifications should be straightforward. If you cannot get the same clock as in zgetTimeRaw() in your language, please stamp with 0.

Python

An example of a python client able to send tags to the acquistion server is provided in contrib/plugins/server-extensions/tcp-tagging/client-example.

In order to use it you need first to start the acquisition server, connect with a driver and play acquisition. To send a tag to the acquisition server, simply run:

$ python example-client.py

Explore the source code of example-client.py to see how to implement a client to send a tag to the acquisition server. Note that in OV v1.2.0 – v1.2.1 there is a bug in the python example which requires you to use python 3. To run it with python 2, replace ‘bytes()’ function in the python code with ‘bytearray()’.

Java

To send stimulations from Java, you can try this simple class.

C++

Finally, in modules/tcptagging/src/CStimulusSender.cpp of the OpenViBE source tree you can find a simple C++ class that implements a TCP Tagging client. The class is used by e.g. the P300 Stimulator and Graz Visualization boxes bundled with OpenViBE.

Implementing custom hardware tagging

To be really precise, you might want to send the markers directly to your EEG amplifier. There is currently no standard to do this as far as we know. As the OpenViBE boxes that support TCP Tagging send the software stimulations over a very simple interface of the IStimulusSender.h, it is relatively straightforward to change them to use hardware tagging with your specific hardware. Simply create another implementation of the CStimulusSender class and change TCPTagging::createStimulusSender() to return pointer to your custom implementation. Then write the low-level device-specific messaging logic in the implementation. Recompiling OpenViBE, the boxes now should use your hardware tagging instead.

Best practices and historical notes

Before OpenViBE 1.2.0, many scenarios used to directly create the event timeline (stimulations with timestamps of requested ‘ideal’ times something should occur at) in the Designer and then merge the timeline to the recorded signal inside the Designer. This approach is not very accurate for various reasons (Designer running simulated time, rendering not necessarily happening exactly when requested, and delays between the execution of the components). The old approach tends to work for BCI schemes such as Motor Imagery and SSVEP where the timing precision may not be as critical as it is in ERP based paradigms. However, we recommend that all new BCI designs should be designed with the TCP Tagging approach. You first generate the event timeline (e.g. with lua), then send it to the stimulating component. The stimulator should use TCP Tagging to send each event to the Acquisition Server using TCP Tagging immediately after its effect has been presented to the user.

It is recommended you do not try to merge stimulations and EEG in the Designer, if the stimulations have been stamped with ‘ideal’ time requests (e.g. all lua scripts, P300 Stimulator, etc bundled with OpenViBE create stimulations with such ideal stamps). The ideal stamp does not guarantee that the system actually is able to deliver the event precisely at that time. That is why re-stamping after rendering is preferable.

Future developments

In order to allow good stimulation synchronization to the EEG across multiple computers, it may be necessary to switch to a protocol resembling ntpdate, or use approaches familiar from LabStreamingLayer (LSL) or TOBI Interface standard D (TiD). If you have interest in this, let us know.

This entry was posted in Documentation, Acquisition Server documentation and tagged , , , . Bookmark the permalink.