Page 1 of 1

Using OpenVibe & Unity 5 for a mini feedback game(TCPWriter)

Posted: Mon Nov 30, 2015 7:52 am
by pvb2k15
Hey guys,

I am new here, hence this is my first post :)

For a psychology seminar at my university I wanted to create a simple feedback game using OpenVibe and Unity.
I have a EEG-SMT device (2 Channels, 4 Electrodes) so the game probably will be very simple. I might want to use alpha waves to detect when someone closes his eyes.
If this is detected I want to react to it in my Unity game.
The basic thing I will need for that is of course a communication line between OpenVibe and Unity. So I tried out the TCPWriter, receiving stimuli in Unity is no problem as those are strings; however, I am not able to decode the raw signal send from the writer ...
I thought it might be possible to process the signal in OpenVibe and then somehow create a stimulus that I can send (e.g. using the SignChangeDetector).

In general I wanted to ask you for suggestions on how to realize this little project, but it also would be nice if someone could help me with the TCPWriter and decoding the raw signal.

Any help is appreciated ;)

Cheers,

Philipp

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Mon Nov 30, 2015 10:53 am
by nfoy
Hi,
The format for raw signal output from the TCP Writer Box is described in the box documentation: http://openvibe.inria.fr/documentation/ ... riter.html
You will also find a client example in openvibe/applications/examples/openvibe-to-tcpip/src/tcpip-client.cpp that shows how to read data received from the TCP Writer box.
For you project you can get inspiration from the scenario neurofeedback.xml for instance, located in dist/share/openvibe/scenarios/bci-examples/neurofeedback.
This scenario shows how to get the beta power from the brain signals.

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Mon Nov 30, 2015 5:24 pm
by pvb2k15
Hey,

thanks for the reply.

I've looked through those examples already and also read the description in the documentation. Maybe I was a bit unclear: I am not able to decode the raw signal data that is sent by the TCPWriter to Unity (I know it is described in the documentation, that it consists of a header of 32 bytes and so on) or in other words I am too stupid I guess. I don't know how I translate the byte data stream into the values of the signal. However, if I send a stimulus it is fine as it is a string. It would just be nice to e.g. display the signal in the Unity program.

Anyway I thought it might be best to process the EEG data in OpenVibe and e.g. use the power of beta like the in neurofeedback example and depending on the power send a stimulus to the game. But at the moment I don't see many possibilities to convert a signal into a stimulus except for the SignChange Box, maybe I am missing something.

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Tue May 10, 2016 12:41 pm
by elcinobacone
Hi to all,
we are a small research group working on a BCI based application at Politecnico di Milano University. We would like to connect the board device to Unity 3D platform for the development of a neurofeedback application. We have the same problem: have you found a solution for that?

Thanks in advance

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Wed May 11, 2016 7:38 am
by jtlindgren
Hi all,

you should be able to extract/adapt the code from the tcpip-client.cpp bundled with openvibe that reads from the TCP Writer. To convince yourself it is working correctly, you can use that small example to record data to a file. Use an understandable signal (e.g. time signal, sinusoid, ...) and then load the recorded file to a hex editor. Setting the display mode as 'double', you should see the signal correctly in the file. To do the same thing in the code, you just use the header to get the signal dimension and after that interpret your buffers as doubles.

You can also use LabStreamingLayer to get data out from OpenViBE.


Best,
Jussi

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Wed May 11, 2016 7:50 am
by pvb2k15
Hey,

For my small application I decided to use the OSC box to send alpha and beta power to unity. But if I remember correctly you can only send single values via that box, so depending on what data you want it might not be the beat choice. I also have some code for the TCP writer that at least reads thr header correctly , but the reading of the signal matrix has not been finished. I am currently on the mobile phone so I might post it later.

Cheers

Edit: The code I have here was not tested or finished, but it should give an Idea on how to deal with the TCPWriter.

Code: Select all

using UnityEngine;
using System.Collections;
using System;
using System.IO;
using System.Net.Sockets;
using System.Text;
using UnityEngine.UI;

// http://answers.unity3d.com/questions/15422/unity-project-and-3rd-party-apps.html
public class s_TCP : MonoBehaviour
{
    internal Boolean socketReady = false;
    TcpClient mySocket;
    NetworkStream theStream;
    StreamWriter theWriter;
    StreamReader theReader;
    String Host = "localhost";
    public Int32 Port = 5678;

    public bool testHeader = true;
    public bool testSignal = false;
    public bool isString = false;
    public int testSampleChannelSize;
    public int testSampleCount;
    public int testChannelCount;

    public double[,] lastMatrix;

    public float power;

    public RawOpenVibeSignal lastSignal;

    Rigidbody2D rb;
    public class RawOpenVibeSignal {

        public int channels;
        public int samples;

        public double[,] signalMatrix;
    }
    void Start()
    {
        setupSocket();
        
    }
    void FixedUpdate()
    {
       power = ((float)readSocket());
        
    }
    // **********************************************
    public void setupSocket()
    {
        try
        {
            mySocket = new TcpClient(Host, Port);
            theStream = mySocket.GetStream();
            theWriter = new StreamWriter(theStream);
            theReader = new StreamReader(theStream);
            socketReady = true;
        }
        catch (Exception e)
        {
            Debug.Log("Socket error: " + e);
        }
    }
    public void writeSocket(string theLine)
    {
        if (!socketReady)
            return;
        String foo = theLine + "\r\n";
        theWriter.Write(foo);
        theWriter.Flush();
    }
    public double readSocket()
    {
        if (!socketReady)
            return 0; // TODO
        if (theStream.DataAvailable)
        {



            // read header once
            if (testHeader)
            {
                readHeader();

            }

            if (testSignal)
                {
                // raw signal data
                // [nSamples x nChannels]
                // all channels for one sample are sent in a sequence, then all channels of the next sample

                // create a signal object to send it to another
                RawOpenVibeSignal newSignal = new RawOpenVibeSignal();
                newSignal.samples = testSampleCount;
                newSignal.channels = testChannelCount;

                double[,] newMatrix = new double[testSampleCount,testChannelCount];

       
                    byte[] buffer = new byte[testSampleChannelSize];

                    theStream.Read(buffer, 0, testSampleChannelSize);


                int row = 0;
                int col = 0;
                    for (int i = 0; i < testSampleCount * testChannelCount * (sizeof(double)); i = i + (sizeof(double) * testChannelCount))
                    { 
                        for (int j = 0; j < testChannelCount * sizeof(double); j = j + sizeof(double))
                        {

                            byte[] temp = new byte[8];

                            for(int k = 0; k < 8; k++)
                            {
                                temp[k] = buffer[i + j + k];
                            }

                        if (BitConverter.IsLittleEndian)
                        {
                           // Array.Reverse(temp);
                            double test = BitConverter.ToDouble(temp, 0);
                           
                            // TODO TEST THIS
                            //newMatrix[i / (8 * testChannelCount), j / 8] = test;
                            newMatrix[row, col] = test;
                        }
                        col++;

                        }
                    row++;
                    col = 0;
                    }

                newSignal.signalMatrix = newMatrix;
                lastSignal = newSignal;
                lastMatrix = newMatrix;

                

               displaySignalText();

                return newMatrix[0, 0];
            }
                else if (isString) {
                    Debug.Log(theReader.ReadLine());
                }

            }
            return 0;

    }

    private void readHeader()
    {
        // size of header is 8 * size of unit = 32 byte

        int variableSize = sizeof(UInt32);
        int variableCount = 8;

        int headerSize = variableCount * variableSize;

        byte[] buffer = new byte[headerSize];

        theStream.Read(buffer, 0, headerSize);

        // version number (in network byte order)
        // endianness of the stream (in network byte order, 0==unknown, 1==little, 2==big, 3==pdp)
        // sampling frequency of the signal, 
        //  number of channels, 
        // number of samples per chunk and 
        // three variables of padding


        UInt32 version, endiannes, frequency, channels, samples;

        byte[] v = new byte[4] { buffer[0], buffer[1], buffer[2], buffer[3] };
        byte[] e = new byte[4] { buffer[4], buffer[5], buffer[6], buffer[7] };
        byte[] f = new byte[4] { buffer[8], buffer[9], buffer[10], buffer[11] };
        byte[] c = new byte[4] { buffer[12], buffer[13], buffer[14], buffer[15] };
        byte[] s = new byte[4] { buffer[16], buffer[17], buffer[18], buffer[19] };
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(e);
            Array.Reverse(v);
            version = BitConverter.ToUInt32(v, 0);
            endiannes = BitConverter.ToUInt32(e, 0);
            frequency = BitConverter.ToUInt32(f, 0);
            channels = BitConverter.ToUInt32(c, 0);
            samples = BitConverter.ToUInt32(s, 0);
        }
        else
        {

            version = 999;
            endiannes = 0;
            frequency = 0;
            channels = 0;
            samples = 0;
        }

       // Debug.Log("Version: " + version + "\n" + "Endiannes: " + endiannes + "\n" + "sampling frequency of the signal: " + frequency + "\n" + "number of channels: " + channels + "\n" + "number of samples per chunk: " + samples + "\n");

        testHeader = false;
        testSampleCount = buffer[16];
        testChannelCount = buffer[12];
        testSampleChannelSize = buffer[12] * buffer[16] * sizeof(double);

        testSignal = true;

        rb = GetComponent<Rigidbody2D>();
    }

    public void closeSocket()
    {
        if (!socketReady)
            return;
        theWriter.Close();
        theReader.Close();
        mySocket.Close();
        socketReady = false;
    }

    void displaySignalText() {
        RawOpenVibeSignal s = lastSignal;

        StringBuilder sb = new StringBuilder();

        for (int i = 0; i < lastSignal.samples; i++) {
            sb.AppendLine("Sample" + i + "\t");
            for (int j = 0; j < lastSignal.channels; j++)
            {
               try
                {
                    sb.AppendLine("Channel" + j).Append(lastSignal.signalMatrix[i, j]);
                }
                catch {
                  Debug.Log("i:" + i + "-j:" + j);
                }
           }
        }

       // Debug.Log(sb.ToString());
        
    }
    void OnApplicationQuit()
    {
        closeSocket();
    }
} // end class s_TCP

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Tue May 17, 2016 7:53 am
by elcinobacone
Just two questions :
so do you reach to realize your realtime application?
And how do you make acquisition data from straits by OpenVibe?

Thanks a lot

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Sat Sep 02, 2017 7:36 pm
by kiyarash
I want to add that the box mentioned earlier is stated as unstable http://openvibe.inria.fr/documentation/ ... riter.html
WARNING : this box has been marked as UNSTABLE by the developer. It means that its implementation may be incomplete or that the box can only work under well known conditions. It may possibly crash or cause data loss. Use this box at your own risk, you've been warned.
so is using it really a viable solution?

Re: Using OpenVibe & Unity 5 for a mini feedback game(TCPWri

Posted: Fri Sep 08, 2017 11:42 am
by jtlindgren
That categorization is not a problem by itself. :)

http://openvibe.inria.fr/faq/#unstable