Introduction to algorithms and boxes

Extending OpenViBE

There are two ways OpenViBE can be extended. One is to write new algorithms, that are only used by programmers in order to perform a specific operation. The second is to create new boxes (also named Box Algorithms) that can be used by authors in scenarios.

Depending on the task at hand, one might have to implement new algorithms and/or boxes. While it is possible to write a box which doesn’t make use of any algorithm (in the sense of OpenViBE, meaning all signal processing code is written directly in the box), it is usually desirable to encapsulate signal processing operations in algorithms. The gain is not necessarily obvious at first, but it becomes evident in the long term, allowing box developers to reuse existing algorithms and build new boxes faster. In any case, it’s up to the programmer to determine what operations are generic enough to justify their encapsulation in an algorithm.

Prior knowledge is mandatory if you want to develop a plugin serenely. Thus we will firstly cover the mechanisms of streams, input, outputs, triggers and how to connect everything properly.

Introduction to plugin development

OpenViBE streams

Boxes and algorithms have inputs and outputs, each with a given type. Each type corresponds to a specific stream structure which naturally leads to connection rules between inputs and outputs. For example, you cannot connect a stimulation output to a signal input.

The streams are hierarchically ordered as follows:

As you can see every stream is an EBML Stream and each subtype has more specific data or constraints. We strongly advise you to take a look at the stream documentation page, which precisely describe each stream structure and what they contain.

Every stream starts with a Header, which contains the structural information (e.g. number of channels in a signal stream). Afterwards the Buffers are sent, as many as needed (e.g. the actual sample buffers in a signal stream). To properly close the stream an End is sent. This last part is just a marker, and is not used by most of the boxes as they don’t need any particular treatment when the End comes.

Encoding and Decoding streams

We provide codec algorithms that handle the decoding and encoding process for every type of stream. For example, the Signal Decoder Algorithm will take the EBML signal stream as input and produce the matrices of samples and the sampling frequency for further processing inside the box.

These EBML decoders and encoders should be used only for a dedicated input or output. 1 decoder per input, 1 encoder per output. Nothing will stop you from using the same decoder for inputs with the same type, however it will result in highly unstable behavior as you cannot trust the codec inner state if the 2 streams differ somehow (corruption of a chunk, process half-done, etc.)

Decode and encode data are the most common actions made by OpenViBE boxes. In order to simplify these process, we designed a dedicated developer tool : the Codec Toolkit, as set of templated classes. This toolkit was introduced with OpenViBE 0.10.0 and every boxes developed afterward make use of it. As you may look into older boxes, you would see that they are relying on “raw” codec algorithms, more verbose and complicated at first sight. We strongly advise you to use the Codec Toolkit as it will simplify your code significantly.

Timing

OpenViBE uses simulated time in order to always, no matter what, treat all the samples coming from a given source (Acquisition Server or data file). Each stream can be seen as a pipe where blocks of data, called chunks, transit. Each chunk is precisely dated with a start time and an end time.

Timing in OpenViBE is crucial, and all boxes must generate output chunks with an accurate timing. This task can be easy, for example if the box does a simple signal processing task (e.g. take signal chunks, multiply all samples by 2 and output the resulting signal – the input and output chunks would have then the very same timing). However some signal processing tasks may require a computation of accurate timing (e.g for the epoching process that can produce overlapping block of data).

Please note that you should never go back in time when processing data chunks. Once a chunk starting at time t has been received, all following chunks should have a start time greater or equal to t.

When implementing your signal processing method, note that you can rely on the chunk timing to perform the task precisely, and make sure the chunks you generate can be used by the following boxes normally.

Algorithm internals

The algorithm is a very generic, low level component which can easily communicate with other algorithms. An algorithm can be called by other algorithms or by a Box – the high level component encapsulating a whole process.

An Algorithm implementation is divided in 2 parts :

An algorithm processes one or more inputs and returns one or more outputs. In the implemented algorithm object, each input and output has a corresponding data structure. In order to easily manage these structures, we provide the template handler OpenViBE::Kernel::TParameterHandler. Therefore the algorithm has one TParameterHandler per input or output parameter. The types of parameters you can specify are described in the OpenViBE::Kernel::TParameterHandler documentation (boolean, int64, StimulationSet, etc.). For example the following handler will manage an input Matrix:

 OpenViBE::Kernel::TParameterHandler < OpenViBE::IMatrix* > ip_pMatrix;

This input matrix handler has to be initialized to be connected to the concrete OpenViBE::Kernel::IParameter.

 ip_pMatrix.initialize(inputParameterMatrixPointer);

The algorithm communicate with its surroundings using input and output triggers (for example input trigger “Do the process“, or output trigger “Process successful“). The kernel (OpenViBE::Kernel::IAlgorithmContext) provides 2 functions related to triggers, to be used on an algorithm object:

  • activateOutputTrigger(OpenViBE::CIdentifier& rOutputTriggerIdentifier, OpenViBE::boolean bTriggerState) sets the state of an output trigger of the algorithm.
  • isInputTriggerActive(OpenViBE::CIdentifier& rInputTriggerIdentifier) checks the current state of an input trigger.

The figure below illustrates the algorithm concept with a processing unit that use an input matrix to output a second matrix. The algorithm is controlled by 2 input triggers that ask the algorithm to initialize or start some process. When the process is done, the algorithm rises a dedicated trigger.

algo-diagram.png
An algorithm, its input, output and triggers

Box internals

A box is an abstract view of a single processing chain, that may include several algorithms linked to perform a precise task. The Box manages all the data structures. The box has its own inputs, outputs and settings described in the box static context OpenViBE::Kernel::IBox. Inputs and outputs all receive or send encoded data in EBML structures. As previously stated, this data is divided in timed blocks called chunks. Chunk management is done through the box’s dynamic context OpenViBE::Kernel::IBoxIO. Once a chunk is received, the box can put it on the input(s) of algorithm(s). Most of the time the first algorithm used will be a EBML decoder, which extracts the data from the chunk. The last algorithm in the chain (most of the time an EBML encoder) rises its output trigger signaling the end of process, meaning that the output chunk has been produced. The box can then mark the output chunk as ready to be sent !

The box declares each used algorithm along with its inputs and outputs. The Algorithm itself is represented by an OpenViBE::Kernel::IAlgorithmProxy object. The inputs and outputs are again contained in OpenViBE::Kernel::TParameterHandler handlers. Here is an example of such declarations in the box definition :

 OpenViBE::Kernel::IAlgorithmProxy* m_pSignalProcessingAlgorithm;
 OpenViBE::Kernel::TParameterHandler < OpenViBE::IMatrix* > ip_pSignalProcessingAlgorithmMatrix;
 OpenViBE::Kernel::TParameterHandler < OpenViBE::IMatrix* > op_pSignalProcessingAlgorithmMatrix;

The IAlgorithmProxy class is a user interface to an IAlgorithm instanciated object. Its purpose is to automatically handle input / output trigger activation and to help in calling processing methods. During the initialization phase, the box asks the OpenViBE::Kernel::IAlgorithmManager instance of the kernel to create an instance of the Algorithm. It also initializes the inputs and outputs handlers, giving them the actual parameters of the created algorithm. The code below illustrates this process:

 CIdentifier l_idAlgorithmIdentifier = this->getAlgorithmManager()
                                                .createAlgorithm(OVP_ClassId_Algorithm_SignalProcessingAlgorithm);
 m_pSignalProcessingAlgorithm=&this->getAlgorithmManager().getAlgorithm(l_idAlgorithmIdentifier);
 m_pSignalProcessingAlgorithm->initialize();

 ip_pSignalProcessingAlgorithmMatrix.initialize(
          m_pSignalProcessingAlgorithm->getInputParameter(
                   OVP_Algorithm_SignalProcessingAlgorithm_InputParameterId_Matrix));
 op_pSignalProcessingAlgorithmMatrix.initialize(
          m_pSignalProcessingAlgorithm->getOutputParameter(
                   OVP_Algorithm_SignalProcessingAlgorithm_OutputParameterId_Matrix));

Finally, the algorithm’s inputs and outputs are linked together with the setReferenceTarget function of the TParameterHandler handler. In the example below, the input ip_pSignalProcessingAlgorithmMatrix of the signal processing algorithm will be connected to the output op_pDecodedMatrix coming from a stream decoder algorithm.

 ip_pSignalProcessingAlgorithmMatrix.setReferenceTarget(op_pDecodedMatrix);

Note that it is the input that sets its output reference target, and not the opposite. The reason is simple : an output can be sent to several inputs, but an input can have only one source output.

Once reference targets are set, inputs and outputs point to the very same data structure. Modifying one will modify the other.

Designing a plugin

Let’s take the example of a signal processing box with one input (a signal stream) and one output (the processed signal), relying on an algorithm for the processing part.

The first Algorithm decodes these EBML memory buffers coming on the Signal input. These chunks are transformed into a convenient data container called streamed matrix, much easier to manipulate than EBML stream. The Signal Decoder also outputs the sampling frequency of the incoming signal.

The second algorithm is the signal processing algorithm performing the task, e.g. computing the mean of incoming signal.

The last algorithm is a Signal Encoder, that constructs new EBML chunks using a sampling frequency value and a streamed matrix containing the new signal samples.

The figure below illustrates the design we will achieve with this example, with the corresponding kernel calls used in the implementation. All variable prefixed with ip_ refer to inputs, op_ refer to outputs.

box-diagram.png
Design of a signal processing plugin

Next !

Now that you have (I hope!) a clearer view of the concepts within OpenViBE, let’s move one to the next step : implementing a simple signal processing box using our developer tools and good practices at hand !

As the algorithm mechanics may sound complex at first sight, this series of tutorials will start with a simple example of signal processing box, where the entire task is done directly in the box. We will then cover the algorithm implementation and how to use it in a box.

Go to Tutorial 1 !

This entry was posted in Box plugins and tagged , . Bookmark the permalink.