Tutorial – Level 2b – How to do repeatable experiments with OpenViBE

  • NB: document written for OpenViBE 0.18.0 (Dec 2014).

Often, when running experiments or demos, it’s important that the conditions stay identical from one experiment to another. If this is not carefully controlled, the results could change because of the setup changes, or your experiment even might not work at all. In situations where OpenViBE is used for several different experiments — or perhaps even developed on the same computer — it’s important to control that modifications made for one experiment do not affect the others.

Here are the basic steps that can be followed to attain this goal.

Hardware

Make sure that the amplifier and the acquisition circumstances do not change. Make sure electrode amount and positions are the same. Do not attempt to perform a demo with a hardware-software combination that you haven’t already physically tested. For example, amplifiers with the same name may still have different versions of the firmware, or a third-party computer may have a different operating system driver for the amp compared to the one you previously used — all these can cause the acquisition to produce different signal or have unforeseen problems.

Software

To ensure that experiments do not affect one another, do the following

  1. Create a folder for the experiment in question, e.g. “experiment/”. Everything related to the experiment will be under this folder. If you’ll notice your setting writing or reading any configuration files elsewhere, either there is a bug in these instructions or you haven’t followed them fully.
  2. Make sure OpenViBE version doesn’t change across experiments. Copy the compiled OpenViBE tree to be in “experiment/dist/”, and make sure you always launch the openvibe applications from this location. This ensures that the binaries are as expected.
  3. Acquisition server and Designer accept a command line parameter “–config ” to specify their main configuration file. Copy the main config file from share/openvibe/kernel/openvibe.conf to be under “experiment/config/”. Always launch these two applications in a way that they take their configurations are taken from the experiment location. You can make short shell scripts for this purpose and keep the scripts under “experiment/”. Modify the config files in question so that they do not include more configuration files from global locations (for example, replace anything having ${Path_UserData} token, esp. CustomConfigurationPrefix* that control where application specific configurations are loaded from) but that they read them as well from under your experiment folder. You may consider also redirecting all logging to be under “experiment/logs”, with each log name time stamped (see item 6).
  4. Create a folder for all the scenarios of the experiment, e.g. “experiment/scenarios”. Put all the scenario .xml files there. Remember to open them from there as well.
  5. In scenarios, always store all data files and output relative to the scenario location of the last step. You can do this by using the token ${Player_ScenarioDirectory} as the first part of the path. It will point to the path where the scenario is opened from, in Designer.
  6. Scenario steps often produce configuration files for the next steps (for example as done by LUA scripts in the SSVEP scenarios bundled with OpenViBE). When creating such a configuration file, it’s a good idea to do it twice: e.g. save both

    ${Player_ScenarioDirectory}/config.txt
    ${Player_ScenarioDirectory}/config-log-$core{date}-$core{time}.txt

    This way, the next steps should read the latest configuration with the known fixed name, but you will get a time stamped log as well. You can do the same thing even for data files, for example by having two boxes writing the stream out to two different files, where the other has a timestamped name.
  7. [ Designer also has a command line switch “–random-seed ” which can be used to have the pseudo random generator in the same state on each launch. Note that some components like SVM might use their internal randomness and not be affected by this switch. But in principle it should help you to get identical cross validation results across two Designer runs (given that the steps you do in Designer are identical in terms of calls to the random generator). ]

Most of the steps above you need to do with just once when you create your scenario. The rest is just remembering to follow the working conventions listed above. If you always use relative paths to your experiment directory, in principle you should be able to zip the folder and have a self contained archive for transferring the experiment to another computer (but remember to control for hardware differences).

This entry was posted in Documentation, Tutorials, User documentation. Bookmark the permalink.