P300 Speller with random letter flash

Making & changing box plugins and external apps
Post Reply
slorenz
Posts: 7
Joined: Tue Jun 22, 2010 8:38 pm

P300 Speller with random letter flash

Post by slorenz »

Hello there. Our lab has tested out the P300 Speller example and it's up and running without a hitch!

We would like to make a new paradigm, however, that involves flashing only INDIVIDUAL letters rather than an entire row or column. I have already dug into the source code (ovpCBoxAlgorithmP300SpellerVisualisation.cpp) but am not quite sure what part of process() needs editing to only flash one letter randomly.

Cheers,
Sean

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: P300 Speller with random letter flash

Post by yrenard »

Dear slorenz,

nice to see you there... thank you again for your interest in OpenViBE and welcome on this board.

I'm glad that you had the P300 speller working without a hitch ! It's good to know when it simply works ;)

About your idea of having single items flashing, have you looked into the P300 Magic Card scenarios ? The do not use the xDAWN spatial filter but I guess you can easily adapt the xDAWN P300 Speller to use the P300 Magic Card presentation boxes. That should fit your request, without changing any of the code.

Can you tell us a little bit more about what you plan to do with OpenViBE and what acquisition device you use ?

Hope this helps,
Yann

slorenz
Posts: 7
Joined: Tue Jun 22, 2010 8:38 pm

Re: P300 Speller with random letter flash

Post by slorenz »

Hi Yann,

First off, it was great meeting you in our speech brainstorming group at the BCI Meeting last month.

I did indeed look into the Magic Card BCI and will probably just adapt that scenario instead, however, I am still interested in knowing what code tells the P300 Speller to select ALL the elements in one column or row. I've had a hard time finding where that gets called.

As for our OpenVibe plans, the goal is to (at least for now) do basic speech phoneme categorization/selection using an EEG BCI. We are currently using g.Tec's GAMMAcap with 32 electrodes. Also, we were using g.Tec's Matlab Simulink BCI software, but are now in the process of transitioning to OpenVibe.

Cheers,
Sean

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: P300 Speller with random letter flash

Post by yrenard »

Dear slorenz,

indeed, the P300 Speller Visualisation box' code could be tricky to read as it works with successive uses of "callbacks".

In the process function, you will find a number of calls to the _cache_for_each_if_ function. This function goes through each cell of the matrix and calls an "if" callback function or an "else" callback function depending on a condition. The "if" callback could e.g. be :
  • - _cache_change_foreground_cb_: to change the foreground of the cell
    - _cache_change_background_cb_: to change the background of the cell
    - _cache_change_null_cb_: to do nothing
    - ...
The condition is computed for each cell depending on the first two parameters of the _cache_for_each_if_ call which are respectively the selected row (or -1 if none) and column (or -1 if none).

I hope this helps,
Keep us posted of you results and I hope to see your g.USBamp driver modification soon in our official release !
Yann

slorenz
Posts: 7
Joined: Tue Jun 22, 2010 8:38 pm

Re: P300 Speller with random letter flash

Post by slorenz »

Hi Yann,

Thanks for the P300 Speller Visualization tip. I am now trying to edit the Stimulator so that it outputs a row AND a column. For example, instead of l_iRow=3 and l_iColumn=-1, it will now output something like l_iRow=3 and l_iColumn=2.

The problem I'm having is storing these valuse to pass on to the Visualization box. I assume this happens in the Stimulator file here:

Code: Select all

case State_Flash:
	l_oStimulationSet.appendStimulation(l_bRow?m_ui64RowStimulationBase+l_iRow:m_ui64ColumnStimulationBase+l_iColumn, l_ui64CurrentTime, 0);
which I have changed to this:

Code: Select all

case State_Flash:
	l_oStimulationSet.appendStimulation(m_ui64RowStimulationBase+l_iRow,l_ui64CurrentTime,0); // NEW
	l_oStimulationSet.appendStimulation(m_ui64ColumnStimulationBase+l_iColumn,l_ui64CurrentTime,0); // NEW
This does not work! Is it because only one value can be passed here?

Cheers,
Sean

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: P300 Speller with random letter flash

Post by yrenard »

Dear slorenz,

indeed, the P300 Speller Visualisation box is, by design, not ready to do that. As you can see at line 628-634, if a line and a column are flashed at the same time, then only the intersection is changed. This is used to display the targets and the results.

So if you want to be able to flash a complete line and a complete column at the same time, you will have to update the visualisation box as well.

I hope this helps,
Best regards,
Yann

slorenz
Posts: 7
Joined: Tue Jun 22, 2010 8:38 pm

Re: P300 Speller with random letter flash

Post by slorenz »

Hi Yann,

Sorry, I should clarify. I only want ONE letter in a certain row or column to flash. Right now it flashes an ENTIRE row or column, which I want to change in order to evoke a more robust P300 signal. This is why I have been looking to pass 2 stimuli in the Stimulator.

Sean

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: P300 Speller with random letter flash

Post by yrenard »

Dear slorenz,

I'm sorry I did not understand your explanations correcly.
Then in this case, you should modify the visualisation box in the sequence stream processing (lines 249-314), leaving the management of the stimulation identifier in the loop and putting the actual processing of those stimulations after the loop, such as :

Code: Select all

IStimulationSet* l_pStimulationSet=op_pSequenceStimulationSet;
for(j=0; j<l_pStimulationSet->getStimulationCount(); j++)
{
	uint64 l_ui64StimulationIdentifier=l_pStimulationSet->getStimulationIdentifier(j);
	boolean l_bFlash=false;
	int l_iRow=-1;
	int l_iColumn=-1;
	if(l_ui64StimulationIdentifier >= m_ui64RowStimulationBase && l_ui64StimulationIdentifier < m_ui64RowStimulationBase+m_ui64RowCount)
	{
		// Blah blah
	}
}

if(l_bFlash)
{
	this->_cache_for_each_if_(
		l_iRow,
		l_iColumn,
		&CBoxAlgorithmP300SpellerVisualisation::_cache_change_background_cb_,
		&CBoxAlgorithmP300SpellerVisualisation::_cache_change_background_cb_,
		&m_oFlashBackgroundColor,
		&m_oNoFlashBackgroundColor);
	// Blah blah
}
This way, you should have what you expect.
Also, take care that the line and column stimulations have the same date so they get into the same stimulation set.

I hope this helps,
Yann

slorenz
Posts: 7
Joined: Tue Jun 22, 2010 8:38 pm

Re: P300 Speller with random letter flash

Post by slorenz »

Ah! Thanks, Yann. So it ALMOST works. The final problem is with placement of the l_bFlash=false statement. I kept it inside the for loop and no flashes got through, so I put it outside the for loop which shows each stim in sequence but no longer gives a flash! The stim stays lit until the next stim lights up (instead of just being a quick flash). Ideas?? Here is the code:

Code: Select all

		if(m_pSequenceStimulationDecoder->isOutputTriggerActive(OVP_GD_Algorithm_StimulationStreamDecoder_OutputTriggerId_ReceivedBuffer))
		{
			// Grab the Sequence Stim Set from the stimulator
			IStimulationSet* l_pStimulationSet=op_pSequenceStimulationSet;

			static boolean l_bFlash=false;
			static int l_iRow=-1;
			static int l_iColumn=-1;

			// The StimCount equals 1 (telling the segment to stop) or 3 (sending a row, column, and stim start).
			for(j=0; j<l_pStimulationSet->getStimulationCount(); j++)
			{
				uint64 l_ui64StimulationIdentifier=l_pStimulationSet->getStimulationIdentifier(j);
				
				// ROWS -- If the StimID is >= than the RowStimBase AND the StimID is less than the RowStimBase+TotalRows (i.e., 7) so: RANGE(1,6).
				if(l_ui64StimulationIdentifier >= m_ui64RowStimulationBase && l_ui64StimulationIdentifier < m_ui64RowStimulationBase+m_ui64RowCount)
				{
					l_iRow=l_ui64StimulationIdentifier-m_ui64RowStimulationBase;
					l_bFlash=true;

					if(l_iRow==m_iLastTargetRow)
					{
						l_oFlaggingStimulationSet.appendStimulation(OVTK_StimulationId_Target, l_pStimulationSet->getStimulationDate(j), 0);
					}
					else
					{
						l_oFlaggingStimulationSet.appendStimulation(OVTK_StimulationId_NonTarget, l_pStimulationSet->getStimulationDate(j), 0);
					}
				}
				
				// COLUMNS -- If the StimID is >= than the ColumnStimBase AND the StimID is less than the ColumnStimBase+TotalColumns (i.e., 13) so: RANGE(7,12).
				if(l_ui64StimulationIdentifier >= m_ui64ColumnStimulationBase && l_ui64StimulationIdentifier < m_ui64ColumnStimulationBase+m_ui64ColumnCount)
				{
					l_iColumn=l_ui64StimulationIdentifier-m_ui64ColumnStimulationBase;
					l_bFlash=true;
					
					if(l_iColumn==m_iLastTargetColumn)
					{
						l_oFlaggingStimulationSet.appendStimulation(OVTK_StimulationId_Target, l_pStimulationSet->getStimulationDate(j), 0);
					}
					else
					{
						l_oFlaggingStimulationSet.appendStimulation(OVTK_StimulationId_NonTarget, l_pStimulationSet->getStimulationDate(j), 0);
					}
				}
				// STOP -- All the stims are done, so reset the the grid.
				if(l_ui64StimulationIdentifier == OVTK_StimulationId_VisualStimulationStop)
				{
					this->getLogManager() << LogLevel_Debug << "Received OVTK_StimulationId_VisualStimulationStop - resets grid\n";
					this->_cache_for_each_(&CBoxAlgorithmP300SpellerVisualisation::_cache_change_background_cb_, &m_oNoFlashBackgroundColor);
					this->_cache_for_each_(&CBoxAlgorithmP300SpellerVisualisation::_cache_change_foreground_cb_, &m_oNoFlashForegroundColor);
					this->_cache_for_each_(&CBoxAlgorithmP300SpellerVisualisation::_cache_change_font_cb_, m_pNoFlashFontDescription);
				}
			}

			// FLASH -- All systems are go to flash a stim! Change the stim element's properties as necessary.
			if(l_bFlash)
			{
				// Change the BACKGROUND color
				this->_cache_for_each_if_(
					l_iRow,
					l_iColumn,
					&CBoxAlgorithmP300SpellerVisualisation::_cache_change_background_cb_,
					&CBoxAlgorithmP300SpellerVisualisation::_cache_change_background_cb_,
					&m_oFlashBackgroundColor,
					&m_oNoFlashBackgroundColor);
				// Change the FOREGROUND color
				this->_cache_for_each_if_(
					l_iRow,
					l_iColumn,
					&CBoxAlgorithmP300SpellerVisualisation::_cache_change_foreground_cb_,
					&CBoxAlgorithmP300SpellerVisualisation::_cache_change_foreground_cb_,
					&m_oFlashForegroundColor,
					&m_oNoFlashForegroundColor);
				// Change the FONT for the flash
				this->_cache_for_each_if_(
					l_iRow,
					l_iColumn,
					&CBoxAlgorithmP300SpellerVisualisation::_cache_change_font_cb_,
					&CBoxAlgorithmP300SpellerVisualisation::_cache_change_font_cb_,
					m_pFlashFontDescription,
					m_pNoFlashFontDescription);
			}

			m_pTargetFlaggingStimulationEncoder->process(OVP_GD_Algorithm_StimulationStreamEncoder_InputTriggerId_EncodeBuffer);
		}

yrenard
Site Admin
Posts: 645
Joined: Fri Sep 01, 2006 3:39 pm
Contact:

Re: P300 Speller with random letter flash

Post by yrenard »

Dear slorenz,

remove the static keyword before the declaration of the l_bFlash, l_iRow and l_iColumn variables. It will most likely help if not solve the problem.
You may be interested in reading the documentation of this keyword as well :)

Hope this helps,
Yann

slorenz
Posts: 7
Joined: Tue Jun 22, 2010 8:38 pm

Re: P300 Speller with random letter flash

Post by slorenz »

Oops. That fix worked perfectly. I am showing my novice status as a C++ programmer! :)

Thanks for all your help, Yann. We will probably try running this version of the P300 Speller next week!

Cheers,
Sean

Post Reply