Graphic card crash / overflowing system load

[EstebanBT]

Hello everyone,

I'm running an experiment where I process a signal in real time with Matlab. Everything seemed to work fine at first. However, I noticed that the graphic card driver crashes after running the application for a while. I could not check exactly if there is any associated error, as the screen goes completely black and the only thing I can do is rebooting the whole computer. I found some work around about this, but it lead to another problem so I think it was important to mention it. Some more info:

- The graphic card is an AMD Radeon R7 250. The driver is correct and up to date, but I already tried to reinstall it again anyways.
- When the computer crashes, I can see it is still connected to the Acquisition server, as this is in a different computer and the number of connected devices shows that.
- Checking the event viewer of Windows I found the event "Display driver amdkmdag stopped responding and has successfully recovered". That happens several times in a row on the nights the graphic card crashes.
- It only happens when OpenVibe is running.

Finally, I decided to run the application but disabling first the driver, and the computer was able to run for the whole night without any crash. Nevertheless, I noticed next morning that the applciation was running super slow, with a system load of 100%. And I think that both problems could be related somowhow... Actually, I believed that the problem could be related to the next topics:
- viewtopic.php?t=9828
- viewtopic.php?t=9925

I attached a screenshot of my scenario. Some more info:
- The Matlab version I'm using is 2020b.
- I am using the latest version of OpenVibe
- When the system load overflows, the Matlab box throws the warning that the script is not running in real time. The signal display has lags of minutes too.
- The computer has a memory RAM of 64 GB and 16 CPUs.
- The Matlab code does not store any data by itself, it just processes some signal, sends some stimulation and writes data in an excel.

Thank you in advance for your help.

Best,
Esteban.

[EstebanBT]

Hello again!

I finally found where the problem is. I am quite sure it is the "Time based epoching" the box that makes OpenVibe consume that much memory.

So far I tried translating the whole Matlab code to Python, because that's where I thought the issue was, but it was still crashing the same way. I also tried to run the same OpenVibe application but removing both EDF Writers, and the problem was still happening.

Finally, I decided to run the application by removing first a different box each time, until I found out that the only way to stop the increasing memory consumption was removing the "Time based epoching" box (while keeping the rest as they are in my image from the previous post). I do not know why this happens but, I guess it is an unexpected behaviour of the box, as it should not accumulate information further than the Epoch duration parameter (in my case, 8 seconds).

I can try to code a Time based epoching box by myself in Matlab/Python, but it would be nice if you could check whether something weird is going on there and find a fix!

Best,
Esteban.

[Thomas]

Hi Esteban,

Thanks for your clear message and the investigation.

On my side I can run a scenario with several TimeBasedEpoching boxes and stay under 5% system load.

I am wondering if it is not the TimeBasedEpoching+MATLAB boxes working together that could cause this problem. What are the settings of the TimeBaseEpoching box in your scenario?
Have you tried running the scenario with the TimeBaseEpoching but without the MATLAB/Python box ?
The MATLAB box likely receives a lot more data when receiving from the TimeBaseEpoching, which might change its behavior.

Could you try running one of the box-tutorials such as box-tutorials/python3/python3-sinus-oscillator.xml on you machine to see how the load is (it has TimeBaseEpoching boxes and runs fine for me)?

Cheers,
Thomas

[EstebanBT]

Hi Thomas,

Thank you for looking into this. You are right, the TimeBasedEpoching does not increase the memory usage by itself (I tried the scenario you mentioned and it remains constant).

So I tried different scenarios, and this is what I got:

- Scenario4 (original scenario without the TimeBasedEpoching): does not overflow (that's why i thought that the problem was from the TimeBasedEpoching).

- Scenario2 (Python/Matlab box removed): overflows memory (which made me think again that the problem was TimeBasedEpoching, but see next).

- Scenario3 (like scenario2 but without EDF writers): does not overflow - here I thought that maybe connecting TimeBasedEpoching directly to an EDF writer would lead to a crazy use of memory.

From scenario3 I thought that it could be an issue of the EDF writers, so I removed both of them from the original scenario (see first post), but it still overflows memory. So maybe you are right and it is the connection between the TimeBasedEpoching and the Matlab/Python box??? (and then the overflow from scenario2 would be due to the TimeBasedEpochin>EDF writer connection).

Some more information that might be helpful:

- I coded my own TimeBasedEpoching in Python and it leads to the same problem.
- The TimeBasedEpoching parameters are duration=8s, interval=0.01s.
- Signal comes from the Generic Oscillator, 4 channels, sampling frequency of 512.
- I checked the RAM memory usage rather than the system load for this post. The RAM memory usage increases in 0.1 GB every 12-13 seconds.

And maybe a little explaination of my python/matlab box:
It computes the phase of a signal in real time. It takes the 8s epoch and computes the phase for the last sample, with a clock frequency of 100 Hz (to match the 0.01s interval of the previous TimeBasedEpoching). It does not store any data at all.

Thank you for your time.

Best,
Esteban.

[Thomas]

Hi Esteban,

Thanks for all the clarification.

I didn't focus too much on the EDF File write, but now that you mention it, this box stores all the data it receives until the scenario stops, and then write it to the file. Until the data is written the file, it's stored in RAM of course, which explains Scenario2.
You could use the GDF File Writer in order to store you data, as this box writes in the file as the data arrives.

I ran some tests and with the same channel/smapling and the same TimeBasedEpoching, connecting to a python box that actually does not do any computing, and I get a memory leak too.
Since the epoching is quite intensive, the RAM fills up quite fast.

I will need to dig more into this, as it may be something not so trivial with the C/Python binding.

Sorry for not bringing a solution now, and thank you for bringing up the issue.

Cheers,
Thomas