BI5 Memory Issue

[JonathanT83]

Hey,

New user after a little advice regarding BI!

We have had BI5 running for sometime now (2 years apx). When we first set it up we had numerous crashes that appeared to be storage related, but after a few updates of BI the issue went away and we have been happily ticking along running continuously without any major intervention.

I have just recently started to see major crashes (full PC crashing) happening, we use the console as a viewer so can see it happening. The last couple of days the amount of crashes seems to have risen to every 30mins.

It looks like it is a memory leak issue as in the windows event viewer I am seeing lots of 'system resource exhaustion' errors:

2004: Windows successfully diagnosed a low virtual memory condition. The
following programs consumed the most virtual memory: BlueIris.exe (4792)
consumed 7587278848 bytes, BlueIris.exe (15884) consumed 1992241152 bytes, and
MsMpEng.exe (4080) consumed 293117952 bytes.

I also get:

7031: The Blue Iris Service service terminated unexpectedly. It has done this 1
time(s). The following corrective action will be taken in 1000 milliseconds:
Restart the service.

Initially I had the virtual memory set to system controlled, and I was seeing the amount of memory used by BlueIris in the errors exceeding 36GB. I have physically turned in to manual and set the allocation to 4GB just to see what happens.. but no difference.

I also see the following error under Applications:

Faulting application name: BlueIris.exe, version: 5.7.9.6, time stamp:
0x64dfd552
Faulting module name: BlueIris.exe, version: 5.7.9.6, time stamp: 0x64dfd552
Exception code: 0xc0000005
Fault offset: 0x00000000012d1380
Faulting process id: 0x0x21EC
Faulting application start time: 0x0x1D9D4E6EAFD1B77
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\Program Files\Blue Iris 5\BlueIris.exe
Report Id: b8b7e252-94eb-40d5-8031-8af18cc32b79
Faulting package full name:
Faulting package-relative application ID:

This error code also points to a memory issue.

It all starts with the system becoming laggy, followed by the BI console freezing the live video. I can go full screen, switch back and fourth like its working but all streams are stuck. The next phase is I loose the desktop icons and taskbar. Ctrl-Alt-Del makes the PC effectively black screen and its then completely crashed. Power button hard off is the only way to recover it.

I am running 21 cameras (Reolink) of which 2 are 8MP, the rest are 1080p on a Dell Optiplex (i5-10505 cpu, 16GB ram, Win 11Pro, 6TB HDD storage drive and an NVMe for OS / BI-db).
The two 8MP cameras used to be a complete nightmare in that they lagged by a huge amount, dropped signal repeatedly etc, had significantly reduced FPS etc. This was solved many moons ago by enabled Intel HA for just those two. Frustratingly I can only view the main stream for those two cameras and it only shows as H265. All the other cameras are happy to run substreams. I can enabled HA for all the other cameras but one by one it turns off - not an issue as the PC seems quite capable of running without. I never usually see my CPU usage above 2% (and GPU above 14%). Memory usage always used to sit around 4GB, but now it creeps up (sometimes runs).
I havent changed any of the settings for the cameras or the software. We are D2D, Intel HA on the two 8mp cameras, etc. All the same. BI is running as a service and the console is running 24/7.

I have ran scans on both HDD in case its a failure of one of them, all shows OK. I have also ran the windows memory diagnostic tool - that also came back with no errors. I even ran an old memory util that stress tests it overnight - came back in the morning and all was fine.

I have update all of the drivers on the system including the intel chipset / UHD graphics drivers (I saw that a lot older version had memory leak issues) and no difference.

I have sat in front of the screen with task manager open and can watch the memory usage go from around 25% up to 90% and then things start to go, once it hits 98% its effectively crashed.

I can confirm from watching it under process view that the memory culprit is BI - its gets higher and higher. I closed the console, rebooted to just have the service running and I get the same result of a crash within 30mins.
I downgraded BI (as I always update to the last critical stable version when asked) and no change. I have windows defender disable via group policy, I dont have any other AV installed. This machine is literally just for BI. The only thing I have installed, but not running, is the Reolink windows software so I can edit the settings on the cameras themselves, but thats been there since the beginning.

I open to suggetions as to where to look before I submit a ticket as I'm hoping that this may be some simple to solves? I consider myself IT competant but this has me beat!

I have found one other oddity in the console - I had all my cameras set to 15FPS max to help with load, for some reason I cant seem to manually set this any more. I set it, then go back in and its change to a random value such as 17fps or 19fps.. dont get that one but I dont think thats causing the issue.

At this point im contemplating a system restore back to factory settings ad then reinstalling BI but that seems drastic!

 

[bp2008]

Memory leaking like that is not common anymore but I think it is still typically caused by Intel graphics. Have you tried disabling intel hardware acceleration on all cameras? Confirm by looking at the Blue Iris Status window. The "HA" column should show - in all rows.

Also don't try to use Intel hardware encoding under any of the streaming profile configurations. In my experience it does not work and just falls back to software encoding anyway, but if Blue Iris attempts and fails to use it, that could potentially be a cause of leaking resources.

In the latest update, Blue Iris supposedly added some compatibility stuff for Reolink main+sub streaming. 5.7.9.3 Aug 14, 2023 Maybe it can help with the 8MP cams.

MsMpEng.exe (4080) consumed 293117952 bytes.

That is Windows Defender. Granted it was only consuming 293 MB, but it nonetheless it is still running. I know you say you disabled it but also try adding exceptions for the Blue Iris 5 directory as well as the directories where the clip database and clips/alerts are stored.

 

[JonathanT83]

Hey, thanks. thats interesting! Will see what that profile does - that is the exact issue I have with the 8mp cams!

Yes, i did the group policy editor method which is meant to disable it completely! I will have to double check that, but yes, will check the excemption list in the mean time.

 

[JonathanT83]

So, a little update..

I have since found out that using group policy editor no longer disables Windows Defender in W11.. apparently it comes back on a reboot. I have found another permanent method that works. Defender is not running and not even accessible via settings!

I disabled all the cameras but the troublesome cameras and its been fine, I have re-enabled 4 cameras at a time and so far it has been ok.

I am now running all cameras again and it has been stable. It looks like the culprit may have been Windows Defender..

Will post back after a little while of running!