Large BI Install CPU @ 100% Ram @ 20GB After Clean Install

[DENcannaTech]

Hi All

I have read through the many posts regarding BI memory leak and CPU pegging issues. I have tried numerous things to figure this out and nothing seems to make a difference.

I have multiple locations protected by BI installs. Most have around 30 to 40 cams per location using BI5. I have standardized using 16GB ram, i7-9700K CPU, Intel UHD 630 GPU (on board Asus motherboard) SSD for runtime and OS drive and 7200rpm raid0 for storage. All locations run smoothly with around 5gb of memory use and 25% to 30% CPU. These are retail stores with around 16 hours of activity per day, cams set to record on motion triggers.

This week I began setting up a larger deployment with around 120 cams. I have 2 brand new machines that I built with 16GB ram, i7 3.6GHz CPU, 16TB storage array via 7200rpm SATA drives, M.2 1TB SSD for OS/runtime, W10Pro clean install from media creation tool, and 4port x 1GB Realtek Ethernet controller bridged to a 4Gbit uplink.

These machines are stupid fast and responsive and seemed to be working very well during setup of BI. Once I got to around 45 cams defined in the BI setup things started to crawl like they do when there is a CPU spike. I've dealt with those in BI in the past and it's always something fairly easy to remedy (windows defender issues, FPS being too high, etc...).

Nothing I do seems to remedy this issue though on these new machines. I did enable the Limit Decoding option on the cam setup which DID seem to eliminate the issue however, with that enabled it seems like the cams are not showing real time video on the consoles or UI3 from web. The behavior is very strange as it seems like they are almost asleep until a motion event occurs and then there is an almost 10 second or longer delay before they start showing real time video and/or recording.

Initially we did have audio streams enabled for every camera which seemed to cause a huge spike in CPU use. So that was disabled on the cams and in the BI cam setup as well. I am also forcing the FPS setting in BI to 15.

With these being slightly larger installs than my other locations, I did expect somewhat higher resource utilization however the 20GB mark seems excessive. Although I built these with 16GB, when I saw it approaching 14GB of utilization I added another 16GB to ensure we had ample resources in these larger deployments. So at the moment my 1st box has 32GB in it.

Once the CPU gets pegged at 100% the only way to make the system responsive enough to use or even shut down the BI service is disconnecting the LAN connection. After about 10 seconds of that being disconnected everything drops off and the system becomes usable again.

Couple of other things in our configuration:
-We are using direct to disc recording and BVR format
-Hardware decode is disabled (when we enable GPU options the video output is bizarre and there is no remarkable impact on CPU use)
-All Windows Defender and AV options are disabled, there is no other AV installed
-We are using the on board video controller, Asus motherboard, Intel GPU and NIC.
-Cams are forced to 15FPS. Audio streams are turned off.

Sorry if this is a wordy post, I wanted to include as much information as possible as I have tried numerous troubleshooting ideas to fix this. Previous forum posts have helped when we've run into CPU issues on our other installs but this one leaves me baffled.

Appreciate any suggestions or tips to fix this.

Regards,
-Ryan

 

[bp2008]

What resolution are the cams? All the same resolution and frame rate? If they are all 2MP then 45 of them at 15 FPS each would be 1400 MP/s which is about the maximum I would try to run on that CPU even with hardware acceleration enabled. With hardware acceleration disabled, cut that number roughly in half. You should figure out the hardware acceleration. It makes a huge difference when it is working properly, but there are limits to what it can do. As you add cameras, monitor GPU usage graphs in task manager.

What brand are the cameras? I don't remember hearing of any that are outright incompatible with Quick Sync.

Limit Decoding is best used with continuous recording where motion detection is not needed, and rapid/fluid live view is not important.

I trust that you are setting the frame rates to 15 FPS in the cameras' web interfaces, not just in Blue Iris where the Max rate setting does almost nothing and certainly does not reduce the frame rate of the camera.

 

[bp2008]

You have probably been through this page already, but in case you haven't: Optimizing Blue Iris's CPU Usage | IP Cam Talk

 

[DENcannaTech]

Thanks for the reply.

To answer some of your questions.

They are Sibell cams, and yes I've forced the 15Fps rate in the cams themselves not just Blue Iris. I've noticed BI seems to keep jumping back to 30fps which is a little concerning given I have forced that setting. 2560x1440 @ 15FPS is the current resolution and setting @ 2048k.

I haven't worked with this particular brand of cam before. Given their web interface and the solid construction of the cam I had high hopes for them but... Perhaps something is weird with them.

Now I've just discovered something very strange with this entire issue. Purely by accident the onboard Intel wifi adapter got enabled. I had joined our wifi during windows setup just to get some updates installed and had disabled the adapter ever since. While I was changing settings manually in BI on the cams I had disabled our wired connection and apparently inadvertently enabled the wifi connection.

With the wifi connection enabled my CPU load stays stable, RAM decreases from 20GB to around 15GB and both remain stable. CPU never goes beyond 45% and BI behaves exactly as I would expect it to.

The NIC I had been using was a quad-port Realtek chipset gigabit adapter. I had initially bridged all 4 ports and then just deleted the bridge and used a single port. The motherboard also has an onboard Intel gigabit NIC. Both the onboard and the PCie Realtek NIC produce the pegged CPU and 20GB ram utilization. Consistently though the wifi adapter produces stable results.

I have checked to ensure drivers are appropriate for both wired NICs, using Intel ProSet etc...

The wifi NIC is also Intel.

So..... What am I dealing with here? I do agree we are probably at the top end of CPU resources with that # of cams and 15FPS.

We do need fluid real time viewing abilities both at the console and via UI3. The decode only when needed option makes that difficult obviously or it would be a suitable remedy.

I'm really baffled after stumbling upon this weird behavior with the wifi NIC. While it does present a "solution" I can't feel good or confident about placing two of these on wifi.

I can reproduce the results consistently over and over, to the point of where if I disable the wifi NIC and enable the wired the system will get overtaxed and freeze within 30 seconds. The only thing that seems to overcome that on the wired connection is the option for decode only when necessary.

Thoughts on this?

What resolution are the cams? All the same resolution and frame rate? If they are all 2MP then 45 of them at 15 FPS each would be 1400 MP/s which is about the maximum I would try to run on that CPU even with hardware acceleration enabled. With hardware acceleration disabled, cut that number roughly in half. You should figure out the hardware acceleration. It makes a huge difference when it is working properly, but there are limits to what it can do. As you add cameras, monitor GPU usage graphs in task manager.

What brand are the cameras? I don't remember hearing of any that are outright incompatible with Quick Sync.

Limit Decoding is best used with continuous recording where motion detection is not needed, and rapid/fluid live view is not important.

I trust that you are setting the frame rates to 15 FPS in the cameras' web interfaces, not just in Blue Iris where the Max rate setting does almost nothing and certainly does not reduce the frame rate of the camera.

 

[DENcannaTech]

One thing worth noting, I've suspected something with these Sibell cams all along as I have numerous other BI installs protecting millions in inventory and greenhouse space that work flawlessly...while the FPS is set right the bitrate was still sky high and buried on another screen. Why their UI is like that is baffling and frustrating.

I just went through and manually tweaked all of them and my ram utilization is under 10GB now. The CPU and NIC issue seem to still be a thing though. I am going to swap in an Intel server NIC and see how I fare there. This is a new Asus motherboard I've been trying in these 2 builds and perhaps the on board has driver issues. The wifi thing still has me perplexed.

 

[bp2008]

2560x1440 * 45 cams * 15 FPS is 2488 MP/s which is way behind what you can reasonably run on an i7-9700K without limit decoding.

I'm guessing that the wifi is simply not keeping up with the video, and because of that, you are seeing less CPU/memory demand from BI while the streams are probably falling behind or dropping frames. But on the offchance that it IS a driver issue, I can think of nothing better to try than an Intel server NIC.