BI bogging down after being busy

[quest100]

For at least the last year when BI gets busy the FPS will drop on one or more cameras and never recover. This usually happens when viewing prior recordings. Once this happens the only way to recover is to restart the camera(s) (in BI) with the slow fps.

Normally my system chugs along nicely using ~20% CPU. This is with 18 active cameras, half of which are 4k and using sub streams. Most cameras are using ONVIF camera triggers and have limit decoding checked and no display of overlays on live views. A few cameras use BI motion detection. When multiple cameras get triggered at the same time the CPU usage increases up to 60-80%, sometimes spiking briefly to 100% before returning to normal.

It's when I go to review recorded footage that BI bogs down and does not recover. Usually no problem, but eventually I'll notice the CPU usage is very high and does not drop below 40-60%, even after I stop reviewing footage. If I then go to BI's info window and look at the cameras at least one will be sitting at a low FPS and the key frame rate will have dropped from 1.00 down below 0.5. Once this happens the camera never recovers until I restart the camera within BI. If I forget and fail to check I will come back the next day and find the camera still lagging. This never seems to happen with my 1080p cameras.

I have tried with and without hardware acceleration. I believe I have antivirus turned off for BI and its storage locations. I have the BI camera buffer set to 40 MB which should be plenty large.

I can see where BI would temporarily bog down when busy, but I would expect BI to recover eventually. I would also expect BI to alert me when the FPS or keyframe rate drops for an extended period of time.

Any ideas or anyone else seeing this?

 

[looney2ns]

4k cams add significant load on BI.
Being certain that you have BI properly excluded from ANY and ALL antivirus is imperative, as it can cause significant issues with BI.
Are you using substreams. Sub Stream Guide If not, you should.

 

[TonyR]

Does your BI sever's Ethernet connect directly to the cam's POE switch, and does not go through your router?

 

[quest100]

1) I will double check that Defender is excluding any BI directories and the app itself. I am not a Windows (10) expert, so any good links on what exactly to do?

2) I am using sub streams for all the higher res cameras. These are the same cameras that give me problems.

3) I am using a UniFi Dream Machine plus several UI switches. The BI machine and all cameras hang off one UI switch. Nothing camera wise is going through the UDM (router). I have verified this by running various speed tests on other devices on my LAN and consistently get around 900 Mbps, which is the max I would expect over a 1 Gbps Ethernet line. The cameras consume around 250 Mbps.

4) My main complaint is that BI never recovers once the frame rate drops for a camera. Even after the CPU usage drops down somewhat. The other cameras remain fine.

 

[quest100]

Defender has everything excluded that is mentioned in the pdf help file - BI, BI folder, alerts folder, db folder, and all three folders for storage of video. Both the alerts and the db are on a SSD with nothing else on them. The SSD made no difference when I installed it a long time ago except that it greatly sped up rebuilding the data base.

 

[Flintstone61]

What is the Computer you are using?
the hardware specs more specifically.....
Not sure if Onvif triggers are more or less resource consuming than BI triggers.

 

[Flintstone61]

My 18 cam Condo system has multi-cam triggers frequently. sometimes 40% of them if the Wind is blowing the trees around outside.
I have a mix of 4MP, 5MP and 2 MP cams. all triggers are from BI.
Resources can jump to 50% -60% or so ,,,,but always go back to normal after spiking.
Substreams on 15 of them.
no reboots have been needed

 

[quest100]

What is the Computer you are using?
the hardware specs more specifically.....

Most hardware specs are in signature. The computer is a Lenovo desktop with a 4TB HD for Windows, a 512 GB SSD for alerts and database, and a 10 TB WD Purple HD. In an external eSATA case are 2 x 16 TB WD Purple HDs. The monitor is 4k. No GPU card in system.

the only other programs running on it are DahuaSunriseSunset, BI Tools, and (sometimes) a browser looking directly at a camera. I did not redo the system using the media creation tool to clean out the extraneous crap (too lazy and don’t want to mess with a working system), but all non BI is generally few per cent CPU. The GPU reported by BI is usually less than 20%.

 

[quest100]

My 18 cam Condo system has multi-cam triggers frequently. sometimes 40% of them if the Wind is blowing the trees around outside.
I have a mix of 4MP, 5MP and 2 MP cams. all triggers are from BI.
Resources can jump to 50% -60% or so ,,,,but always go back to normal after spiking.
Substreams on 15 of them.
no reboots have been needed

This is very similar to my system. When nothing is happening the CPU is around 18%. With lots of triggers it may reach 50-60%. I have around 14 cameras using sub streams at 720p.

However when I review the previous nights footage from the 4k cameras the CPU jumps to around 50% and then starts increasing and eventually pegs at or near 100%. Once I stop reviewing footage the CPU drops down to 40-60% (never back to 18%) and the camera I was reviewing bogs down until I bring up the BI window for that camera and hit return.

 

[Flintstone61]

The one thing I see from your recent info that might be a clue to help you is that your saying your running a 4TB drive for windows. If that is a Spinner that could be a source of lag when putting a demand on the system to simultaneously write data, and pull data to generate a video review.
Optimally,( from what I learned here) it's going to perform best if Windows and BI are on an SSD. I have Windows/BI ( and by installation default) the db and Alerts on C:\ the Samsung SSD.
on both my BI machines.

 

[IAmATeaf]

Are you recording direct I go disk in BVR format?

 

[Flintstone61]

All i know for sure is that I want a streamlined data path from Cam to Computer. So I pull "home runs" straight to the POE switch, ( less points of failure) and Computer straight to POE switch. I am running a pair of Ubiquiti Loco Nano 2.4 Ghz. They supply a link to 2 5442 Turrets across the property with very little lag.

 

[Flintstone61]

720P subs might be resource heavy, if there are a lot of them. You could experiment on whether or not, changing those, has any noticeable effect on System performance.

 

[looney2ns]

See here: SouthernYankee BI Allocation
In short, put Windows, Blue Iris and BI database on the SSD.
Then record video to the WD purple drives.

 

[looney2ns]

In the trash is where they belong. Xeon processors are power hogs.

 

[quest100]

The Idea of not using a SSD for windows does not sound likely since I have looked at the activity on the C drive and see nothing.
The data drives are all and always very lightly loaded
All cameras are recording directly in BVR format and none have overlays showing during live viewing.


I will try shrinking the sub stream to less than 720.
I will try disabling most cameras (temporarily) to see if I can duplicate with one or a few cameras.
I will try changing the size of the camera buffers in BI - currently set at 20 MB.
I may try replacing the C drive with something else. I might try updating to Windows 11 via the media creation tool.
Anything else I should try?

Since no one else has reported seeing anything like this I’m going to have to assume that it is something unique in my setup, probably a miss configuration somewhere.

 

[IAmATeaf]

Another thing worth trying is disabling hardware acceleration both globally and checking and disabling in each cam, then see how the system behaves then enable bit by bit, cam by cam ….

I have seen post here before where a particular cam has been causing an issue.

 

[Flintstone61]

Put Windows / BI on an SSD and report back. whatever activity you think your seeing or not seeing may not be at issue. Everybody else is doing it 1 way, and you are on doing it another way and are having problems.
Maybe all those Switches you have mentioned in your signature,,,,are doing something?

 

[quest100]

I have completely redone my system and no longer have the problem with BI bogging down and staying bogged down. No sure what made the difference, but here is what I did.

I came back from a three week trip to find my BI computer had unsuccessfully rebooted. It looked like for some reason it decided to update from Windows 10 to 11. I tried various ways to save my old system or complete the update, but nothing worked. I suspect the 4TB hard disk that the system was on died.

I finally gave up and used the windows media creation tool to install Windows 11 on the 500 GB SSD drive. The SSD now has both the operating system and BI's data base instead of just the database. Installed a new copy of BI and added one camera with my usual settings. Everything worked fine so I duplicated that camera and changed the IP address for each other camera.

I now have all 17 cameras running in BI - 1 x 12 MP camera, 8 x 4k, 4 x 4M, and 4 x 1080P. A total of around 230 Mbps. All are using continuous recording, no live overlays and direct to disk recording. All but three cameras are using ONVIF triggers from the camera and no decoding unless required. These three are using BI's motion detection.

I started out with no sub streams. Using this the quiescent load reported by BI was 3-6%. As soon an any motion occurred the load would jump up to 60-100%. I think the system permanently bogged down but it may have been the system never had time to recover before the next set of triggers.

Once I added sub streams back in the quiescent load was again 3-6% but the triggered state usually did not max out the CPU and when it did max out, it quickly recovered.

I am still have a couple of problems that I did not have with my old system. I may ask about them in a separate thread.

1. 3 cameras won't trigger using ONVIF.
2. The 12 MP camera does not reply correctly to inspection. No longer able to see the sub stream.
3. Neither of my email server can send a test message.
4. The load reported by BI matches the load reported by adding up the loads of the individual processes running, but this falls far short of the total load reported by Performance Monitor. This seems to be a Windows issue, not BI.

 

[fenderman]

I have completely redone my system and no longer have the problem with BI bogging down and staying bogged down. No sure what made the difference, but here is what I did.

I came back from a three week trip to find my BI computer had unsuccessfully rebooted. It looked like for some reason it decided to update from Windows 10 to 11. I tried various ways to save my old system or complete the update, but nothing worked. I suspect the 4TB hard disk that the system was on died.

I finally gave up and used the windows media creation tool to install Windows 11 on the 500 GB SSD drive. The SSD now has both the operating system and BI's data base instead of just the database. Installed a new copy of BI and added one camera with my usual settings. Everything worked fine so I duplicated that camera and changed the IP address for each other camera.

I now have all 17 cameras running in BI - 1 x 12 MP camera, 8 x 4k, 4 x 4M, and 4 x 1080P. A total of around 230 Mbps. All are using continuous recording, no live overlays and direct to disk recording. All but three cameras are using ONVIF triggers from the camera and no decoding unless required. These three are using BI's motion detection.

I started out with no sub streams. Using this the quiescent load reported by BI was 3-6%. As soon an any motion occurred the load would jump up to 60-100%. I think the system permanently bogged down but it may have been the system never had time to recover before the next set of triggers.

Once I added sub streams back in the quiescent load was again 3-6% but the triggered state usually did not max out the CPU and when it did max out, it quickly recovered.

I am still have a couple of problems that I did not have with my old system. I may ask about them in a separate thread.

1. 3 cameras won't trigger using ONVIF.
2. The 12 MP camera does not reply correctly to inspection. No longer able to see the sub stream.
3. Neither of my email server can send a test message.
4. The load reported by BI matches the load reported by adding up the loads of the individual processes running, but this falls far short of the total load reported by Performance Monitor. This seems to be a Windows issue, not BI.

Most of your initial problem of maxing out on motion was related to the use of limit decoding, you should not be using that option.
make sure that onvif is setup correctly in the cam and BI and that you are using the onvif login