Excessive CPU load with newer versions than 5.8.6.7 version of BI

[Gyula614]

My PC (i7 8700, 24 GB RAM, GTX 1660, 170 Mbit/sec total IP data rate from 28 cameras, CodeProjectAI 2.6.2) was around 85% CPU usage with maximal 1/sec trigger rate per camera.
With the latest BI update versions since 5.8.6.7, as the version number increments it gradually flats to 100 % CPU.
Is it normal, and be in future versions so should I revert back to 5.8.6.7 or earlier versions?

 

[wittaj]

Are you using substreams?

Have you done all of the optimizations in the wiki?

 

[jaydeel]

This issue might be related to this thread...

Clip: Disk can't keep up

Having trouble with this error all of sudden. All my disks are cached and not full. Any help would be great.

ipcamtalk.com

 

[tf9623]

Yes. Everything including as past 5.8.8.0 eventually dies. I had decided to roll back from 5.8.8.x and run 5.8.7.11.
Whenever I saw 5.8.9.x popup I waited and updated to 5.8.9.2 and within 10-12 hours I had errors - out of memory specifically -
and had to shut down. I rolled back to 5.8.7.11 and will figure it out.

I've had no issues - everything has been rock solid - no changes at all. I've got 16 or 18 cameras, 32GB, i5-9600k, 3x2tb NVMe,
some SSD, and finally a pair 18TB drives. No issues with disks or any of that.

One thing I have noticed if this helps - running 5.8.7.11 when I start up and run a few minutes I see CPU about 8% and
memory at 5.21GB (from Blue Iris) - when running a newer version memory starts off at 12GB and just goes up. Once I
saved an event log message which told me I was out of memory and using 78 GB.

I haven't updated or changed video drivers (Intel QSV) and maybe it could be a video driver issue and maybe the
newer (to me) Blue Iris (>5.8.8.x) are doing something different that wasn't present in earlier versions?

I know it will be addressed and I will just run this older version and check ever so often.

I'm just throwing what I have observed out there.

 

[Gyula614]

Are you using substreams?

Have you done all of the optimizations in the wiki?

Yes, the AI runs on substreams. No mainstream usage is allowed for any cameras. Eventually I tweaked the substream (third if available) quality to reasonable level 2 Mbit/s for 2 MP res., @ 15 fps, etc.) and AI really didn't give better results with with mainstreams. Haven't checked the wiki optimizations. In overall it was doing well with earlier versions. Run for months without issues. Just tried to upgrade today, hoping replacing YoloV5 with YoloV8 in CP AI may give better recognition accuracy. It really didn't, did the same mistakes. And in one go also updated BI. Then CPU went full load. I had to keep only a few cameras (3 or 4) with trigger enabled to reduce the CPU usage around 70-80 %. Previously 28 of them were triggering around 85 % CPU. After reverting BI version everything went fine.

 

[Gyula614]

Yes. Everything including as past 5.8.8.0 eventually dies. I had decided to roll back from 5.8.8.x and run 5.8.7.11.
Whenever I saw 5.8.9.x popup I waited and updated to 5.8.9.2 and within 10-12 hours I had errors - out of memory specifically -
and had to shut down. I rolled back to 5.8.7.11 and will figure it out.

I've had no issues - everything has been rock solid - no changes at all. I've got 16 or 18 cameras, 32GB, i5-9600k, 3x2tb NVMe,
some SSD, and finally a pair 18TB drives. No issues with disks or any of that.

One thing I have noticed if this helps - running 5.8.7.11 when I start up and run a few minutes I see CPU about 8% and
memory at 5.21GB (from Blue Iris) - when running a newer version memory starts off at 12GB and just goes up. Once I
saved an event log message which told me I was out of memory and using 78 GB.

I haven't updated or changed video drivers (Intel QSV) and maybe it could be a video driver issue and maybe the
newer (to me) Blue Iris (>5.8.8.x) are doing something different that wasn't present in earlier versions?

I know it will be addressed and I will just run this older version and check ever so often.

I'm just throwing what I have observed out there.

I haven't left mine running for so long. Just about 5 minutes. As CPU didn't fall from 100 decided to revert the version and really everything after 5.8.6.7 ate a bit more CPU as the version number increased and went flat from about late-March versions.
To the memory usage I saw similar when AI was not using GPU and the CPU couldn't keep up with the trigger rate in AI processing. Then the frames were buffered in the RAM and the PC restarted after being frozen for a while at full memory. Maybe now some other operation overloads the CPU and uses the RAM as a buffer of the pending operations.

 

[bp2008]

If your baseline is 85% CPU usage, it really does not take much to bump it up to 100%. 85% is already considered excessive.

Here are some things I recommend you do:

1. Check task manager to see how much of the CPU usage is Blue Iris and how much is something else.
2. Make sure direct-to-disk recording is enabled for all of the cameras.
3. Screenshot the Blue Iris Status window > Cameras tab. This includes a lot of useful information that helps us figure out if 85% is a reasonable CPU load for the amount of work you are throwing at Blue Iris.

 

[Gyula614]

If your baseline is 85% CPU usage, it really does not take much to bump it up to 100%. 85% is already considered excessive.

Here are some things I recommend you do:

1. Check task manager to see how much of the CPU usage is Blue Iris and how much is something else.
2. Make sure direct-to-disk recording is enabled for all of the cameras.
3. Screenshot the Blue Iris Status window > Cameras tab. This includes a lot of useful information that helps us figure out if 85% is a reasonable CPU load for the amount of work you are throwing at Blue Iris.

Camera status is great. Thanks for the idea. I was missing something like that for a long time. Attached the screenshots. It's running only for about an hour. 85 % is a good average. Now it's at version 5.8.6.7. Daytime it's somewhat more. All cameras are direct to disk (Although some privacy masking should be applicable on some cameras. That would be a real pain in the ... for BI). This PC is doing only AI lookup for unvanted visitors. There are separate NVRs for full recording.

 

[wittaj]

Something still seems off. Folks run 50 cams on a 4th gen at half the CPU% you are running.

I don't know if the FPS for main and sub not matching is a cause or if BI is struggling with H265 - what camera makes and models?

Another hardware question

I know this has been asked in the past (many times) and apologize for asking again, but I want to make sure that I get this right the 1st time and not make a dumb mistake that will cause me to start over partway down the road. I am researching hardware for the Blue Iris computer that will run...

ipcamtalk.com

 

[Gyula614]

Something still seems off. Folks run 50 cams on a 4th gen at half the CPU% you are running.

I don't know if the FPS for main and sub not matching is a cause or if BI is struggling with H265 - what camera makes and models?

Another hardware question

I know this has been asked in the past (many times) and apologize for asking again, but I want to make sure that I get this right the 1st time and not make a dumb mistake that will cause me to start over partway down the road. I am researching hardware for the Blue Iris computer that will run...

ipcamtalk.com

Yes. That would be nice. There are motion zone masks to reduce false detections on some cameras. But it would de-compress the stream anyway to hand frames over to AI. So it's de-compressing frames on motion trigger. That's a considerable work to do. And it may do resize to AI input, but I'm not sure.
Mostly they are Hikvisions and Dahuas, some Herospeed as well.
FPS mismatch is good idea. May try that tomorrow. Without motion triggers (unchecked) it uses at about 30% CPU with BI GUI open.

Hikvisions are 1023, 1027, 2026, 2051, 2T55, 2086, 3086, 2387. Dahuas are 5442, 5842. HeroSpeeds are mostly GK. The two no signal != 0s are also HeroSpeed but don't remember their model.

 

[wittaj]

Something seems off. Especially if you turn motion off and at 30%.

Is the computer used for anything else?

Try disabling every camera in BI.

With nothing running but BI it should be at 0% CPU.

Then one by one enable a camera. Enable it and watch it a bit.

If the CPU doesn't jump, then enable another camera.

Repeat cycle.

Maybe you have a wonky camera that is killing the CPU and a simple delete and re-add will bring it back in check.

 

[Gyula614]

Something seems off. Especially if you turn motion off and at 30%.

Is the computer used for anything else?

Try disabling every camera in BI.

With nothing running but BI it should be at 0% CPU.

Then one by one enable a camera. Enable it and watch it a bit.

If the CPU doesn't jump, then enable another camera.

Repeat cycle.

Maybe you have a wonky camera that is killing the CPU and a simple delete and re-add will bring it back in check.

It's dedicated to this task. Nothing else uses this PC.
Tried to disable the cameras earlier today. CPU went down to 0 %. Then gradually up to 30. It was a nice linear slope.
Will try the FPS. Lower FPS for subs gives better image.. Will try on some of them to see if it helps.
Now it's 3:18 AM here so getting some sleep.

 

[Gyula614]

Saved screenshots on disable and enable. Also got YoloV5 working. It has some internal bugs in CP AI, also attached. Is it something to do with camera resolution? This flats the CPU now.

 

[wittaj]

WOW that is insane. I have the same processor with more cameras than you and mine idles sub 5%.

If you stop the CodeProject Service, what is the CPU?

How many cameras are you using CodeProject on?

Are you using the camera AI instead on the cameras that support AI recognition?

 

[Gyula614]

WOW that is insane. I have the same processor with more cameras than you and mine idles sub 5%.

If you stop the CodeProject Service, what is the CPU?

How many cameras are you using CodeProject on?

Are you using the camera AI instead on the cameras that support AI recognition?

That's promising.
Just did a stop and restart to the CP AI on its webpage then the CPU goes down to 35% and then back to 100 on restart. The RAM usage also fell some GBs.
Had a sreenshot but haven't saved. Now tried again but it always autostarts. I can't stop it.
Anyway it seems to be related to the AI execution.
CP gets triggers from all cameras. Triggers are limited to 1/sec. And there are 3-5 additional images 500 ms wise depending on the camera, on a trigger.
Was wondering on using the camera AI. Is that working with ONVIF messages?

 

[wittaj]

But going down to 35% without AI on is still way too high.

Yes the camera AI is great and what I use the most. I use CodeProject for some of my cameras, but found the camera AI to be superior, at least for my use case.

 

[Gyula614]

But going down to 35% without AI on is still way too high.

Yes the camera AI is great and what I use the most. I use CodeProject for some of my cameras, but found the camera AI to be superior, at least for my use case.

The substreams are set to 720p or 1080p whichever is possible on the camera. May it have some rescaling to expected AI input resolution, that causes the CPU load?
So, maybe rescaling of sub's frames can raise the CPU.
What resolution do you use on substreams?
I guess there's an optimal resolution to the AI. And above that because of a possible downscaling there's no benefit to set the resolution.
I've also found that the cameras' recognition is more accurate. There are way less false positive detections.

 

[wittaj]

Yes, AI downrezes it to D1 or less I think. Most of my subs are at D1 resolution.

 

[Gyula614]

Ok. Will check it out. Usually, as I remember it was at about equal in detection with similar resoltion (around 640x480). But some high resolution cameras have distant scenes. For them I kept the higher res. And after that as Dahuas can do 1080p on third stream I just preferred to have 720p or 1080p on each cameras. That may be the issue. Will check it out later during the day.

 

[biggen]

If you go down to D1 on your sub streams you will see a massive improvement. If you are running your substeams at 1MP/2MP then why have substreams turned on at all? Might as well disable them and just use a single stream.