Random CPU spiking & Missing Frames

[norderredron]

Hello folks, hoping you all can help me diagnose an intermittent issue with my system. I've been hitting the forums for the last couple weeks and finally decided I better just post something.

I am using Blue Iris v4.0.2.3 x64 on a PC with the following specs:
i7 2600 @ 3.4Ghz Sandy Bridge
8 GB RAM 120 GB Intel SSD (Operating system, software, new clips/alerts)
4 TB Western Digital 5200rpm drive (Storage only)
ATI Radeon 7770 (for kicks, I know it isn't necessary)
Intel Corporation DH67BL motherboard

I have 2 Swann NHD-830s, 1 NHD-831, and 3 NHD-820s (two 820s have the firmware update to produce 3MP images, one does not, and only produces 1080p. They came this way.) I also have an old Panasonic KX-HCM10 PTZ in the building, and a remotely connected Trendnet TV-IP572 w/ audio (only camera w/ audio). I also have two NETGEAR ProSafe GS110TP Gigabit Smart Switch w/ PoE to power it all, one of which is connected to the NVR and a Cisco switch that handles the building's main network.

Here's my problem. I have the system buttoned down pretty well it seems, all cameras running at 15 fps, i-frame at 15, pre-buffer at 30, network buffer set to 10MB, Direct-to-Disc enabled, all cameras recording in h.264 format (except the PTZ records in MJPG), and have BI running as a service. Each of these settings are reflected in both the camera and the software. EXCEPT the PTZ and Trendnet may be different frame rates and forced down slower in BI, I'm not certain as they keep changing framerates slightly and those two aren't mine to configure freely.

Daytime CPU usage sits between 25-35% typically. Night time usage sits between 35-50%, almost all of the time, even when recording all 6 high-res cameras at once. Then, without rhyme or reason, the CPU jumps to 85-100% usage and resource monitor reports the CPU maximum frequency drops to 47%. During this time the bandwidth from each of the 2-3MP cameras drop from 1.2MB p/s to 400-800KB p/s and the frames drop by half, some of the cameras going down to 0 frames despite bandwidth still being received. Writing to disc also drops off dramatically, as it would if the CPU was lagging behind. After about 30-60seconds the CPU catches back up and everything goes back to running smoothly again. **UPDATE** As I write this, of course the night-time CPU usage is bottle-necking regularly off and on.

I am about to pull out what little hair I have left, or consider another option, but I am so deeply invested in BI already that I would hate to abandon it. I have been in contact with Ken somewhat but I have a feeling I'm overwhelming him with questions, so hopefully somebody here has something to suggest..

Things I have tried:
Disabled ATI Radeon 7770 HDMI audio driver.
Disabled Realtek audio driver (read people had some problems with sound latency being introduced by drivers)
Made certain Make High-Res JPEG is unchecked for all cameras.
Originally had numerous Hard Faults associated with writing new clips/alerts. Solved by using the main internal SSD for new clips/alerts, now only pushing Storage to the slow 4 TB.
DPC Latency Checker didn't show the lack of latency until about half-way through the CPU maxing out (even though BI reported getting 0-6 frames from each 15FPS camera.) Not certain how to isolate this.
I have reduced clip size to 1 hour/2gigs for all cameras to reduce the overhead of big files some. Object Detect/Reject is Unchecked (Due to suggestions about missing frames/recording not starting early enough.. seems to have helped w/ pre-buffer set correctly)
Wondered if the 5200rpm 4TB was introducing the latency, but after watching it write a multi-gig file with minimal CPU change I decided against it.
Played with numerous settings and thought I had it figured out a few times, only to find it would crop up again, usually within an hour or two in day-time, or 5-30minutes at night.

I'll be happy to lay out more specifics to anyone who asks, I'm at my wits end tonight. Thoughts??

 

[fenderman]

Welcome to the forum. Are you using antivirus? If so have you excluded the blueiris.exe file? (there are complete instructions in the help file)...
Are you certain BI is responsible for the spike and not some other software?
Have you tried a clean install of blue iris?
Is the pc free of malware? Is it a clean install of the OS?
Have you tried disabling one camera at a time to see if you can pinpoint a camera causing an issue?
Try only writing to the ssd and see if it eliminates the issue...

 

[norderredron]

Welcome to the forum. Are you using antivirus? If so have you excluded the blueiris.exe file? (there are complete instructions in the help file)... Are you certain BI is responsible for the spike and not some other software? Have you tried a clean install of blue iris? Is the pc free of malware? Is it a clean install of the OS? Have you tried disabling one camera at a time to see if you can pinpoint a camera causing an issue? Try only writing to the ssd and see if it eliminates the issue...

Thanks for the welcome and quick reply!

I am using Avast and have excluded blueiris.exe, blueirisadmin.exe, blueirisservice.exe, and all clip folders.

Blue Iris is the program capping out on CPU use 69-90%, leftover is system, Avast antivirus (1-3%) and Splashtop streaming (problem shows in mobile app viewing even when not streaming Splashtop) You can tell it's happening in the mobile app because the frames are reported as 0, or anywhere between 4-8 FPS... until it catches up, then all display 15 happily.

It's a fresh install of the OS (Windows 7 64-bit) and should be clean.

Working on pinpointing one at a time now, was hoping it was a setting/umbrella effect as I've already returned 3 cameras to Swann. To be fair, their 'tech support' initially gave me the wrong reset information, so 2 of them might have been good.

Unfortunately the SSD is only 120gigs, but I will reduce the folder sizes, wipe the clips clean, and try it if/when isolating cameras doesn't fix it.

Thanks for the suggestion to isolate, I was avoiding that for no good reason. I do have one NHD-820 that seems to be sending back really noisy night-time footage compared to the other two NHD-820s. Might be the culprit. First however, I set all three 820s to 1080p since that is what they are marketed as and I want to eliminate that possibility first.

 

[norderredron]

So isolating cameras didn't seem to make a difference. Dropping down to any 3 cameras meant no issue.

Decided to run intel's CPU diagnostic tool, passed accordingly. Thanks to their instructions I disabled Intel Speedstep (power management->Minimum Processor State -> from 5% to 100%), and for a very long moment, thought I had fixed my problem.

CPU is now sitting neatly at 28%-35% during night time with heavy recording. The only way I was able to reproduce the problem was by streaming index to my phone while viewing a 3gig+ file in the local BI viewer and jumping around in it. Then same old problem. Might be that 5400 rpm is just too damned slow, but you'd think I'd be able to reproduce the issue more easily if that was all. I watched BI write a 1.3gig file WHILE watching a 2gig file all off the same 5400rpm drive and the issue didn't come up then.

The real problem is everytime I 'fix' something, the problem doesn't always come back right away, so knowing what's helping or not is difficult at best. I also deleted all the clips/alerts I had. Another likely culprit.

 

[bp2008]

Then, without rhyme or reason, the CPU jumps to 85-100% usage and resource monitor reports the CPU maximum frequency drops to 47%.

This sounds like the CPU could be overheating and throttling itself to cool down again. Have you tried monitoring the CPU temperature? Download Core Temp 1.0 RC6 - MajorGeeks

 

[norderredron]

This sounds like the CPU could be overheating and throttling itself to cool down again. Have you tried monitoring the CPU temperature? Download Core Temp 1.0 RC6 - MajorGeeks

That was an original concern of mine so I ended up replacing the stock fan/sink with a solid aftermarket. Now the temperature sits at 63-67 degrees celsius. When it starts tacking out the temperature never goes above 73 degrees. When I ran the intel diagnostic it reported this temperature as 45 degrees below max.

I've ruled this out for the most part since the temperature doesn't climb before the usage maxes out.

 

[bp2008]

Hmm... well lets see. The clip database repair/regeneration I think is scheduled to happen every night at the same time. But it sounds like this CPU issue happens more often than that. It might not hurt to repair/regenerate the clip database anyway just in case.

 

[norderredron]

While bench testing I did notice that messing with clips/alerts rapidly within the BI console while streaming to my phone DID seem to cause the issue, but now I cannot reproduce it, though it's too early to say since the problem sometimes didn't show for an entire hour. Repairing the database didn't seem to bother it even though doing so before when I moved files around did cause the issue. I am wondering about all of the alert images/clips being processed at the same time when you go through the list, but don't see how to bench that besides staring at the hard drive queue length, write speed, and memory hard faults for correlations.

 

[bp2008]

If it is clip metadata loading causing the CPU to spike, it is probably related to the clip database. Maybe if you actually move the clip database off the SSD to the slower drive, maybe then the CPU will have to sleep a little bit more while it waits for disk I/O and the problem will go away?

 

[norderredron]

Problem is that is how I had it before, and it was throwing up all kinds of memory hard faults. Switching to the SSD seemed to clear that up. Maybe if the secondary drive was 7200rpm and not 5400rpm, which is what I'm considering switching to.

 

[bp2008]

If you get a new drive, WD Purple is a good choice.