Sudden decrease in recording quality (green frames in particular), lots of troubleshooting info inside

GreyTheOld

Young grasshopper
Joined
Oct 26, 2017
Messages
40
Reaction score
6
All-
I suspect this remains a video card driver issue but I cannot find helpful information.

This may get long, troubleshooting is what I do professionally and so I try to be as thorough as possible. I have had this machine running flawlessly for over a year, it's an Optiplex 7040, which is an i7-6700 Win10 Pro currently pushing 24Gb of ram, onboard video through an intel 530. My entire home network is gigabit + a/c, but all cameras are wired. Main "alert" drive is an m.2, storage drive is a HGST 3Tb, BI running as a service. Historically, alerts have fired within 2-5 seconds, videos queue and start within 2-3 seconds no matter my connection, and no skips or frame drops.

Currently running 6 cameras, all Dahuas, I think there's 3 or 4 total models but this issue affects *all* of them. All cameras are hardwired, running at 15fps, h264, iframe at 15, lowish bitrate whatever the 4000 option is. SmartCodec OFF on all. All cameras set with the largest possible receive buffer of 40mb, none pre-caching more than 3 seconds. CPU runs around 3-5%, ram usage for total system is less than 40% of available, even when recording. One camera set to record 24x7, one set to record at night, other cameras all with hotzones and rules. ALL are set to direct record. All sub-streams are OFF, overlays OFF. I do have one camera duplicated to give "up close" warnings. Again, this has worked for a year without issue.

Suddenly maybe 2 months ago, I started getting frequent disconnects from 2 of the cameras, and then 4 of the cameras. "No signal" warnings in BI. Oddly they still pinged, so I logged into them and they were streaming to the IP just fine. Also, around this time, I started getting significant green frames when watching clips via the app. Badly. Green frame then severe lag then ghosting. The app has been incredibly problematic recently anyway, skipping forward or back seemingly ignores alerts. I have uninstalled and reinstalled the app, no change. The disconnects went away when I dropped the cams from h265 to h264. At the same time my alerts on the duplicated camera went from being basically immediate to taking 30 seconds to 3 minutes if they fire at all.

Watching via the actual PC, it's *better* but it's not great. Clicking an alert takes 10-15 seconds to start playing and there are frame drops and skips.

What I have done so far:

  • Originally all cams were at 30fps because I had the power to do it. A few were h265. I've bumped them all down to 264/15 and lowered bitrates. This fixed the disconnects but not the other problems.
  • I've run a persistent ping for hours, each camera stays sub-2ms, usually sub-1ms.
  • While logging into each camera to check settings, I let the video stay up, and no ghosting/artifacts/etc in the browser
  • I've run tons of diagnostics on each drive, no warnings no errors.
  • I replaced the network cable, but that's not it either. Speed tests and pings are great.
  • There was a significant windows update in there, so I went to the dell website and found a host of new drivers and pushed them all. This SEEMED to have an immediate improvement, especially with the video driver, but now a week later we're back where we were.
  • I have deleted and readded a few of the cameras to see if this improved anything, no change.
  • I turned DOWN the size of files from 4 to 2gb and increased the available storage on the fast drive.
 

looney2ns

IPCT Contributor
Joined
Sep 25, 2016
Messages
15,609
Reaction score
22,838
Location
Evansville, In. USA
This will most likely boil down to a video driver issue. You don't want the driver from Dell.
Get it directly from Intel, you may need to try different drivers to find one that works.
You might even try rolling back the video driver first.
Then read the Wiki on how to stop windows from updating drivers.
 

GreyTheOld

Young grasshopper
Joined
Oct 26, 2017
Messages
40
Reaction score
6
Thanks for that. The intel driver won't install, it says I need a dell driver. I don't have a rollback option. There's another new driver since I tried a few weeks ago, I grabbed that and one from last may, I guess I'll experiment and see what happens?
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
For a test , turn off the hardware acceleration on the CPU, and see if this solves the problem
What does motion look like on the play back, have some one walk across the field of view.

It is strongly recommend to do a clean windows 10 install using Microsoft media creation tool. This will allow you to use the non dell driver.

As a side note: I would up the bit rate on the cameras to higher than 8192 kb/S.
 

GreyTheOld

Young grasshopper
Joined
Oct 26, 2017
Messages
40
Reaction score
6
For a test , turn off the hardware acceleration on the CPU, and see if this solves the problem
As a side note: I would up the bit rate on the cameras to higher than 8192 kb/S.
I think you mean in the graphics card options, not BIOS, is that correct? Disabling acceleration I mean? This box runs headless but I can throw on a monitor and edit that later this evening. I'll also look at rebuilding as much as I don't want to...it also runs a few critical vm's but I can work around that I think.

I went ahead and upped bitrate to 10240 and set quality to best and it is a marked improvement in record quality, obviously. Actually, as alerts start to roll in, it's performing better all together however viewing in the app or the webserver (usually I RDP and I'm getting the same thing with RDP) I'm still getting a ton of motion artifacts and green frames on view, HOWEVER when I went to export a clip to show you what I'm seeing, the exported AVI is perfect, so it's just in display...so...graphics.
 
Last edited:

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
About the GPU hardware acceleration, i mean in BI.
IN blue iris setting screen -> camera tab set the hardware acceleration decode to NO
SYSTEM_CAMERA.jpg
On each camera setting -> video tab set the hardware acceleration to default.

CAMERA_VIDEO.jpg

Head less may cause problems. Put a monitor on it for now. OR try a DUMMY HDMI plug , search amazon
 
Last edited:

GreyTheOld

Young grasshopper
Joined
Oct 26, 2017
Messages
40
Reaction score
6
OOoooooooooooooooooh. Sorry for the dumb question. In our environment we've been battling nvidia acceleration issues for one app with no graphics settings on like 2500 machines and that's just the way my head was working.

Headless has never been an issue but I will look more into it! I've got a dozen unused monitors, no issue to plug one in. Thanks again!

Edit: I also rolled back the graphics driver to one from feb of last year and the ugly clips from 10 mins ago are showing fine now...cross your fingers for me.
 

GreyTheOld

Young grasshopper
Joined
Oct 26, 2017
Messages
40
Reaction score
6
About the GPU hardware acceleration, i mean in BI.
IN blue iris setting screen -> camera tab set the hardware acceleration decode to NO

On each camera setting -> video tab set the hardware acceleration to default.
Head less may cause problems. Put a monitor on it for now. OR try a DUMMY HDMI plug , search amazon
Ok, so yesterday I rolled back the driver to last years version and everything was fixed. Upped bitrate and quality went super. Then, like a dumbass, I decided to go ahead and finally uninstall BI4 because I'd never uninstalled it. I told it to keep settings and license info but it didn't, cratered it all and ate my db. Thankfully I had a recent backup and managed to get it back up quickly, although it did weird things like turn on PTZ everywhere and a few other oddities, but I put everything back EXACTLY as it was yesterday working great, and it promptly recreated the same behavior.

So I turned off hardware acceleration and it's fine. I'm only doing direct to disc, and exporting as necessary. Is there any real reason to even care if it's turned on?
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
If you are not running a large number of cameras or having high CPU when viewing the cameras I really see no need for it. Before my current BI 5 Intel system I had an AMD BI 4 system, there were no performance problems with the AMD used for BI.
 

GreyTheOld

Young grasshopper
Joined
Oct 26, 2017
Messages
40
Reaction score
6
Thanks. I really appreciate all your help.

I can run all 6 cams at 1440/15 with multiple viewing connections at once and neither cpu or ram goes above 40%, I'd say I'm good and it's fast as blazes. Now if only I could get a BI instance for my headless debian rig...lol.
 

miles267

n3wb
Joined
Dec 25, 2014
Messages
17
Reaction score
4
Hi - may I ask if or how you finally resolved this? Am experiencing something similar. Have disabled Settings > Cameras > Hardware accelerated decode = No (was Yes) and my CPU utilization still seems OK. Only it's much more sporadic now. Was previously about 7-10%. Is now about 15-20% and will periodically spike to 97-100% for a brief moment. I believe the pesky green screens/frames have gone away. Did you end up reverting to an older Intel video driver and which version?
 
Top