Hardware acceleration cuts out intermittently

[mrc545]

I am running a dedicated BI PC (BIv4) with an i-9900k processor, and two NVIDIA cards for hardware accelerated decoding (RTX 2060 for the brunt of the load-mostly 8mp cams, and a GT 1030 for some of the less tasking cams). During the daytime, BI is reporting about 2200 MP/s. I have NVIDIA CUDA enabled globally for hardware acceleration, and I select the discrete GPU I want to use in the local camera decode settings. When functioning properly, each GPU is showing about 50-60% decode, and GPU-Z is reporting 20% overall GPU load for the 2060, and 70% for the 1030.

I have this PC in the other room (it would trip the circuit breaker in this room due to the amount of other devices in here, and is extremely noisy from the fans) and have a persistent RDP session set up on my everyday computer that is being displayed to a 4k monitor (this computer also has an RTX 2060). Hardwired gigabit networking throughout the house. I configured the remote viewing to "unrestricted", but set the live preview rate to 5 FPS so the BI PC doesn't get absolutely crushed from the RDP session. CPU load on the BI PC hovers around 80% in this configuration.

As a foreword- Quick Sync would never play nice with my setup via RDP. It seems like it would expect a monitor to be plugged in, and even with a headless display adapter, it still wouldn't get picked up by BI (however, it would work with HandBrake during testing so I know the QS functionality was working for the PC as a whole). Whenever I would select Intel (then restart), it would just revert back to "No". Therefore I went the discrete GPU route, and it has been working fine for my purposes.

Yesterday, I noticed that the CPU load shot up from the low 80%'s to 100%, and the video decode on each GPU dropped down to 10-20%. So it seems like something in BI isn't letting the GPU's do their job.You can see in the task manager screenshot where the GPU usage drops down.

If I exit BI and restart, everything will work fine again... until it doesn't. Sometimes it's a few minutes, sometimes it's a few hours.

The only reason this is bugging me is that no changes have been made whatsoever to the system in the past week. No config changes, no new cameras added. I don't get it. It has been running fine up until yesterday.

As part of my troubleshooting, I carried over a 1080p monitor, keyboard and mouse to the other room to test the BI PC locally, instead of through RDP. It did the same thing with both GPU's. I went into the BIOS to enable iGFX so I could test with QuickSync, and the same thing happened.

Any ideas? I am BIv5 upgrade eligible, but was trying to hold out a little longer, as v4 has been working fine. I also did a clean reinstall of v4, and the same thing is going on.

 

[bp2008]

BI v5 was built with a newer Intel media SDK so that might help as far as Quick Sync is concerned. Have you tried updating Nvidia drivers? I'd also suggest checking the Event Viewer in Windows to see if you can find any warnings or errors related to GPUs, hardware acceleration, or Blue Iris.

 

[SouthernYankee]

I am on BI4, I disable the Nvidia GPU and my CPU load went down. Can not explain it. Also I am on a Windows home server 2011 running a AMD FX-6600 processor.

I plan on building a new Intel system with BI5 on a windows 10 pro in the next month or two.

 

[mrc545]

Most current Nvidia drivers are loaded (8/27 release date, per Nvidia). Conveniently enough, the 2060 and 1030 use the same driver, so they play nicely together. I added the 1030 about a month ago and it's been working well in tandem to handle the overflow loads from upgrading cams. Nothing alarming in Event Viewer from the past 24 hours as well.

I reduced the framerate of a few less essential cameras to get it down to around 1900 MP/s, and it has been running fine since I started this thread about 4 hours ago. Just seems odd, as I had a fair amount of CPU overhead when I was cranking out 2200 MP/s.

 

[vidmo]

@SouthernYankee Windows home server 2011 is basically Windows XP as you probably know. I would not be surprised if the drivers just don't support the GPU offload that well.

 

[mrc545]

The issue is coming up every other day or so now. Event viewer is still showing nothing out of the ordinary. I will update to BI5 within the next day or so and report my findings for posterity.

 

[mrc545]

Upgraded to BI5. The hardware acceleration is still cutting out when the MP/s is over 2000. This isn't really dependent on CPU load. When I increase the framerate on the cameras, the CPU load will still be in the 80% range, but the video decode and CUDA will drop to near nothing on the GPU's, and as a result, the CPU gets maxed out. Also when this happens, the framerate on some of the cameras will drop down from 25-30 fps to maybe 1-5 fps per the BI status window.

If I lower the framerate across the board, everything runs fine. Just kind of bummed, since I have the resource overhead to run them at a higher framerate, but something is triggering hardware acceleration to stop working past a certain point.

 

[mikeynags]

Is there any debug logging for the CUDA that you can enable? It sounds like you are hitting some kind of threshold that triggers the cutout.

 

[mrc545]

Is there any debug logging for the CUDA that you can enable? It sounds like you are hitting some kind of threshold that triggers the cutout.

I see that NVIDIA has a CUDA Toolkit. I might see what that tells me.

I wonder if the two video cards just aren't playing nicely with BI in tandem. I don't remember it doing this when I just had the RTX 2060 installed. I'll disable the GT 1030, and max out the video decode capabilities on the RTX and see if this happens.

I'll also take bp2008's advice and see how nicely QuickSync plays with my system and BI5 since it uses the new SDK.

 

[mrc545]

bp2008, QuickSync is now working with my setup since going from BI4 to BI5. Thanks for the advice.

I have noticed, however, that CPU load goes up when I delegate QuickSync to perform decoding functionality per camera. I figured since it was a seperate part of the die, video decode and CPU load would be mutually exclusive. I'm thinking this has something to do with my peculiar setup, using it as a headless BI PC via RDP.

I have disabled the GT 1030 via device manager, and have just enough cams decoding via QuickSync so that the RTX 2060 is hovering around 90% on video decode. I'll crank the framerates back up and report back with my findings and see if hardware acceleration cuts out in this configuration.

 

[Zanthexter]

'm thinking this has something to do with my peculiar setup, using it as a headless BI PC via RDP.

Semi-related, but BI 5 has a "remote admin console". In the help file (now a PDF) look for the "Remote Management" section. It might work better for you than RDP for day to day monitoring and adjusting BI config. You'll still need RDP for other stuff. I would expect it to largely offload the significant resource drain of the BI GUI on to your remote system.

The main limitation is that you can EITHER have a remote OR a local admin GUI, not both, and only one remote at a time.

When I tested it, I was working remotely, and it wasn't reliable enough for that purpose. It'd crash, refuse to reconnect as a "2nd" remote, and I'd have to remote into the distant system and restart the BI service. But that was the day 5 was released and remotely. I haven't tested since, you might have better luck.

 

[mrc545]

Semi-related, but BI 5 has a "remote admin console". In the help file (now a PDF) look for the "Remote Management" section. It might work better for you than RDP for day to day monitoring and adjusting BI config. You'll still need RDP for other stuff. I would expect it to largely offload the significant resource drain of the BI GUI on to your remote system.

The main limitation is that you can EITHER have a remote OR a local admin GUI, not both, and only one remote at a time.

When I tested it, I was working remotely, and it wasn't reliable enough for that purpose. It'd crash, refuse to reconnect as a "2nd" remote, and I'd have to remote into the distant system and restart the BI service. But that was the day 5 was released and remotely. I haven't tested since, you might have better luck.

I wish there was a way to test it out before buying the second license. I watched a video of it on YouTube and the guy said it was blurry at times. I have the RDP session on my 4k monitor, and it's nice to be able to view footage from my 4k cams in native resolution. Not sure how the Remote Management functionality handles this. Does the client/remote viewing PC share some of the resource load with BI remote management? Or is it like RDP, where the host (BI PC) is doing all of the legwork, and it's just putting out a display to the remote PC?

 

[Zanthexter]

I wish there was a way to test it out before buying the second license. I watched a video of it on YouTube and the guy said it was blurry at times. I have the RDP session on my 4k monitor, and it's nice to be able to view footage from my 4k cams in native resolution. Not sure how the Remote Management functionality handles this. Does the client/remote viewing PC share some of the resource load with BI remote management? Or is it like RDP, where the host (BI PC) is doing all of the legwork, and it's just putting out a display to the remote PC?

You can test it with the demo. Download and install BI on the remote machine. Pick Demo when it asks for a license. You have 15 days to test. If you need more time, uninstall deleting settings, reinstall, and 15 more days.

Demo also works if you want to provide the raw unconverted footage to the police or a lawyer. I just throw it on a flash drive or DropBox share along with the .BVR files. This lets them see the time stamp and camera name metadata when you don't record overlays. (Performance issues aside, I don't like obscuring part of the view with text.)

The video quality is going to be affected by bandwidth and the how loaded up the BI server is. Was the YouTube guy viewing locally over a wired gigabit connection or N WiFi? Etc.

I just tested using the Demo by installing it on my office desktop and connecting it to the office Blue Iris. I can't speak to the video quality since I'm working remotely on both computers. What surprised me was that the CPU% on the BI server was hardly hit at all whether the GUI was local or remote. You couldn't really see a difference in the graph. On the one system we tried hooking to a 4K TV, the GUI would hit CPU% for 25-30%. Typical BI weirdness. One difference is the number of cameras, the office system only has 8, that 4K system had 48. So your mileage may vary quite a bit.

Please let us know how well it works for you

 

[mrc545]

You can test it with the demo. Download and install BI on the remote machine. Pick Demo when it asks for a license. You have 15 days to test. If you need more time, uninstall deleting settings, reinstall, and 15 more days.

Demo also works if you want to provide the raw unconverted footage to the police or a lawyer. I just throw it on a flash drive or DropBox share along with the .BVR files. This lets them see the time stamp and camera name metadata when you don't record overlays. (Performance issues aside, I don't like obscuring part of the view with text.)

The video quality is going to be affected by bandwidth and the how loaded up the BI server is. Was the YouTube guy viewing locally over a wired gigabit connection or N WiFi? Etc.

I just tested using the Demo by installing it on my office desktop and connecting it to the office Blue Iris. I can't speak to the video quality since I'm working remotely on both computers. What surprised me was that the CPU% on the BI server was hardly hit at all whether the GUI was local or remote. You couldn't really see a difference in the graph. On the one system we tried hooking to a 4K TV, the GUI would hit CPU% for 25-30%. Typical BI weirdness. One difference is the number of cameras, the office system only has 8, that 4K system had 48. So your mileage may vary quite a bit.

Please let us know how well it works for you

Definitely didn't know you could do this in demo mode. Thank you very much for the info! Will try this out tonight.

Not sure what the specs were on Youtube guy's setup, but I'll report back and see how it runs for me. If it works well, I'll just get another license. Thanks again.

 

[mrc545]

Tried remote management out. The images are so blurry/pixelated to the point where they're pretty much unwatchable (you can't tell an F-150 from a Silverado, just a blob shaped like a pickup truck).

Not sure where the bottleneck is. I am only receiving about 50-100kbps down on my remote viewing PC in my office (I have end to end hardwired gigabit. I have seen 900+ mbps from this NIC when I'm pulling stuff down from my NAS). In the remote management stream configuration (the "..."), I upped everything to maximum quality settings, and even enabled hardware acceleration. I have an RTX 2060 in both PC's, both of which are cabable of NVENC/NVDEC functionality. When I start up the remote session, Video Encode on the RTX 2060 on task manager only hits 1%. Nothing is being stressed. Plenty of CPU, GPU, RAM, and network overhead on both ends.

That being said, CPU usage did drop down by HALF. That's pretty awesome. I could even make the "remote viewing" set to unrestricted, and it'll only hit 70-80% CPU load. The promise is definitely there, but I think it needs to mature a little. It seems like BI isn't yet taking full advantage of hardware to provide a clear picture. Not complaining or armchair quarterbacking. I have been a happy BI user for the past 5 or 6 years or so, and am constantly pleased with the quality and innovation that one single guy can put out. I'm sure it'll get better eventually. Heck, Microsoft hasn't implemented h.265 for RDP yet, and they have an army of engineers and developers at their disposal.

 

[Zanthexter]

Tried remote management out. The images are so blurry/pixelated to the point where they're pretty much unwatchable (you can't tell an F-150 from a Silverado, just a blob shaped like a pickup truck).

Not sure where the bottleneck is. I am only receiving about 50-100kbps down on my remote viewing PC in my office (I have end to end hardwired gigabit. I have seen 900+ mbps from this NIC when I'm pulling stuff down from my NAS). In the remote management stream configuration (the "..."), I upped everything to maximum quality settings, and even enabled hardware acceleration.

Are you using an external address instead of an internal one? Some routers don't handle things well when you access something inside the firewall using an outside the firewall address. Try using the servers internal IP address as the Primary URL.

When you upped everything did you do it by double clicking on the entry for your remote server in the remote tab? Or elsewhere? Looking at mine, it defaulted to 512kb, resizing to 720p, etc. Helpful when working over the internet, not so much local. Try turning off everything that might save bandwidth as you've got bandwidth to spare.

I only have 1080p to test with here, but it looks pretty reasonable.

 

[mrc545]

I don't have an external address configured, just the local LAN. Same subnet and everything. I have my router set up as a VPN server, so when I open the app on my phone when I'm not on my network, I just initialize a VPN tunnel to the router, and I get access via that route. Figured it was the most secure option.

Yeah, I double clicked on the entry on the remote tab. When I would double click again to go back in, all of the settings were as I had previously applied them. I moved the screen over to my 1080p monitor, and it still looks pretty cruddy, but not as bad as on the 4k monitor.

 

[Zanthexter]

I don't have an external address configured, just the local LAN. Same subnet and everything. I have my router set up as a VPN server, so when I open the app on my phone when I'm not on my network, I just initialize a VPN tunnel to the router, and I get access via that route. Figured it was the most secure option.

Yeah, I double clicked on the entry on the remote tab. When I would double click again to go back in, all of the settings were as I had previously applied them. I moved the screen over to my 1080p monitor, and it still looks pretty cruddy, but not as bad as on the 4k monitor.

How does clip playback compare?

I threw a 4K camera on there, and it wasn't so good for the live view. Low frame rates, etc.

When I played a 4k clip back from the same camera it was fine. (Well, still not so great, but it's a cheapie camera lol)

If Blue Iris is streaming live view but downloading and playing back clips would explain the discrepancy. I don't know that there would be a fix outside of encouraging Ken to change how it displays a live feed. I'd be willing to take a 1 second or so hit to get a better picture when it's over the local network.

 

[mrc545]

Playback still isn't as good as it looks from the RDP view.

I agree, if there was some kind of delay that included a buffer to make the images look better, I'd be ok with that.

 

[mrc545]

I think I found the culprit, but I cannot understand the logical explanation as to the cause. Any advice would be appreciated.

When I didn't know any better, I bought a few Reolink RLC-410's, which would cut out on me all the time and just be a royal pain in my side. When I upgraded my primary/critical cameras to Dahua models, I took one of the Reolinks and repurposed it to provide coverage for a first floor window that was a blind spot for the other cameras. I didn't really care if it cut out occasionally, as I had the camera on hand, and figured I'd use it for something less important.

I got tired of coming to my PC every day with the CPU maxed out because hardware acceleration had crapped out, so I leveraged the fantastic functionality of BI Tools to restart BI once CPU load hit 95% and decided to live with using that as a mitigation. I had noticed that when BI Tools restarted BI, that one Reolink camera would show that it was offline for at least 5-10 minutes. All the other cameras would come back up immediately. I could log into the Cameras web UI and it would come right up, no issues, so the camera wasn't actually offline. Also did a ping -t on the camera and there was no packet loss at any point.

Figuring something was fishy, I disabled the Reolink camera on BI on 9/20. It has been over a week now and HA hasn't dropped out a single time, and BI Tools hasn't restarted BI at all.

This camera is direct to disk, and I had even disabled HA on it for testing purposes. My understanding is that BI simply ingests the feeds from the cameras that are broadcasting on the network, so I'm confused as to how taking the feed in could cripple my system and cause HA to cut out. I know Reolinks have a history of not playing nice with BI, but it's still pretty interesting nonetheless.