BI consuming 98GB RAM and locking up

[martinjdoutlook-com]

I am looking for some help with an issue I am having and am unsure if the issue is with Blue Iris or DeepStack directly or due to recent changes I have made to the PC configuration.

Here is my PC configuration

Dell XPS 8950

12th Gen i9-12900

128GB RAM

1x onboard NIC

1x 2 port PCIe NIC

Drive C NVMe 2TB

Drive D 7,200RPM 2TB

Drive E NVMe 2TB

Drive V 7,200RPM 18TB

Nvidia GTX 3070Ti



I have been running the latest version of Blue Iris, currently 5.5.8.1, along with the latest version of DeepStack, with no issues for a couple of months.

Recording is set to Continuous + Alerts

The cameras are split across 2 NICs, one on a 10.10.10.0 and the other 192.168.86.0 network


I made the following recent changes

• 6-16-2022 Installed Hyper-V
• 6-19-2022 Installed Docker and WSL2 components
• 6-20-2022 Installed the "latest update available" for BI but don't remember the version

I would summarize that my cameras "go dark" over time which is observable when I am in the timeline view.

6-21-2022 Noticed Blue Iris using ~30% CPU. ~14GB RAM and no GPU

• DS was not running and could not start through BI console
• There was no indicator of any motion from 5:47:54 until 19:39:44
• It would appear that the first of 12 cameras went dark
• 5:46:57 first of 12 cameras went dark
• 5:47:04 next camera lost
• 5:47:06 next camera lost
• 5:59:50 next camera lost
• 5:59:52 next camera lost
• 5:59:54 next camera lost
• 5:59:54 next camera lost
• 5:59:55 next camera lost
• 6:00:07 next camera lost
• 7:59:52 next camera lost
• 7:59:52 next camera lost
• 19:26:33 last camera went dark, possibly due to me stopping service
• 19:27:35 killed Blue Iris process
• 19:27:43 First camera to come back on
• 19:27:54 Last camera to come back on

BI started but DeepStack would not start



In case it was related to the Hyper-V install I stopped Hyper-V, removed the external vSwitch in Hyper-V to reset NIC and still could not restart DS

Tried a few other things and nothing worked

I rolled back to 5.5.8.1 and issue seemed to be resolved

6-29-2022 installed 5.5.9.3 since there were no longer issues as a test

7-5-2022 9:18 Noticed Blue Iris using ~43CPU. ~98GB RAM

Stopped service / killed process and restarted

7-5-2022 9:18

Everything is back up and running with ~4%CPU and 4.7GB RAM and ~40-75% GPU

Observations

First camera stopped 7-4-2022 21:49 and last camera stopped at 7-5-2022 1:59:51 with no video from 7-5-2022 1:59:51 to 9:19:29



From BI log

1 7/5/2022 2:00:34.189 AM Clips DBCompact: could not obtain exclusive DB access 0/7167/0/0

This is the only instance in the logs since I started saving to file on 7-1-2022

I also noticed that the camera Garage2 (the first to stop) stopped recording 7-24-2022 21:49:21 and there is an entry in the BI log

2 7/4/2022 9:50:28.045 PM Garage2 FORCE quitting CGadSocket thread 265 310 1652897

I also noticed that the camera FrontLeft stopped recording 7-24-2022 22:25:58 and there is an entry in the BI log

2 7/4/2022 10:26:54.591 PM FrontLeft FORCE quitting CGadSocket thread 18 310 1036323

There were no similar entries for the other cameras when they stopped working

There are no related events in any Windows event logs.

 

[martinjdoutlook-com]

Update:

Now regardless of the version of BI I am having this issue. I just installed 5.5.8.2 and this occurs.

I actually had the first blue screen I have seen in years Video_Scheduler_Internal_Error just a minute ago.

When the computer restarted it took 20 minutes for everything to start and it seems that when I killed DeepStack it began to respond. I stopped the BI service as well.

Not sure what to think of this.

 

[Flintstone61]

Somebody else had something like this happen last week and they were advised to downgrade BI to a stable version. You running a Beta version essentially.

 

[martinjdoutlook-com]

Somebody else had something like this happen last week and they were advised to downgrade BI to a stable version. You running a Beta version essentially.

I downgraded to 5.5.8.2 which is the latest stable and am still having the issue and it was a few minutes after installing 5.5.8.2 that I had the blue screen.

Here is the latest summary info:

Email to: support@blueirissoftware.com

Support data:
Version: Release 5.5.8.0 x64 (6/13/2022)
OS: Windows 11 Pro
CPU: 12th Gen Intel(R) Core(TM) i9-12900 [9%]
RAM: 4.09G/127.7G
Clips: 12449 items, 2.08T/15.72T
Storage: V: +763.6G, E: +1.61T

 

[Flintstone61]

go back more. you dont need to be on the bleeding edge.
"I have been running the latest version of Blue Iris, currently 5.5.8.1, along with the latest version of DeepStack, with no issues for a couple of months. "
What are your settings for Hardware decode?

 

[bp2008]

A BSOD (blue screen of death) indicates a hardware failure or a driver failure, and is basically never the fault of ordinary software like Blue Iris. Intel hardware acceleration seems like the most likely culprit if you're using that on any of the cameras. It has been getting buggier lately and the value proposition has been going down particularly if you utilize sub streams already.

 

[sebastiantombs]

I'm running DS on a GPU as well and noticed that when SenseAI got included with Blue Iris thing went out of control. I ended up with 48+ instances of python stuck in memory. I cured the problem by reverting to 5.5.7.11 Unless you're going to run SenseAI instead of DeepStack I'd suggest staying below 5.5.8.x

Here's a link to the full update list for BI if 5.5.7.11 doesn't show in the update window of the console, but it should be there as a stable version.

Blue Iris Update Vault

Blue Iris Updates (64 bit) - History

ipcamtalk.com

Blue Iris Software Updates

ipcamtalk.com

 

[martinjdoutlook-com]

go back more. you dont need to be on the bleeding edge.
"I have been running the latest version of Blue Iris, currently 5.5.8.1, along with the latest version of DeepStack, with no issues for a couple of months. "
What are your settings for Hardware decode?

I am running Nvidia NVDEC which has worked solidly for weeks until whatever changed impacted the system. All cameras are set to use the default.


This morning, after the issues yesterday and downgrading to 8.5.5.2 late last night, BI was still operating.


I have 12 cameras set to Continuous + Alerts, and am using substreams, and all of the other recommendations I could find from various sources to maximize performance and minimize system impact. I do have the cameras set to a very low threshold for motion but that was the reason for running the heavy hardware.

I made three changes to the environment within a couple of days that I could see as remotely playing a part in the issue:

1. Installed Hyper-V
   I installed Hyper-V to run an instance of Home Assistant although I think I will move that to docker. This would have taken over one of the three NICs when I configured a virtual "external switch" in Hyper-V. That should not be an issue since I configured the virtual switch for a specific NIC and deselected "Allow management operating system to share network adapter".
2. Installed WSL2 components and Docker for Windows.
   This also created a virtual switch in Hyper-V but it is internal only. I created a Pi-hole instance and later after the issue began created an instance of codeproject/ai-server. ai-server does not support or didn't last time I checked, NVIDIA GPU yet so I am holding off on that.
3. Updated the version of Blue Iris.
   I updated Blue Iris during or right after making the changes above. Once I began having issues I downgraded back down to 5.5.8.1 and thought things were working properly again but I was apparently wrong.

I did have something weird going on last week with Pi-hole that I have since fixed but I am not sure how that would have impacted Blue Iris or DeepStack. One change that I did make before wrapping things up last night was to point my DNS to 8.8.8.8 instead of Pi-hole (the local host).

The reason for the DNS change was the Docker for Windows UI could not reach the docker site after the bluescreen and reboot. This may have been because Pi-hole was not running. I am not up to speed fully on docker and wasn't sure of the relationship between the UI and the underlying services.

I did notice however when originally taking the performance screenshot above that Python and Vmmem were either not running or were not consuming enough resources to show up near the top. I started Docker for Windows and checked again and now they are.

Since BI stayed up overnight I think my next steps will be to let it run for a few hours since starting Docker for Windows and then if all is well change DNS back to Pi-hole and watch it for a while.

Thoughts?

 

[martinjdoutlook-com]

Thanks for the links. I will pull it down if it continues to act up.

I'm running DS on a GPU as well and noticed that when SenseAI got included with Blue Iris thing went out of control. I ended up with 48+ instances of python stuck in memory. I cured the problem by reverting to 5.5.7.11 Unless you're going to run SenseAI instead of DeepStack I'd suggest staying below 5.5.8.x

Here's a link to the full update list for BI if 5.5.7.11 doesn't show in the update window of the console, but it should be there as a stable version.

Blue Iris Update Vault

Blue Iris Updates (64 bit) - History

ipcamtalk.com

Blue Iris Software Updates

ipcamtalk.com

 

[sebastiantombs]

Your operating environment has too many variables to be able to diagnose the problem remotely in text messages. I'd suggest back out changes, one at a time, until things stabilize again. If you're running sub streams there is no point to using HA, Intel or NVDEC. In fact it may actually add to CPU utilization, just a little, because the CPU has to offload the video for processing by the GPU. Sub streams will reduce CPU utilization by a factor or five or more which is way beyond what HA can reduce it.

 

[martinjdoutlook-com]

Your operating environment has too many variables to be able to diagnose the problem remotely in text messages. I'd suggest back out changes, one at a time, until things stabilize again. If you're running sub streams there is no point to using HA, Intel or NVDEC. In fact it may actually add to CPU utilization, just a little, because the CPU has to offload the video for processing by the GPU. Sub streams will reduce CPU utilization by a factor or five or more which is way beyond what HA can reduce it.

Definitely agree with the text thing. I do this sort of thing for a living and a 2-minute call is worth days of back and forth in text or email.

I did not realize that there was no advantage to using HA when using substreams. Is that because nothing is actually being processed with the feed? Is that unrelated to DS using the GPU? What about when converting to MP4 during export, unrelated as well?

I know that once I started using substreams, D2D, and enabled HA my CPU utilization dropped way down but I just assumed that was because the GPU took over the load. Now I feel like an idiot.

Things appear to be stable, although it has only been a few hours, but I will turn off HA and see what impact that has.

Thanks for the response.

James

 

[sebastiantombs]

I used to do help desk many moons ago, to many. The advantage I had was it was "in house" so if I couldn't get things going over the phone it was only a short walk and maybe a few flights of stairs to put hands on.

As long as it's stabile again is what counts. I'm waiting for SenseAI to support GPU processing before I upgrade past 5.5.7.11 and that will have to be support for NVidia GPUs, CUDA based. The bleeding edge of updates can be very painful when the styptic is applied to stop the bleeding.

 

[wittaj]

Around the time DeepStack was introduced, many here had their system become unstable with hardware acceleration on (even if not using DeepStack). Some have also been fine.

This hits everyone at a different point. Some had their system go wonky immediately, some it was after a specific update, and some still don't have a problem, yet the trend is showing running hardware acceleration will result in a problem at some point.

However, with substreams being introduced, the CPU% needed to offload video to a GPU is more than the CPU% savings seen by offloading to a GPU. Especially after about 12 cameras, the CPU goes up by using hardware acceleration.

 

[martinjdoutlook-com]

I used to do help desk many moons ago, to many. The advantage I had was it was "in house" so if I couldn't get things going over the phone it was only a short walk and maybe a few flights of stairs to put hands on.

As long as it's stabile again is what counts. I'm waiting for SenseAI to support GPU processing before I upgrade past 5.5.7.11 and that will have to be support for NVidia GPUs, CUDA based. The bleeding edge of updates can be very painful when the styptic is applied to stop the bleeding.

I never worked a service desk but I have some teams that do. I have what might be an even more difficult job, explaining to a bunch of executives why when we spent $10M on cybersecurity last year, we are going to need to spend another $5M this year. Just kidding about the level of difficulty. The guys and gals that work in today's Tier 1/2 support have a lot more technology to deal with than I did 25 years ago and all their customers can Google enough to slow the process down.

I am waiting on SenseAI as well. DeepStack has not been accurate for me. It may detect a cat with 75% accuracy when it is really one of my two 80lb German Shepherds. To be fair I am using the out-of-the-box configuration.

After turning off HA I noticed that the CPU usage seems to be about the same or maybe a little lower. It varies from 4 to 12 percent. Memory usage is considerably less at around 1.6GB right now compared to 5GB or more before. That number may go up as the service continues to run.

What I do see as standing out is that in Task Manager Blue Iris is not showing any GPU usage where it was using up to 75%. I assume that DeepStack is using the GPU but I am not sure how to tell.



I also checked in Process Explorer (sorted by GPU usage)


I do see entries in the BI log for AI but assume it could be using CPU?




Any idea how to tell if DeepStack is using the GPU?

Thanks again.

 

[martinjdoutlook-com]

Around the time DeepStack was introduced, many here had their system become unstable with hardware acceleration on (even if not using DeepStack). Some have also been fine.

This hits everyone at a different point. Some had their system go wonky immediately, some it was after a specific update, and some still don't have a problem, yet the trend is showing running hardware acceleration will result in a problem at some point.

However, with substreams being introduced, the CPU% needed to offload video to a GPU is more than the CPU% savings seen by offloading to a GPU. Especially after about 12 cameras, the CPU goes up by using hardware acceleration.

I didn't catch what you were saying until I read @sebastiantombs post.

 

[sebastiantombs]

Open Task Manager and go to the performance tab then the GPU on the list on the left. It won't show, specifically, what is using the GPU but it will show GPU utilization and give you a pretty good idea.

Incidentally, convincing management is harder than convincing end users. Worked both sides of that fence.

 

[martinjdoutlook-com]

Found it

 

[sebastiantombs]

I'll also say that switching to a custom model will help as well. Using the default model for DS has it looking for all kinds of things we'll never see, or if we do it'll be the least of our worries. We don't need to detect giraffes, zebras, elephants, laptops, tables, chairs and so on. The result is scanning takes a lot longer than necessary to find a person, cat, dog or vehicle.

This is a 34 page thread but will get you the right models to use for your own situation -

IP Cam Talk Custom Community DeepStack Model

I figured out how to download the images that DeepStack used to train the object model. With these images I can start to make the community custom DeepStack model, below are the steps to create the custom model. The first step would be to create a new DeepStack custom model using the same image...

ipcamtalk.com