BI Crashing

[DurandClan]

Seems more likely to happen if I'm viewing remotely from UI3 using either Chrome or Edge. These are the two errors with the 2nd being the most common. This has been happening for quite a few versions now and I guess I keep hoping it will be addressed with each update that comes along. Any ideas on where to go deeper in troubleshooting? Hardware is a Ryzen5 3600, 32GB of RAM and with a GT1030 in the machine I handle of the camera processing via the CUDA cores. CPU in the machine usually runs at about 25-30%, nVidia card at 35% consistently and the card is plenty cool. the BI process levels off and usually stays comfortably at 7-8GB of RAM consumed so I'm not thinking this is resource-related.

Faulting application name: BlueIris.exe, version: 5.6.0.6, time stamp: 0x6304f083
Faulting module name: ucrtbase.dll, version: 10.0.22000.1, time stamp: 0x00e78ce9
Exception code: 0xc0000409
Fault offset: 0x000000000007dd7e
Faulting process id: 0x1a60
Faulting application start time: 0x01d8ba3d49df6da0
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\WINDOWS\System32\ucrtbase.dll
Report Id: c0e1abd1-4e87-4d20-8911-7a94d6b8c7b8
Faulting package full name:
Faulting package-relative application ID:

Faulting application name: BlueIris.exe, version: 5.6.0.6, time stamp: 0x6304f083
Faulting module name: ntdll.dll, version: 10.0.22000.832, time stamp: 0x3907dfbc
Exception code: 0xc0000374
Fault offset: 0x000000000010c9a9
Faulting process id: 0x1298
Faulting application start time: 0x01d8b9973d6af34f
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: af8c5d40-e40f-45c6-9802-e6b191edc875
Faulting package full name:
Faulting package-relative application ID:

 

[wittaj]

How about rolling back to a stable version?

 

[DurandClan]

How about rolling back to a stable version?

This has been happening since 7/1/22 and the errors on the July dates oddly enough show the BlueIris.exe version to be 1.0.0.1 which obviously is not correct. For the record, I have tried rolling back a number of times over the last month or so and no change in behavior. All of the crashes but one crash on faulting module ntdll.dll

 

[wittaj]

Are you using hardware acceleration?

 

[sebastiantombs]

Are you using sub streams? If not, implement them and stop using the NVidia card for hardware acceleration, with sub streams the CPU won't be overloaded at all. Alternately, as a test, shut down hardware acceleration as a test to see if there's a problem there. The CPU will take a hit to do that, but it may be a worthwhile test.

 

[wittaj]

Now that substreams are available, GPU whether internal or external is not needed for normal BI use.

Around the time DeepStack was introduced, many here had their system become unstable with GPU hardware acceleration on (even if not using DeepStack). Some have also been fine.

This hits everyone at a different point. Some had their system go wonky immediately, some it was after a specific update, and some still don't have a problem, yet the trend is showing running hardware acceleration will result in a problem at some point.

However, with substreams being introduced, the CPU% needed to offload video to a GPU is more than the CPU% savings seen by offloading to a GPU. Especially after about 12 cameras, the CPU goes up by using hardware acceleration.

You now only would use a graphics card if you are using Deepstack and use the GPU to process the DeepStack to improve processing time over the DeepStack CPU version, but do not use it for hardware acceleration within BI.

For kicks and testing, I disabled and deleted the internal graphics card from hardware manager and the CPU didn't go up....

 

[DurandClan]

checking....I think I went the path of the nVidia card because of the AMD processor. When you say 'sub-stream', you're not talking about the crappy, quality secondary camera stream, are you?

 

[wittaj]

Yes the substream is the "crappy" quality but saves a ton of resources.

NVRs have been using them for years to process the video.

Believe me if we thought using the substreams impacted any ability, we wouldn't use them.

You can up the bit rate quite a bit as well.

So it sounds like you are not using substreams and are using hardware acceleration- that is the issue. Either use subs and turn off hardware acceleration or rollback to the version before Deepstack was introduced.

 

[Rob2020]

Just adding a point of reference;

Ryzen 2700, 16GB Ram, have 5 Dahua cams recording 24/7/365, have a GT1650Super that I do not use. Tried it both ways (CPU With & W/O GPU) and BI runs perfect without adding in the GPU. I do run sub-streams.

Usage runs around 10% on my CPU.

 

[DurandClan]

Yes the substream is the "crappy" quality but saves a ton of resources.

NVRs have been using them for years to process the video.

Believe me if we thought using the substreams impacted any ability, we wouldn't use them.

You can up the bit rate quite a bit as well.

So it sounds like you are not using substreams and are using hardware acceleration- that is the issue. Either use subs and turn off hardware acceleration or rollback to the version before Deepstack was introduced.

It was working fine before...what changed? I have a system dedicated to the cameras so I could care less if the CPU is 90%....that is what it is there for. The only reason I went with the nVidia route was the make sure the CPU was there for other tasks, i.e. export operations, and handling other I/O tasks. If I wanted crappy video I could have bought a legacy Swann system when the video was interlaced. The investment was to make sure I had better quality cameras in certain places to have better control overall. I have a couple of Xeon servers at home that I could run the cameras in a virtual machine if the quality didn't matter. So hardware acceleration is an option but I'm hearing it just simply shouldn't be used. I feel like I'm taking two steps back here.

 

[wittaj]

What changed was when Deepstack was introduced, something in the coding changed to allow AI to use the GPU and for many it then created problems with using hardware acceleration.

It has hit everyone differently at different updates and some still don't have a problem. But turning it off has so far fixed a crashing issue with everyone that comes here with this problem...

You realize you are losing nothing by using substreams and gaining everything. Recorded video is still mainstream. Nobody here would use substreams if it degraded the experience or resulted in missed triggers...

It only "impacts" you on mulitcamera view, but you can always up the bitrate and make it not noticeable. When you solo a camera it goes to mainstream.

I tested it once and if a plate reader can read plates with D1 resolution, then it is fine for viewing how ever many cameras you have in a multiview.

Also keep in mind you don't have to be on the latest update. Many here, including myself are running an older stable update. If you aren't using direct-to-wire or senseAI then rollback to where your system is stable.

I encourage you to read up on substreams and realize that NVRs have used them for years for the same thing.

Sub Stream Guide

.substreamGuide td { border: 1px solid #000000; } .substreamGuide a { color: #0080ff; } .substreamGuide code.inline { margin: 0; background: #f6f6f6; border: 1px solid #e0e0e0; border-radius: 0px; padding-top: 1px; padding-right: 3px; padding-bottom: 1px...

ipcamtalk.com

 

[DurandClan]

What changed was when Deepstack was introduced, something in the coding changed to allow AI to use the GPU and for many it then created problems with using hardware acceleration.

It has hit everyone differently at different updates and some still don't have a problem. But turning it off has so far fixed a crashing issue with everyone that comes here with this problem...

You realize you are losing nothing by using substreams and gaining everything. Recorded video is still mainstream. Nobody here would use substreams if it degraded the experience or resulted in missed triggers...

It only "impacts" you on mulitcamera view, but you can always up the bitrate and make it not noticeable. When you solo a camera it goes to mainstream.

I tested it once and if a plate reader can read plates with D1 resolution, then it is fine for viewing how ever many cameras you have in a multiview.

Also keep in mind you don't have to be on the latest update. Many here, including myself are running an older stable update. If you aren't using direct-to-wire or senseAI then rollback to where your system is stable.

I encourage you to read up on substreams and realize that NVRs have used them for years for the same thing.

Sub Stream Guide

.substreamGuide td { border: 1px solid #000000; } .substreamGuide a { color: #0080ff; } .substreamGuide code.inline { margin: 0; background: #f6f6f6; border: 1px solid #e0e0e0; border-radius: 0px; padding-top: 1px; padding-right: 3px; padding-bottom: 1px...

ipcamtalk.com

I think I was misunderstanding....you are referring to the 'viewing' and not the recording of the cameras? So the recorded video is defined in the video configuration and the higher quality is still recorded to disk but the lower quality is used for the UI only. Honestly had no idea. I did notice the variety of UI3 quality settings available. I'll look at the guide you sent and go from there to see if I can benefit accordingly.

 

[wittaj]

That is correct. It is substreams for multicam live view and mainstream for solo live view.

You have several options for recording.

You can record 24/7 mainstream like you are now OR you can tell it to record substreams until triggered and then record mainstream for the event. It allows you get more storage that way.

But using substreams will save a lot of CPU%.

Let us know if you have any questions!

 

[DurandClan]

That is correct. It is substreams for multicam live view and mainstream for solo live view.

You have several options for recording.

You can record 24/7 mainstream like you are now OR you can tell it to record substreams until triggered and then record mainstream for the event. It allows you get more storage that way.

But using substreams will save a lot of CPU%.

Let us know if you have any questions!

ok....I think I have it all set correctly now...adjusted everything to match your referenced doc, went into each camera's vendor interface and made sure both streams were configured properly and aligned settings in BI to match for each stream. Removed the hardware acceleration from each camera. Right now I have about 10 Amcrest cameras ranging from 3MP to 8MP and a couple of ReoLink 5MP and I think after these changes the Ryzen hopped up about 5% and the GeForce card down to zero obviously. Is there any harm in using the GeForce cores in the export process for flagged videos and manual export jobs as long as the card is in there and idle?

 

[wittaj]

Awesome - did you tell BI to use the substreams? The CPU should go down, not up when using the substreams. You need to go into the BI camera setting and do the pulldown for the substream.



I have not heard of anyone here showing problems with using the GPU for anything else except for hardware acceleration, so you should be ok on the exports.

Now take a look at the Reolinks as some of the models don't play well with BI and substreams, so make sure it looks like they are processing and showing live. You can do a search here and see if the model you have is one that folks have reported issues with the substream.

You can confirm you are using substreams as there should be numbers in the respective columns under the BI camera status page:

 

[DurandClan]

Awesome - did you tell BI to use the substreams? The CPU should go down, not up when using the substreams. You need to go into the BI camera setting and do the pulldown for the substream.

View attachment 138138

I have not heard of anyone here showing problems with using the GPU for anything else except for hardware acceleration, so you should be ok on the exports.

Now take a look at the Reolinks as some of the models don't play well with BI and substreams, so make sure it looks like they are processing and showing live. You can do a search here and see if the model you have is one that folks have reported issues with the substream.

You can confirm you are using substreams as there should be numbers in the respective columns under the BI camera status page:

View attachment 138137

Yeah...CPU didn't change much but the RAM went down profusely. Hardware acceleration is completely turned off and every camera does have a working substream so I believe this is now optimal...not to mention BI has not crashed since.