A Plead for Help With Random App (BI) Crashing

[dbrannon79]

Hello all,

I have been plagued by BI randomly crashing on me for quite a good while. in my troubleshooting and talking here in the forums in the past it was advised that I needed to upgrade my old PC from an Intel 3 gen to something newer. well I finally replaced the PC to a intel 6gen I came across to use. it's a Dell Optiplex 7040 I am using now. I just got things setup last night installed a fresh copy of windows 10, disabled all updates, Installed BI (latest version) I could not find a full installer for 5.7.7.3 which I was on prior. Install BI tools and loaded the configuration for the cameras into BI and settings into BI tools. Also renewed my license since I had to upgrade to a newer stable version in the process.

at first everything seemed to be operating fine and with lower CPU usage with the newer system, but as I left it run over night, this morning I received notifications from BI tools that the app has crashed several times and continues to do so seemingly at random.

one thing I have noticed (on the older PC) was when it crashed I was starting to get alerts constantly on multiple cameras like in a thunderstorm my cameras would go crazy with alerts and the app would constantly crash.

When the app crashes it corrupts recordings in progress and has corrupted the BI database in the past. it can get very annoying at times.

anyhow I am pleading for someone to help me go through all of my setting in the app and in each camera, Help me go though BI and windows logs and try to isolate any possible issues that could be causing this crashing.

 

[dbrannon79]

Here is a screenshot of the crash notifications I have received since implementing the new to me PC last night. I can post some logs for some of these time stamps this afternoon after work.

 

[SouthernYankee]

please provide
1) the BI log file at the time of the crash.

 

[dbrannon79]

please provide
1) the BI log file at the time of the crash.

Thank you in advance. I will post the logs after work (around 6:30pm tonight

 

[dbrannon79]

OK, Here are the logs from BI and some info on the last app crash from windows event viewer. the time stamp in the BI logs for the last crash is 5:03:17.363 PM. One thing I did notice is there are several logs for AI not responding during the night. I need to look at that since the AI is running on my PowerEdge server not directly on the BI machine. I've had it this way due to the old PC I was using for BI couldn't handle the AI and would overload the CPU. Now that I have gone from a 3rd gen intel to a 6th gen, maybe I can try using it directly.

A little more info: I have all cameras set up to record 24/7 and will move footage over to my NAS after 5 days. I may need to trim back on this or install a larger drive in the new BI PC now that the chassis has room.

There event files is a screenshot of the error in event viewer as well as the same log in a text file. I looked at each error in event viewer and they are all showing the same info. while I was pulling this info I looked at my CPU usage as BI is running with the event viewer open while using remote desktop (BI PC is headless) and my CPU is running around 40% usage.

Hope this all helps.

 

[nmbgeek]

You mentioned that you are running headless. Do you have an HDMI/DP dummy plug connected? If not I would highly recommend one as sometimes GPU tasks need to think a display is detected.

Edit: also did you happen to install Dell Command Update and make sure all of the latest Dell Drivers and BIOS updates were installed?

 

[dbrannon79]

The PC has two DP connections available, should I connect a monitor to the PC? it's located inside a server rack but there is room for a monitor if needed. Not sure what a dummy plug is.

 

[nmbgeek]

The PC has two DP connections available, should I connect a monitor to the PC? it's located inside a server rack but there is room for a monitor if needed. Not sure what a dummy plug is.

If you have a spare monitor I would leave it connected and see if you notice improvements. If you do see improvement and you don't want to be powering a monitor 24/7 you can use a dummy plug which essentially emulates a screen being connected. Some consumer motherboards will essentially all but disable the iGPU (Intel Quicksync) if there is no display connected. Sometimes this can be overridden in BIOS but a dummy plug just makes the PC think a screen is connected.

 

[dbrannon79]

Well I ran into a show stopping dead end with the new PC I setup. I failed to realize the windows install failed to activate, apparently even though it shipped from dell with windows 10 pro it would not activate no matter what I did. So my old PC is now back in service running BI same setup just with a dell optiplex 7010 until I can ether get a genuine license or figure out why it's not activating. Not sure where I can find one for wind 10 pro since they are dropping support in a year.

I will monitor the old PC and see if / when it crashes and post some logs but looking into it's logs in the past it has the same readout as the new one.

as for what you were saying @nmbgeek this PC does have a monitor connected but is VGA not DP. I can swap out the monitor for one with DP as the PC has a DP connection if this will help any. all video acceleration in the BI settings are disabled for this PC as well as all cameras are set to H.264 other than two of which one is a USB to video adapter running to an old CCTV camera and the other is the same only connected to another PC in the house that runs yawcam to broadcast to the BI PC via lan. all others are IP cameras of various makes. I can post details on each if needed.

 

[nmbgeek]

Well I ran into a show stopping dead end with the new PC I setup. I failed to realize the windows install failed to activate, apparently even though it shipped from dell with windows 10 pro it would not activate no matter what I did. So my old PC is now back in service running BI same setup just with a dell optiplex 7010 until I can ether get a genuine license or figure out why it's not activating. Not sure where I can find one for wind 10 pro since they are dropping support in a year.

I will monitor the old PC and see if / when it crashes and post some logs but looking into it's logs in the past it has the same readout as the new one.

as for what you were saying @nmbgeek this PC does have a monitor connected but is VGA not DP. I can swap out the monitor for one with DP as the PC has a DP connection if this will help any. all video acceleration in the BI settings are disabled for this PC as well as all cameras are set to H.264 other than two of which one is a USB to video adapter running to an old CCTV camera and the other is the same only connected to another PC in the house that runs yawcam to broadcast to the BI PC via lan. all others are IP cameras of various makes. I can post details on each if needed.

If there is a monitor connected and turned on then that should be fine. You had just mentioned you were running it headless which means no display.

Regarding the Windows activation, not being activated isn't an actual problem. There is a difference between being licensed and activated. Your windows can be licensed and not activated and still be legitimate. I believe not being activated may limit a few customization things but it isn't illegal to run it unactivated.

Cameras connected shouldn't have any affect on your PC stability either unless you just had way too many connected and it was causing instability due to CPU usage. You mentioned video acceleration is disabled but you really want the Quick Sync helping with the video acceleration to reduce CPU usage. If it is 6th gen it should work fine.

When you did the fresh Windows 10 install did you download Dell Command Update and install the latest drivers and bios updates?

 

[nmbgeek]

Well I ran into a show stopping dead end with the new PC I setup. I failed to realize the windows install failed to activate, apparently even though it shipped from dell with windows 10 pro it would not activate no matter what I did.

Also did you do the Windows 10 download directly from Microsoft using the Media Creation Tool? You don't want to use a version from torrent or third party that could have been altered.

 

[wittaj]

You mentioned video acceleration is disabled but you really want the Quick Sync helping with the video acceleration to reduce CPU usage. If it is 6th gen it should work fine.

Around the time AI was introduced in BI, many here had their system become unstable with hardware acceleration (Quick Sync) on (even if not using DeepStack or CodeProject). Some have also been fine. I started to see errors when I was using hardware acceleration several updates into when AI was added.

This hits everyone at a different point. Some had their system go wonky immediately, some it was after a specific update, and some still don't have a problem, but the trend is showing running hardware acceleration will result in a problem at some point.

However, with substreams being introduced, the CPU% needed to offload video to a GPU (internal or external) is more than the CPU% savings seen by offloading to a GPU. Especially after about 12 cameras, the CPU goes up by using hardware acceleration. The wiki points this out as well.

My CPU % went down by not using hardware acceleration.

Here is a recent thread where someone turned off hardware acceleration based on my post and their CPU dropped 10-15% and BI became stable.

A GPU is only needed if you plan to do a lot of CodeProject, which would be insane as most find the AI in the cameras to be more than enough.

But if you use HA, use plain intel and not the variants.

Some still don't have a problem, but eventually it may result in a problem.

Here is a sampling of recent threads that turning off HA fixed the issues they were having....

No hardware acceleration with subs?


Hardware decoding just increases GPU usage?


Can't enable HA on one camera + high Bitrate

 

[dbrannon79]

When you did the fresh Windows 10 install did you download Dell Command Update and install the latest drivers and bios updates?

Yes, all the drivers were updated and I used an old copy of win 10 I had saved from the MS site prior to most recent updates.

I have the optiplex 7040 out on my table so I can work with it figuring out windows activation. I picked it up after it was retired from a business that ran win 10 enterprise so my guess is they never used it with the pre-installed OS and imaged it with their own. looking on dell's site it did ship with 10 pro but I have no documentation on it for licensing. I'm sure I will need to purchase one to use it otherwise deal with the watermark on the desktop, not that I need to worry about that though. the only time I will see that is when I remote in to it.

 

[dbrannon79]

as for the hardware acceleration, on the older 7010 I've been running since 2021 the intel 3rd gen cpu didn't support it and when enabled in any one camera BI would go into a continuous crash loop with BI tools running. Once I get the newer 7040 6th gen one back up I can try enabling it and see what happens

Funny thing is on the older PC when I am remoted in, the CPU usage is around 20% until cameras start getting triggered and it sends out alerts. for the newer PC for the short time I had it running, just being remoted in with BI running it was showing around 40% usage. maybe the difference is the newer PC only has 8gb ram where the old PC has 16gb.

I will post the physical specs on both PC's later tonight after work. I seem to recall the old one is a i5-34xx something to that effect. the new one I think is a i5-6400

 

[dbrannon79]

My old PC is a i5-3570 cpu. I trimmed back on how long to keep recordings to only 5 days then delete so it's not moving anything to my nas storage for now. it's staying in service and the new one is going back since windows 10 won't activate with the embedded key in the bios. It keeps telling me I need to purchase a legitimate license, gives my the dreaded watermark on the desktop. Maybe it will be a good Linux box for someone or something.

in the meantime maybe you all can help cure the random crashing on this older PC. I will post logs when BI crashes again.

Added info to my signature that might help.

current PC specs:
Dell Optiplex 7010 SFF
Intel i5-3570
16gb DDR3
Boot SSD - 240g
Footage SSD - 500g
BI version 5.9.5.4 x64
BI tools
Code AI 2.8 (installed on another machine) with modules face processing and YOLO v5.net for objects
All cameras have hardware decoding disabled and set internally for H.264 video with audio, set to record 24/7 with alert triggers
All cameras have both main and sub streams except for the CCTV cameras
Camera list:
4x IMOU Bullet 2C 4MP IP cameras (DAHUA clone cameras, the DAHUA utilities work to setup camera)
1x IMOU Ranger 2C 4MP turret camera (DAHUA clone cameras, the DAHUA utilities work to setup camera)
1x Anpviz 4MP POE bullet camera with audio
1x Swann (older indoor turret) IP camera
1x SRIcam SP006 Generic indoor Turret IP camera
2x Old CCTV analog video cameras using a generic USB video capture device (one directly connected to the BI pc, other connected to separate PC using yawcam broadcasting to BI via LAN)
1x weather radar api shown as a camera set at 2fps frames with local weather radio audio broadcast. (no recording or alert triggers set, just for visual 24/7 radar viewing through web UI)

I also have one PC setup in our living room with the web UI on screen 24/7 viewing all cameras (same PC running Yawcam for one CCTV camera)

I plan to have the old CCTV cameras replaced with Reolink POE door bell cameras at some point.

If there are any settings or other info you might need please let me know and I will post.

 

[dbrannon79]

Had a couple more crashes, looks like the same information.

Here is what I found in event viewer:

Error 9/6/2024 4:50:39 PM Application Error 1000 (100)
Faulting application name: BlueIris.exe, version: 5.9.5.4, time stamp: 0x66bf9889
Faulting module name: ucrtbase.dll, version: 10.0.19041.423, time stamp: 0xccf6a09c
Exception code: 0xc0000409
Fault offset: 0x000000000007284e
Faulting process id: 0x17ec
Faulting application start time: 0x01db009e91fa4f6c
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\windows\System32\ucrtbase.dll
Report Id: 56dadf3f-3d50-4a3d-ae78-29e22bb28393
Faulting package full name:
Faulting package-relative application ID:

Time stamp in BI log is 9/6/2024 4:50:45.296 PM but there are about 4 other crashes logged here.

 

[dbrannon79]

A little update. I went back though all of my camera settings and discovered that two cameras had the hardware acceleration set to "Default" instead of "NO" so I changed them to NO and so far I have not had a single crash.

And a little good news. I have the newer PC with the Intel i5-6500 cpu back in service. I figured out the windows activation issue.

New system specs:
Dell Optiplex 7040 SFF
Intel i5-6500
8gb DDR3
Boot SSD - 120g NVME
Footage HDD - 2tb
BI version 5.9.5.4 x64

I replaced the 500g SSD I had for the recordings with a 2tb sata HDD.

so far it's doing fine. I did run across a log entry that has me puzzled. it shows a file it encountered but I cannot find this file on the drive anywhere.

Here is the log entry.
9/8/2024 12:08:14.788 PM Log Encountered unexpected file Back_Driveway.20240904_160000.2001761.3-129.dat; stopping

This shows up when starting up BI. I have tried rebuilding the database and looking for this file but it's not on the drive that I can find. any ideas?

 

[dbrannon79]

First crash, and this time I see different entry in the logs.

Here is the event viewer log from windows.

Faulting application name: BlueIris.exe, version: 5.9.5.4, time stamp: 0x66bf9889
Faulting module name: KERNELBASE.dll, version: 10.0.19041.423, time stamp: 0xed02c0bc
Exception code: 0xe06d7363
Fault offset: 0x0000000000023e49
Faulting process id: 0x25cc
Faulting application start time: 0x01db0217f18922c0
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\windows\System32\KERNELBASE.dll
Report Id: 919c135a-1454-4559-811e-fa9d43708c73
Faulting package full name:
Faulting package-relative application ID:

and attached BI log file, time stamp is 2:47:15.685 PM in the file

 

[dbrannon79]

Not sure if anyone is still reading this thread, but I noticed today that it seems that BI seems to be crashing when cameras are being alerted or triggered. it's been almost 5 hours since the last crash and there hasn't been any activity or movement around the house to cause them to set off alerts, neither has BI crashed.

I'm doubtful the AI is causing the crashing since it runs on a different machine, BI only sends info via lan to that machine, then receives it back for the confirmed alerts.

I also have confirmed that hardware acceleration (quicksinc) is also not the issue. since now running the new PC, I went in and enabled all cameras for Intel + VPP. looking in the camera status I only see two cameras using the GPU though. GPU never goes over 8% but mostly stays at 0%

 

[dbrannon79]

One other thing I have a question about. I was looking online about the most recent crash log from windows event veiwer (log below) and noticed that ucrtbase.dll is part of Visual C++ Redistributable. I looked at my installations and see several versions installed.

there is a readme text file in the folder for blue iris in program files that states that I need the 2019 version. would it be possible that it's crashing due to all these versions are installed? below is a screenshot of them when I opened add/remove programs in control panel.

Faulting application name: BlueIris.exe, version: 5.9.6.4, time stamp: 0x66d73b67
Faulting module name: ucrtbase.dll, version: 10.0.19041.423, time stamp: 0xccf6a09c
Exception code: 0xc0000409
Fault offset: 0x000000000007284e
Faulting process id: 0x1c88
Faulting application start time: 0x01db023741523fe9
Faulting application path: C:\Program Files\Blue Iris 5\BlueIris.exe
Faulting module path: C:\windows\System32\ucrtbase.dll
Report Id: f5614147-a643-4a23-bf34-73a9fc26fad0
Faulting package full name:
Faulting package-relative application ID: