Blue Iris computer poof in the middle of the night - any ideas what happened?

[ipcdal]

Woke up this morning, Blue Iris machine totally unresponsive, monitor powered off, had to do a forced reboot, can't figure out what happened.

All my footage just ceased to record at 12:18am last night. Nothing in the Blue Iris log tab about it, in fact, it seemed to have cleared itself when I loaded up Blue Iris (I wish Blue Iris would preserve old log files by default, I did not have "Save to file" checked in the BI log tab!).

I checked out the Windows Event Viewer, and it looks like at 12:19am Windows reported a LiveKernelEvent, but the attached event files don't exist. Obviously something happened...

Also, looking through the Event Viewer, it looks like the computer kept on working all night though on other tasks, at least partially, but like I mentioned, it was unresponsive this morning, and then I did the forced reboot.

Any ideas? Everything is running fine right now.

I wonder if it's a hard drive issue? The drive that was being written to is very old, and I've been meaning swap it out. Maybe the drive did something crazy which caused this odd behavior? I will check that later today.

Also, I've now enabled "Save to file" in the log tab, so if it happens again, maybe BI will give me a clue...

Any thoughts appreciated.

 

[Jim I.]

It would help to post some info about your setup. CPU, Windows version, Blue Iris version, etc.
I had a similar problem when I built a new PC for Blue Iris. I installed Windows 10 Pro and was having random shutdowns. After I updated it to Windows 11 I no longer had any issues. Not sure if it was a video driver issue or what..

 

[ipcdal]

Thanks!

Windows 10 Pro, up to date as of today, BI 5.7.6.8 (currently listed as "latest highly stable" version according to BI update screen), Intel Core i5-8500, 16GB RAM. Nothing special, nothing extra installed except antivirus software, very stable machine, no issues so far other than this event.

No plans to update to Windows 11 until I absolutely have to.

 

[Jim I.]

Maybe a power supply issue? You could install a system stress test program and see if you can recreate the crash.

 

[Perimeter]

Run prime95 torture test with large FFTs for some hours or a day.

 

[ipcdal]

Good suggestions, thank you. I will run some more tests. BTW, I did run a complete Dell diagnostic on it, and the system passed just fine. But a Dell diagnostic is not a torture test obviously.

 

[ipcdal]

BTW, I'm really annoyed at myself for not having the "Save to file" box checked in the log tab. I feel that would have given me some good clues as to what happened. Note to self: ALWAYS check "Save to file" in the log tab when you install BI!

 

[IAmATeaf]

Did the BI service just stop? If it did look at the options for the service and enable the option to automatically start if it crashes and stops.

 

[Mike A.]

BTW, I'm really annoyed at myself for not having the "Save to file" box checked in the log tab. I feel that would have given me some good clues as to what happened. Note to self: ALWAYS check "Save to file" in the log tab when you install BI!

Probably not. It's usually just an expanded version of what you see in the log window. Can help with some things but random Windows stops, not sure it would tell you a lot. I wouldn't worry about it too much if it's just a one-off event. If you have a continuing problem, then it's worth chasing.

 

[OldBobcat]

It would help to post some info about your setup. CPU, Windows version, Blue Iris version, etc.
I had a similar problem when I built a new PC for Blue Iris. I installed Windows 10 Pro and was having random shutdowns. After I updated it to Windows 11 I no longer had any issues. Not sure if it was a video driver issue or what..

I had an issue where my Blue Iris PC was crashing almost every day. Thought it was power-related and bought a replacement as well as added a backup (PC is in a server configuration where two power supplies can be added). Finally figured out that the NVIDIA graphics driver was the culprit. Went back to an earlier version and the crashes stopped.

 

[ipcdal]

Did the BI service just stop? If it did look at the options for the service and enable the option to automatically start if it crashes and stops.

Will check that out, thank you.

 

[ipcdal]

Probably not. It's usually just an expanded version of what you see in the log window. Can help with some things but random Windows stops, not sure it would tell you a lot. I wouldn't worry about it too much if it's just a one-off event. If you have a continuing problem, then it's worth chasing.

Thanks, yeah, this is hopefully a one-off event. First time this ever happened at least... I'll run some more tests in the next day or two, but cross fingers it was indeed just a one-off and we'll see what happens in the future. The uptime has been great on this machine -- has never crashed that I can recall. Last time I rebooted was about a week ago, has been behaving just fine up until last night. No changes to the system, nothing I can see that's odd.

Anyway, I'm not going to stress out about it at this point. It's been fine all day, who knows what the heck happened. Appreciate the response!

 

[ipcdal]

Quick update:

I did further stress tests on the computer CPU/RAM, as suggested, and everything checked out fine. This is consistent with the Dell diagnostics I ran previously which also passed just fine.

However, BI went poof again last night at 11:50pm, exact same scenario as the first time... but this time I had the log files from BI to look at, and coincidentally the last item in the log file was a Delete operation at 11:50pm. There was nothing in the Windows Event Viewer this time though... super strange.

So I think today I'll do a SMART test on the hard drive, and maybe swap hard drives. Maybe some hard drive exception or timeout happened during the delete operation? Who knows. But it's worth looking at.

I also updated to BI 5.7.7.6 just now. Also, BI was already configured to "prevent power management from putting the PC to sleep" but I switched Windows to "high performance" power profile and manually disabled sleep just as an added assurance.

A bit frustrating, but will update again if I find out something else. I'd like to get to the bottom of this.

 

[Perimeter]

Have you checked file structure on drives?

 

[ipcdal]

Have you checked file structure on drives?

Yes, thank you. I forgot to mention that in prior post. All good.

Also just finished SMART check -- health status is Good. Checked via command prompt + CrystalDiskInfo + DiskCheckup

Also just ran Windows System diagnostics -- all good (and hard drive shows as good in that too)

Also just ran SMART short self-test, just completed without error

Currently running SMART extended test... in progress...

 

[mattp]

Maybe a power supply issue? You could install a system stress test program and see if you can recreate the crash.

Power supplies in PCs are generally pretty good. But, it's always good to use a UPS.

Having dealt with this particular issue in the past, it is always good to make sure you get one with AVR. Most any cheap surge protectors will protect you from the + spikes in voltage. But, what can be worse are the - spikes in power. That's where AVR comes in.

 

[ipcdal]

Power supplies in PCs are generally pretty good. But, it's always good to use a UPS.

Having dealt with this particular issue in the past, it is always good to make sure you get one with AVR. Most any cheap surge protectors will protect you from the + spikes in voltage. But, what can be worse are the - spikes in power. That's where AVR comes in.

Yes, agreed, thank you, in this case, the BI computer is attached to a fairly decent UPS system with AVR. It's not a sine-wave UPS though, which would be even better. I have one of those, but that one is connected to a file server/NAS I have in another room. This UPS is decent enough though, been running fine for about a year, never had a problem that I know of. However, you never know, and it's a good reminder, so I'll look more closely at it if this mystery continues. I could move this machine to the other room with my other UPS that is a step up in quality, so it will be easy to eliminate as a potential cause of the problem.

Still waiting on the extended SMART test...

 

[ipcdal]

Extended SMART test finished, passed totally fine...

Summary - all hardware tests passed, plus hardware stress test (CPU + RAM) passed, hard drive tests passed, extended SMART test passed, filesystem checked and passed, Windows diag passed, everything passed. History of stability problems with this computer prior to this strange situation: zero problems. There is nothing else on this machine except antivirus running. It's a stock, clean install of Windows 10 Pro, up to date.

And poof last night at 2:32am, the footage stops. And again, the logs show that BI was doing a delete process at that time.

However this time was different such that BI apparently did keep running some tasks, but it just had stopped recording at 2:32am. But then the system eventually became unresponsive and required a hard reboot this morning.

So the same outcome, but slightly different story getting there.

At this point I guess I can move the machine to the sine-wave UPS and see if that helps, but I think that's a long shot. I've got a bunch of other computers, server/NAS, doing various things and nothing is flaking out anywhere in the house. If anything, this machine has been tested and re-tested now more than any of them.

I think I may wipe the machine and start over with a fresh OS install and see what happens. But that defies logic since I recently just did a fresh OS install.

Or maybe I'll throw in an nVidia graphics card and see if that makes a difference. Right now it's just running off the built-in Intel GPU on the i5-8500. What do you think?

Pretty frustrating.

 

[Tinman]

Make sure you have left enough headroom on your data drive. Run a compact/repair and see if that goes ok. Does your database status look all green ?

 

[mattp]

Have you considered the possibility of ghosts?
Kidding aside, if it's not on the internet, get rid of the anti-virus and as @Tinman said, defrag the hard drive.

Edit to add:
Speaking of defrag, check to see and make sure no tasks are occurring on a schedule at that time...especially, if you have it scheduled to defrag the hard drive, I could see BI choking.

Does anyone know if you can set up a performance capture to see what the throughput is on the drive, along with memory and cpu usage throughout the night?