BlueIris causing hard lockups on new server (Solved)

[Steve Staples]

OK, I have a REALLY weird one. I have installed BI on many different machines, with all different cameras... at my work, we recently switched servers, from an old Dell 2950 DualCore Dual Proc Xeon, with 16gb ram, to a new computer...

SuperMicro C7Z87 motherboard
Intel i7-4790K
32GB Corsair Vengeance ram
1x 512G Samsung 850 Pro (os)
1x 2TB WD Red Pro ("new" drive)
3x 6TB WD Red Pro ("storage" drives)

The server works great, no issues (well, not anymore). I installed Win7Pro 64bit on the machine, and well, windows updates suck, it ran for 18hrs without getting/downloading any updates... so I went ahead, and installed BI anyways, and that's when the issues started. The server will randomly just "Freeze up", requiring me to physically reset it. There is no network activity, no console, no keyboard mouse, but the video is there (completly frozen)... it doesnt' blue screen.

Now, initially, i htought maybe is was a harddrive issue, so I swapped all the SATA cables, still happened. I swapped the OS drive for a spare 2TB platter drive, still happened. I unhooked all the drives, except the OS, still happened. I swapped the ram around, removing all but 8gb stick, swapped all 4 as single only, still happened. I installed Win10Pro 64, using 1 drive ata a time... still happened.

The only thing that causes it to lock up, is when BlueIris is running. It is a crap shoot as to when it will lock up. I have run computer burnin software from cpubenchmark, testing harddrives, cpu, memory, video, everything. without BI running, everything works perfectly. And, as I just said, as soon as BI starts, it could take 1 minute, it could take 45 minutes, but it will lock up.

The cameras we have 'connected', all worked perfectly fine on the old server (the server was running at 100% constantly though), and on this new server, it runs at 30%. The cameras are:
12 Speco cams (not 100% sure of the model number right now) @ 10fps 1080p (capable of 30fps)
2 Samsung cams (again, not 100% sure of the model) @ 10fps 1080p (caplable of 30fps)
3 "other cameras", don't know what they are, but they are just flakey, and are being replaced.

Now, all these cameras, and they're set up worked perfectly fine on the old server, but on the new one, I can't run BI without it dying.

I have been at this new server for jsut about 3 days now, and I can't figure out what the issue is. There is nothing in the Windows Event Viewer, that shows something happened. There is nothing anywhere that I can see, for any log, as to what is going on... I am hoping tat someone here can help me figure this one out.

HELP!

 

[fenderman]

Re: BlueIris causing hard lockups on new server

Did you install the windows updates?

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

I have uninstalled the latest BI, and installed 4.2.9.1 I had from a while ago (so weird going back a few versions). I haven't put in my "registration code", and I am slowly adding 1 camera at a time.

I guess time will tell what happens with this. Going to add 1 camera every 2 hours, since I haven't had the computer last longer than 1 hour with BI running.

Now, doing this, when do you think I should enter my registration code? should I do it now? do you think it will matter?

 

[fenderman]

Re: BlueIris causing hard lockups on new server

Did you install the windows updates? Direct to disc won't work in demo mode..

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Did you install the windows updates?

With the Windows7 install, it never installed them. With the Windows10 install, it installed them right away, without issue.

I am running it on Win10Pro64 right now... the server was up for 16 hours without any issues though (i restarted it about 40 minutes ago, to apply some changes), and I just added a camera at 10:00am EST (10 minutes ago)

- - - Updated - - -

Did you install the windows updates? Direct to disc won't work in demo mode..

Ok, then I guess I will add my registration code right away (now) then

 

[fenderman]

Re: BlueIris causing hard lockups on new server

ensure that all drivers are updated as well as bios..

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

ensure that all drivers are updated as well as bios..

Everything is updated. I was on the phone with supermicro, thinking that this could have been an issue with their motherboard, and everything pertaining to the motherboard, is up to date.

 

[fenderman]

Re: BlueIris causing hard lockups on new server

Everything is updated. I was on the phone with supermicro, thinking that this could have been an issue with their motherboard, and everything pertaining to the motherboard, is up to date.

did you do a clean install of windows 10?

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

did you do a clean install of windows 10?

Yes, but I removed the SSD from teh equation, when I thought that maybe it was causing the issue. Fresh install on a 2TB WD Red drive as the OS, and it is writing "direct to disk" on a separate 2TB WD Red drive, and then I have it moving from that, onto a single 6TB WD Red drive when the drive is at 1.5TB or 1 day.

That's as far as I have right now. I need to verify that this is going to fix it... once I get all the cameras hooked up (again, adding 1 more every 2 hours until they are full), then I will do the update on the BI software... that seemed to be the common denominator was the newest version of BI.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Well, after it being up for 4+ hours, I had added 7 cameras, and it has locked up. Back to the datacenter I go, to see what the issue is.

I had sent an email to support yesterday, and still havent' gotten any reply back yet... I don't know where to go from here now...

 

[bp2008]

Re: BlueIris causing hard lockups on new server

This kind of lockup is almost always hardware or driver related. Hardware problems like this drive me absolutely mad, because they are so damned difficult, and expensive, and time consuming to solve. If you can, try a different power supply and/or run a memory test for about 48 hours. This also sounds like the crash is more likely when the machine is heavily loaded with video streams, so when you swap the power supply I suggest you bring Blue Iris back to its full load for stability testing.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

If you can, try a different power supply and/or run a memory test for about 48 hours. This also sounds like the crash is more likely when the machine is heavily loaded with video streams, so when you swap the power supply I suggest you bring Blue Iris back to its full load for stability testing.

What I've done, is I've replaced the power supply, swapped all 4 sticks around, used just 1 (tying all 4 as a single), replaced all the SATA cables, HDD's in different ports, ran memtest86 for over an hour with all 4, exchanged all the drives, original ssd, another ssd, and a 2td platter drive, disabled the onboard sata ports, and used my JBOD supermicro 8port raid card from my UnRAID server, tried Windows 7, and windows 10, different NIC ports, and a NIC card... the only common item is, is the CPU and MOBO.


I have ran CPUBENCHMARK's burnin software, 3d passmark to see if it was video, and cpubenchmarks graphic benchmarks as well.

Using the burnin, i tested all the drives, 100%, stressed the CPU to 100%, everything has been tested using that software, at 100%, on all peripherals... eveyrthing is a pass.

I can run the computer overnight, without BI running, and it is fine.

As soon as I start BI, between 5 seconds, and 3 hours, the computer locks up. Currently, I have 2 cameras running, and it's been alive for about 3.5 hrs. I had 7 cameras running, for about 3 hrs, when it locked up.

I have currently about 20 more cameras to add to this server... but it's not stable.

I don't know what else to test... or try... with all the tests I've ran, and everything is good, until I run BI, leads me to think it's not hardware, or drivers...

I am willing to try anything... what sucks though, is that the server is at our datacenter, so when it dies, I have to drive there to restart it (since there is no drac)

 

[Q™]

Re: BlueIris causing hard lockups on new server

Bring it home.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Bring it home.

I suppose I could... but, yeah... I am thinking about buying an IP power bar. Not ideal, but it works... maybe

 

[Q™]

Re: BlueIris causing hard lockups on new server

On the upside, at least it's not January in The Great White North.

 

[Git]

Re: BlueIris causing hard lockups on new server

On a fresh install of Win7 with SP1 it can takes hours to finally show all the updates that are available and then start downloading/installing them. (I believe it is because windows has to scan all the files first)

If your want to go back to Win7 - just give it some time and get everything updated before moving on

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

On a fresh install of Win7 with SP1 it can takes hours to finally show all the updates that are available and then start downloading/installing them. (I believe it is because windows has to scan all the files first)

If your want to go back to Win7 - just give it some time and get everything updated before moving on

Yup, after 16 hours, I had to move on... LOL

 

[Git]

Re: BlueIris causing hard lockups on new server

I have had it take overnight before - but I agree, 16 hours is too long, something must be screwed up

https://blog.krissmilne.tech/windows/windows-7/windows-7-stuck-on-checking-for-updates

 

[bp2008]

Re: BlueIris causing hard lockups on new server

Wow you have done it all then, short of replacing the MB and CPU. I hear that Handbrake (a free video re-encoding app) can crash unreliable systems that otherwise appear to be stable. No doubt it is a very similar workload to what Blue Iris is doing. You might try that and see what happens. But it really sounds like you need to replace the CPU and/or motherboard. Or even just check the motherboard's CPU socket to see if there are any bent pins.

Say, you've monitored the CPU temperature while running Blue Iris, right?

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Wow you have done it all then, short of replacing the MB and CPU. I hear that Handbrake (a free video re-encoding app) can crash unreliable systems that otherwise appear to be stable. No doubt it is a very similar workload to what Blue Iris is doing. You might try that and see what happens. But it really sounds like you need to replace the CPU and/or motherboard. Or even just check the motherboard's CPU socket to see if there are any bent pins.

Say, you've monitored the CPU temperature while running Blue Iris, right?

When I was stress testing the cpu, at 100%, and the onboard Intel HD 4600, and the HD's, Network... the temp hit 70*C with the top of the case off (it is a 4U rack mount case). using a STOCK i7 cooler/fan.

I just logged into the server, and it appears to still be working! UPtime is 9hrs currently, and I only have 2 cameras set up. I've attached 2 pics, 1 of resources, and 1 of the temps and voltages... this is typical of the server, it doesn't get hot at all (not to mention, it is in our datacenter, which is a balmy 18*C or so... with good ventilation

Tomorrow, I am going to add another camera... but so far, this is hte longest it has been running with BI running as well. the ONLY change, is that I am using the other NIC port on the mobo... that's it.