BlueIris causing hard lockups on new server (Solved)

[SpinnerRow]

Re: BlueIris causing hard lockups on new server

Windows 7 not updating is a Windows bug. If you have an old install ISO, the number of updates needed overwhelms the update database and causes it to sit for ever at downloading. I fixed this problem a while back with a computer at work but I can't remember the update number that fixed it. Google windows update problems or downloading and there are lots of threads related to this problem.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Well, I ran since Friday night at 5pm, til Monday morning at 2:30am, when it "locked up". I had 2 cameras running, til about 5pm sunday, and then I added another camera at that time.

I got to our datacenter this morning, to check, and restart. I ran another Burnin test, FULL 100% on everhting again, and no issues. When I went to do something, it had locked up... BI wasn't running, so I can now rule out BI as being the culprit. So, I am thinking now, it is probably the MOBO and/or the NIC. I am going to go put in a gig card, and disable the onboard NICs and see if that solves it... this has been the BIGGEST, and HARDEST problem I have seen. I has stressed me out to no end

 

[bp2008]

Re: BlueIris causing hard lockups on new server

I still think you should try handbrake. If that program causes the lockup, it should happen faster than with Blue Iris because handbrake will run the CPU to 100%. Of course now that you've replaced nearly everything, there isn't a lot more things to change that require stability testing. LOL

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

I still think you should try handbrake. If that program causes the lockup, it should happen faster than with Blue Iris because handbrake will run the CPU to 100%. Of course now that you've replaced nearly everything, there isn't a lot more things to change that require stability testing. LOL

I have been using this:
http://www.passmark.com/products/bit.htm

I have it set to run video, CPU, memory, HDDs and NIC (currently only the loopback) to 100%.


EDIT:
It can pretty much only be the CPU or MOBO, as I can't imagine all 4 sticks of ram are bad (I've tried each one individually), and the HDD's I've run multiple different ones. I suspect it is the mobo, or some aspect of it.

 

[Q™]

Re: BlueIris causing hard lockups on new server

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

So, I am 99.9% sure it is the motherboard at this point. Dealing with Supermicro now

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

OK... after a LOT of testing, I can now verify that it is the GPU.

Now, it is an i7-4790k processor, and a SuperMicro C7Z87 motherboard. It has Intel HD4600 video, and from what I understand, the onboard video gets signal from the CPU... but which is the issue?

Anyone have any ideas?

AND lastly... the "spec" on te BI website suggests to use the quick sync i7, and an nvidea gpu for local viewing... should I get a video card as well? would a GT 710 suffice?

Thanks!

 

[bp2008]

Re: BlueIris causing hard lockups on new server

AND lastly... the "spec" on te BI website suggests to use the quick sync i7, and an nvidea gpu for local viewing... should I get a video card as well? would a GT 710 suffice?

No, as it is almost always just a waste of power. An nvidia GPU did help reduce my CPU usage but ONLY when the Blue Iris application is open and displaying video, particularly at high resolutions and high frame rates. Otherwise it was completely unused, including when running BI as a service with the GUI closed. I used a GTX 950 by the way, since I wanted an HDMI 2.0 output and that was the cheapest card at the time with HDMI 2.0.

In your case, getting an nvidia card and disabling the integrated GPU might solve the freezing problem, but it would prevent you from using hardware accelerated video decoding so it is not an ideal solution. The Intel GPU is built into the CPU, by the way, but that doesn't mean the GPU itself is the cause of your freeze. It could be the motherboard too...

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Ok, the BI machine is at our Data Center, so we RDP in, or use the Web/Android Interface. Very rarely would we be hooking up a monitor to view it at the DC. So, don't worry about the video card then...

Now, to figure out who wants the RMA, or both.

I can reliably freeze up the server running the "Intel XTU" stress test on the GPU, and within a 10 minute test, it will fail.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

WELL! Here is the conclusion (for today... and hopefully from now on) of my saga!!!!

After RMA'ing the Mobo and the CPU (not thinking which one it could be... lets just do both), and it coming in, and still doing the SAME thing... I was about ready to give up, and go live up north, in solitude. I have been at this for 2 weeks now, and this morning, I took a colleague to the data center, and had him look it over... and, he was stumped as well.

We tried disabling the onboard graphics, and it "fixed" the issue, but only temporarily... because it locked up again. We tried numerous other things, and finally... we got it.

Changing the Ring Voltage Offset, and the UnCORE voltage offset, made the system "stable". The Intel XTU software ran a stress test on the GPU for 30 minutes... and no lockups (previously, it would lock up anywhere from 10 seconds, to 10 mintues). So, I said lets set each of these values back to 0, one at a time, and see which one is the culprit. Turns out, the Ring Voltage Offset of +75mv is the key. When we set it to 0, it locks up... having it at 75, runs fine (currently, I am 2hrs into a 4hr gpu stress test).

NOW... is +75 the magic number? I don't know... could it be lowered? maybe... could setting it higher be better? who knows at the moment. I have sent SuperMicro an email, telling them of my findings... just waiting to hear back from them now.

So, for now... I am waiting for the 4hr test to finish, and then I am going to go replace the stock CPU cooler, with an aftermarket one... since the CPU/GPU temps approach 70-80*C at peak... and I know the stock cooler is "meh" at best.

Hopefully I can start adding back the cameras, and putting the HDD's back in... and this issue will be behind me... cuz man, it has been a ROUGH few weeks.

Sorry to blame BI, but at the time, it was the only thing that would put a load on the computer, and lock it up at that time... but, it is not BI!

 

[Q™]

Re: BlueIris causing hard lockups on new server

The old Ring Voltage Offet! Shit! I was gonna suggest that a few weeks ago!

 

[Q™]

Re: BlueIris causing hard lockups on new server

BTW...great work, congratulations...and...

...I was about ready to give up, and go live up north, in solitude...

This was VERY funny.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

BTW...great work, congratulations...and...



This was VERY funny.

It is weird though... even my buddy doesn't understand fully why the "default" settings weren't sufficient? You would think, that it would have been just fine. I mean, it should have been? I am not doing anything special with this... putting the processor on the motherboard should have just "worked".

Oh well. Guess I can unpack now... or, maybe I will wait a little longer... it's only been a few hrs.

 

[bp2008]

Re: BlueIris causing hard lockups on new server

Yes, it should have just worked. Usually you don't have to increase voltages to achieve stability unless you are overclocking. When you increase the clock rate you usually have to increase voltage to make the system remain stable. Higher voltage typically grants higher stability at the cost of higher power consumption, more heat, and potentially shortened CPU life. My guess is the motherboard isn't quite as compatible with the CPU as it should be.

 

[Q™]

Re: BlueIris causing hard lockups on new server

Effective cooling solutions are key to stable overclocking!

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Effective cooling solutions are key to stable overclocking!

View attachment 13104

BUT I AM NOT TRYING TO OVERCLOCK IT! lol (and, I have actually used a 20" box fan on my server, with cardboard box cut up and ducting into the front of the 5U chassis. don't ask. but it wasn't as effective as when I finally replaced the internal fans

 

[Q™]

Re: BlueIris causing hard lockups on new server

BUT I AM NOT TRYING TO OVERCLOCK IT!...

I know...but I could not resist making a joke when Brian mentioned overclocking. Sorry. You can have your topic back now.

 

[pschmehl]

Re: BlueIris causing hard lockups on new server

Kudos to you for sticking with it until you identified the problem. It's a sign of a true technician.

 

[Steve Staples]

Re: BlueIris causing hard lockups on new server

Kudos to you for sticking with it until you identified the problem. It's a sign of a true technician.

Thanks, and more like i HAD to figure it out, my job kinda depended on it... LOL.

It really stresses me out, when there is something so strange, that doens't make sense... I am a PHP/MySQL programmer by "trade", so "weird issues" is kinda common place for me.

I Currently have 10 cameras running on the server, continusouly recording... so, hopefully it lasts for 2 more hours... since that is hte longest it ran previously before it died.

<fingers crossed>

 

[pschmehl]

Re: BlueIris causing hard lockups on new server

Sure, but you could have easily punted and told your boss you needed to bring in a specialist. Instead, you stuck it out and got it working. That's commendable.