Help - BI Lag/hanging/crash, and remove viewers hanging - Any suggestions?

[masterdk]

The problem:
I've had this problem since BI3, and I've moved into BI4x64 bit with still same results.
When BI turns on, all everything is functional for the first 5-10minutes. If there is any kind of motion, or long video stream, the BI just hangs for a long time. The remote viewers (IE/ff/chrome web browser, android) all have video that hang. I'm sure the application is hanging and not sending video to the remote viewers. If we wait ~15-20min, the video will jump forward, work a bit, and hang again.

Back in BI3, the hanging seems to happen when BI3 eats up ~1.5GB of ram. It would never go past 25% CPU usage. Understandable for ram, as it was 32bit.

On BI4x64, BI4 constant hogs around 2.5-4GB of ram, and CPU seems to hover around 50%. Once in a while it spikes to 100%, but it drops real quick.

For the system, there's still plenty of extra ram, cpu, and disk IO available. Why is the remote viewers and BI application hanging? How can I troubleshoot this? Do I have too many cameras connected? This issue is less noticeable when I disable a lot of cameras .


The Setup details:
-The set-up is done on a windows 2008r2 server OS.
-All cameras are continuous recording, and files are broken base on motion. I do this, as i don't trust the record on motion to work perfectly.
I- have the "Pre-Tirgger fram buffer size" unchecked.
I-Recording to AVI format, using H264, Quality 50%, 1024 bit rate, 30 GOP, 0 Max B between P frames, preset: superfast. I have all the files re-encoded to AVI so that we can play back the files remotely on another PC using VLC/Mediaplayer/etc. I also do this is to save on disk space
-All cameras(a mix of foscam, dahua, axis) running ~13FPS with these resolutions:
10 x 640x480
1 x 1280x800
1 x 1280x960
4 x 1280x720
9 x 1920x1080
3 x 1440 x 1376

This is on a gigabit network, running on a supermicro X9DRi-LN4F+ with DUAL socket Xeon E5-2650v2 2.6Ghz(8 core), 16GB Ram, SSD for OS/BI (Images/cache/ db), 2x raid arrays(each composed of 8x4TB drives) for video storage.

Why can't my monster server handle this??

 

[fenderman]

@masterdk Welcome to the forum.
First start with the basics...
Have you determined whether blue iris hangs on its own when not remote viewing? This will eliminate any issue with the webserver..
Record to BVR
Make sure your antivirus excludes the blueiris.exe files as explained in the help file.
Do you have any other software running on the server?
Have you emailed support?

 

[masterdk]

The issue is with the blue iris software hanging. Any remote view will hang when the console is frozen.
There is no AV or any other software running on the server. It's a dedicated DVR machine.
If i record to BVR, how do i view the files remotely. I'm trying to play back the files without RDPing into the DVR machine.

I did contact support, he suggested i reach out to the forum.

 

[fenderman]

The issue is with the blue iris software hanging. Any remote view will hang when the console is frozen.
There is no AV or any other software running on the server. It's a dedicated DVR machine.
If i record to BVR, how do i view the files remotely. I'm trying to play back the files without RDPing into the DVR machine.

I did contact support, he suggested i reach out to the forum.

You can simply use the webserver to view files remotely...just do it as a test regardless.

 

[bp2008]

How sure are you that the RAID is not a bottleneck? 28x 1 Mbps streams may not be a lot of bandwidth, but each stream will be recording to a different physical location on disk which means all 8 drives in the array are doing a lot of seeking. It would only take one underperforming drive to bog down the entire RAID array.

By the way, CPU occasionally spiking from 50% to 100% sounds very suspicious to me. Is it blueiris.exe doing the spiking or is it something else?

 

[masterdk]

The CPU Occasionally spikes from 50 to 100% when i'm trying to look at it. I'm not sure what the blueiris.exe is trying to do. I also have the blueiris running as a service.

For the HD, its technically 2 arrays, so its 14 x 1mbp (i think its more like 512ish kbit) streams hit a raid of 8 drives. This is a real raid controller (LSI 2108) with a 512MB ram buffer to do quick write back. I know there's no IO contention, as i can do other disk io activity on the arrays with no problem.

 

[bp2008]

Well lets see... 50% should be 100% of one of your CPUs. Is the load being split evenly across both CPUs or is it maxing out one and not using the other at all?

 

[masterdk]

The load is being split across evenly on all 32 threads in task manager. I think i can correlate the CPU spikes now a little further. The CPU seems be steady at 50%. When its being opened remotely via web browser, it spikes up to 70-100%, but hovers mainly around 70%. Once the browser is closed for like 10min, it drops back to 50%

Recording into BVR, and using the web to playback remotely will be a pain for me. Web is pretty difficult to navigate the files. There's hundreds of files/clips, as each camera clip is broken base on motion. If i use media player classic to play back the AVI's, i can queue them up, and quickly "skim" the video with my keyboard. I can also jump to a specific time of day in the past without hitting the << >> in the browser to look for the video by looking at the file dates.

Anybody know why BVR is so special? Is there such a thing as a BVR codec for media player of VLC ? If there was, i switch to BVR format in a heartbeat.

 

[paarlberg]

AVI has a 1GB limit, BVR allows viewing while recording and doesn't have the limit. You can set a limit for each BVR file based on size or record time.

Could it be that you are hitting the 1GB limit on multiple cameras and it is creating new files causing a surge in IO? With that many cameras and the amount of data, it could be a tweak is needed on the controller.

 

[fenderman]

Try BVR for testing purposes regardless of whether you will stick with it. A paarlberg says BVR allows viewing during recording. It is also more efficient than AVI. How are you even viewing the AVI files without RDPing into the machine? Shared folder?

 

[bp2008]

BVR is proprietary and can't be played in anything but Blue Iris.

About how many clips/alerts do you have? I have noticed that when you have very many clips (many tens or hundreds of thousands of them) then you can get poor performance of the web server and lots of CPU usage. That has only happened to me on one occasion and I fixed it by moving all my old clips out of the folders that BI put them in, and then after a database rebuild BI ran fine again.

 

[masterdk]

For alerts, i set the limit to be 9GB or 2 days worth. But interesting enough, all folders is empty for alerts. I then went under each camera and the "Create an Alert Image when Triggered" is unchecked.

When i'm recording to AVI, am i not suppose to be able to see real time? Because right now i see real time. The only problem(not for me) is i can't play back the last "real" time until another clip is created.

 

[fenderman]

For alerts, i set the limit to be 9GB or 2 days worth. But interesting enough, all folders is empty for alerts. I then went under each camera and the "Create an Alert Image when Triggered" is unchecked.

When i'm recording to AVI, am i not suppose to be able to see real time? Because right now i see real time. The only problem(not for me) is i can't play back the last "real" time until another clip is created.

You can always see live video in realtime...differnece is BVR will let you playback the clip while its recording. This is particulaly important when you have long combine and cut times.
Have you tested with bvr? you have nothing to lose if your machine is freezing up...try it..just for trouble shooting purposes.

 

[masterdk]

I have not tried BVR yet. It's going to take me some effort to change all the cameras to bvr recording. I'll give that a try this weekend and update later.

 

[masterdk]

I changed the format to BVR, and couldn't reproduce the hang. I swapped it back to avi, and couldn't reproduce the issue. I'm going to wait a bit, and see if the issue occurs again. This is a bit odd.

 

[masterdk]

The system started to lag again, and I seemed to see the pattern now. When there's a lot of camera activity, the lag will slowly start happening if you keep watching the stream. There's still plenty of CPU & Ram available. I Swapped it over to BVR again, and it still continues to lag after a while. Quick solution is to restart BI and lag goes away for a while.

 

[fenderman]

The system started to lag again, and I seemed to see the pattern now. When there's a lot of camera activity, the lag will slowly start happening if you keep watching the stream. There's still plenty of CPU & Ram available. I Swapped it over to BVR again, and it still continues to lag after a while. Quick solution is to restart BI and lag goes away for a while.

I guess its an improvement over the freezing you had...
What do you mean by lag?
Have you emailed support?
Have you tried a clean install of the OS?

 

[masterdk]

When i say lag, it means the console UI just super slow. It takes a couple of minutes to respond for each click and the web gui takes couple of minutes to respond.
Yes, i did email support. And no, i did not try a re-installing the OS. This is a plan, clean OS already, with nothing but blueiris. I don't have other software installed.

 

[fenderman]

When i say lag, it means the console UI just super slow. It takes a couple of minutes to respond for each click and the web gui takes couple of minutes to respond.
Yes, i did email support. And no, i did not try a re-installing the OS. This is a plan, clean OS already, with nothing but blueiris. I don't have other software installed.

I believe the other time I saw someone with this issue it was running a server os as well..cant remember where that post is...you might want to try running it on 7/8...

 

[masterdk]

Out of curiosity, does anybody else on the forum run as many high resolution cameras with 24x7 recording?

I did a little bit of experimenting, and i found that if i run multiple VM's with BI processes, the overall system does not lag or hang. I'm tempted to run a bunch of VM's and split the cameras up, but that means I'll have to buy some more BI licenses and have a more complex storage/network setup. To me, this sounds like I'm hitting a BI limitation somewhere.