[FIXED] Cameras falling behind in BI vs in Camera interface

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
I'm running the latest version of BI 5. My cameras in BI are randomly running up to 10 minutes behind the actual camera's web interface. I've tried modifying certain settings in BI and in the cameras themselves but nothing seems to really fix the issue for more than a day or so. I'm constantly getting the notification that BI is slow in the web interface. All the cameras are Dahua 4MP except for 1 offbrand 5MP. (7 total cameras)

Host:
Server 2016 Datacenter
2x Xeon X5670 2.93GHz (12 core 24 thread)
144GB RAM

BI VM:
12 Cores
6GB RAM (73% used) - dynamic RAM is setup, but I also tried dedicating 16GB RAM to the machine with no difference.
500GB drive allocated (RAID 5 array) originally had this going over eSATA to a RAID5 array. Now it's internal SAS 6Gbps.
Gigabit Ethernet from the hosts card to the router.
VM is setup for 100% resources to be utilized.

Network throughput looks to be around 20-50Mbps
CPU Load is around 40%
RAM is ~73%

Has anyone run into something like this before? I would expect the CPU load to be high or the network load to be high to cause such a massive delay in the cameras. If I restart the BI service the issue goes away for a few minutes before it returns again. If I leave it alone it goes away for a bit, then shows up on another, or multiple cameras.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
I'm running the latest version of BI 5. My cameras in BI are randomly running up to 10 minutes behind the actual camera's web interface. I've tried modifying certain settings in BI and in the cameras themselves but nothing seems to really fix the issue for more than a day or so. I'm constantly getting the notification that BI is slow in the web interface. All the cameras are Dahua 4MP except for 1 offbrand 5MP. (7 total cameras)

Host:
Server 2016 Datacenter
2x Xeon X5670 2.93GHz (12 core 24 thread)
144GB RAM

BI VM:
12 Cores
6GB RAM (73% used) - dynamic RAM is setup, but I also tried dedicating 16GB RAM to the machine with no difference.
500GB drive allocated (RAID 5 array) originally had this going over eSATA to a RAID5 array. Now it's internal SAS 6Gbps.
Gigabit Ethernet from the hosts card to the router.
VM is setup for 100% resources to be utilized.

Network throughput looks to be around 20-50Mbps
CPU Load is around 40%
RAM is ~73%

Has anyone run into something like this before? I would expect the CPU load to be high or the network load to be high to cause such a massive delay in the cameras. If I restart the BI service the issue goes away for a few minutes before it returns again. If I leave it alone it goes away for a bit, then shows up on another, or multiple cameras.
The generally happens when there is a network or cpu bottleneck. Is your pc connected to the same switch as the cameras or one coming off that switch? or is it connected to the router?
 

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
It's connected to the same switch but with only about 8 devices connected there I wouldn't imagine the switch is being overloaded. There is only about 20-30Mbit of traffic coming into the VM. Disk activity is under 1MB/s. Disk queue length is under 0.01.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
It's connected to the same switch but with only about 8 devices connected there I wouldn't imagine the switch is being overloaded. There is only about 20-30Mbit of traffic coming into the VM. Disk activity is under 1MB/s. Disk queue length is under 0.01.
It should be connected to the switch as you have. This issue often arises when users connect the BI pc to the router that cannot efficiently deal with all the traffic.
The other place to look is the vm. It can cause all sorts of unexpected issues. If you want a solid reliable BI installation, use bare metal. Your server in particular is really inefficient as its old and a power hog. Running blue iris on it or for that matter keeping it running 24/7 for anything is waste full and costs you more to run than replacing it.
 

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
It should be connected to the switch as you have. This issue often arises when users connect the BI pc to the router that cannot efficiently deal with all the traffic.
The other place to look is the vm. It can cause all sorts of unexpected issues. If you want a solid reliable BI installation, use bare metal. Your server in particular is really inefficient as its old and a power hog. Running blue iris on it or for that matter keeping it running 24/7 for anything is waste full and costs you more to run than replacing it.
I'm not worried about the power consumption. Since the CPU usage is so low there is likely another reason for the issue, right? It's sitting with 8 cores virtually (haha) unused at the moment.

1589494775679.png
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
I'm not worried about the power consumption. Since the CPU usage is so low there is likely another reason for the issue, right? It's sitting with 8 cores virtually (haha) unused at the moment.

View attachment 61743
Looks like some of your cores are maxed out. My point is you have this space heater of a server that can be replaced by something that works well for 100 bux. You are using a 10 year old server presumably to save money so you should be concerned about power consumption. You are spending 100 bux extra to run that system per year over an efficient pc.
 

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
A dual processor CPU will not work efficiently with BI or any other program that is not designed to support a dual processor. So 48% is maxed out on a single processor.

A two processor system is completely different than a multi core single processor chip.
 

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
Looks like some of your cores are maxed out. My point is you have this space heater of a server that can be replaced by something that works well for 100 bux. You are using a 10 year old server presumably to save money so you should be concerned about power consumption. You are spending 100 bux extra to run that system per year over an efficient pc.
I'm certainly open to a $100 system to run these cameras off of. This was simply for convenience as the system is used for several other virtual machines anyway.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
I'm certainly open to a $100 system to run these cameras off of. This was simply for convenience as the system is used for several other virtual machines anyway.
You’re better off replacing this entire system as well. You are paying your electric. company ransom to keep it running
 

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
You’re better off replacing this entire system as well. You are paying your electric. company ransom to keep it running
I appreciate your concern for my power bill but it really isn't as power hungry as you seem to be suggesting. You mentioned something about a $100 system to replace the current BI VM?
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
As an eBay Associate IPCamTalk earns from qualifying purchases.

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
As an eBay Associate IPCamTalk earns from qualifying purchases.

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
The machine is on a SmartUPS that shows power consumed. Thanks for the link. Is there any reason to move away from Windows 10 onto a server OS, or is Win10 just fine?
how much power is consumed? Have you done the math?
No windows 10 is great. I run over 20 machines on w10 pro.
 

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
how much power is consumed? Have you done the math?
No windows 10 is great. I run over 20 machines on w10 pro.
520w for the rack. 2 gige 24 port switches (1 cisco, 1 linksys, 1 poe), 1 cisco 1g router, 1 R710, 1HP G7 that just sits there with 64 cores not really doing much, 2 other towers that are for a SQL DB and a Connectwise server and a 5bay eSATA array.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
520w for the rack. 2 gige 24 port switches (1 cisco, 1 linksys, 1 poe), 1 cisco 1g router, 1 R710, 1HP G7 that just sits there with 64 cores not really doing much, 2 other towers that are for a SQL DB and a Connectwise server and a 5bay eSATA array.
That doesnt tell you how much of that is the server.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
do you know how much 520w of power 24h per day costs you per year?
one more thing to consider for the new system is that if you want to use intel hardware acceleration and h265 you will need a sixth gen processor like an i5-6500, those systems are about 150.
What frame rates are you running on your cameras?
 

cronic

n3wb
Joined
Oct 17, 2018
Messages
22
Reaction score
5
Location
flowduh
do you know how much 520w of power 24h per day costs you per year?
one more thing to consider for the new system is that if you want to use intel hardware acceleration and h265 you will need a sixth gen processor like an i5-6500, those systems are about 150.
What frame rates are you running on your cameras?
Yeah, with the AC unit in the server closet it's about $100/mo. Cameras run around 20fps.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
Yeah, with the AC unit in the server closet it's about $100/mo. Cameras run around 20fps.
The 500w alone cost you about 40-70 bux a month depending on where you live. Then add the AC. With efficient systems you would be paying a tiny fraction of that since you wont need an ac.
Are you using direct to disk on all cams?
You can try using blue iris's new substream feature to lower cpu in the interim. The function is not 100 percent implemented and you may have some issues so proceed with caution only after reading the threads on the subject.
 

bp2008

Staff member
Joined
Mar 10, 2014
Messages
12,666
Reaction score
14,007
Location
USA
Well, there are 7 cams, and each is likely being decoded on only a single thread. Your CPU cores aren't very fast individually, made worse by virtualization overhead, and possibly also affected by non-uniform memory access. Lowering the frame rate should help immensely.
 
Top