BI and Cameras are 7000+ Miles Apart, Consistent freezing every 60-70 seconds for 7 to 10 seconds

[Norcal]

Hi everyone,

I've been having this issue since the BI4 days for my one unusual setup. Not a novice user here, I have deployed BI for many locations ranging from just 4 cameras to 50+ on large manufacturing facilities and also use it at home BUT they are all setups that have both the cameras and BI server within the same physical network. Latency between the BI server and IP cameras are almost nil, just what you expect on a standard local network.

The unusual setup I have is that the BI Server is running at a datacenter here in California with 1000+Mbps up/down bandwidth (used by other stuff and not just BI of course) while all the IP Cameras are 7000+ Miles away located in the Philippines at Subic Bay. The cameras are behind a 50Mbps symmetrical Fiber internet and connected to the BI server via a site-to-site VPN tunnel. All the cameras in the Philippines are Amcrest IP Cameras (outdoor poe bullet, outdoor poe PTZ and indoor non-poe PTZ).

When they are connected to record via the Amcrest PC-NVR running in California or viewed via the Amcrest App, no freezing whatsoever is exhibited. But when I use BI even to just view the IP cameras (no recording), the stream freezes every 60-70 seconds for around 10 seconds and that nice Orange Clock shows up.

The only difference this setup has with all of those that I have deployed here in the US is that there is about 170ms latency between the IP Cameras and the BI Server since it is in two different continents. There are a total of 7 IP cameras used for monitoring two elderly folks (my 97 year old granny and her 98 year old older sister) and the care staff. Running just one or all cameras, increasing the buffer size, forcing UDP or changing camera profile does not make any difference.

I hope someone here can shed some light as to why this only happens with BI and not via Amcrest PC-NVR. Thanks!

 

[looney2ns]

Hi everyone,

I've been having this issue since the BI4 days for my one unusual setup. Not a novice user here, I have deployed BI for many locations ranging from just 4 cameras to 50+ on large manufacturing facilities and also use it at home BUT they are all setups that have both the cameras and BI server within the same physical network. Latency between the BI server and IP cameras are almost nil, just what you expect on a standard local network.

The unusual setup I have is that the BI Server is running at a datacenter here in California with 1000+Mbps up/down bandwidth (used by other stuff and not just BI of course) while all the IP Cameras are 7000+ Miles away located in the Philippines at Subic Bay. The cameras are behind a 50Mbps symmetrical Fiber internet and connected to the BI server via a site-to-site VPN tunnel. All the cameras in the Philippines are Amcrest IP Cameras (outdoor poe bullet, outdoor poe PTZ and indoor non-poe PTZ).

When they are connected to record via the Amcrest PC-NVR running in California or viewed via the Amcrest App, no freezing whatsoever is exhibited. But when I use BI even to just view the IP cameras (no recording), the stream freezes every 60-70 seconds for around 10 seconds and that nice Orange Clock shows up.

The only difference this setup has with all of those that I have deployed here in the US is that there is about 170ms latency between the IP Cameras and the BI Server since it is in two different continents. There are a total of 7 IP cameras used for monitoring two elderly folks (my 97 year old granny and her 98 year old older sister) and the care staff. Running just one or all cameras, increasing the buffer size, forcing UDP or changing camera profile does not make any difference.

I hope someone here can shed some light as to why this only happens with BI and not via Amcrest PC-NVR. Thanks!

Since you mentioned the orange clock, To be clear you are referring to viewing feeds with UI3, not directly on a BlueIris console, correct?

 

[bp2008]

Hey @Norcal. As @looney2ns noted, the orange clock is only a thing in UI3 which means your problem is actually between Blue Iris and the device you are running the web browser on. It could be due to a slow CPU but usually it is caused by a connection that is too slow or unstable.

The internet connection between California and the Philippines could also be a potential problem though if it temporarily drops causing BI's video processing routines to stall in such a way that it also stalls webserver streaming. I'm not sure that is a thing that can happen though.

To help diagnose network issues, there's a tool I wrote where you enter a remote IP address and it does a traceroute and then continuously pings every IP address between you and the remote IP address. It runs on Windows. bp2008/pingtracer For best results, make sure the remote IP address responds to pings, otherwise you won't be able to monitor the entire path.

 

[pov2]

When they are connected to record via the Amcrest PC-NVR running in California or viewed via the Amcrest App, no freezing whatsoever is exhibited. But when I use BI even to just view the IP cameras (no recording), the stream freezes every 60-70 seconds for around 10 seconds and that nice Orange Clock shows up.

@bp2008, it's very interesting, I have exactly the same problem when I view cameras in tinyCAM Pro through BI on my Android phone. I am calling you because I think you developed the UI3, if I am not mistaken, and UI3 is used for streaming in this case. My BI is set to stream and I pointed my tinyCAM Pro app to its external forwarded port and used Camera Brand: Blue Iris. It happens from my cell network and from my internal network through hairpin NAT built in my router. There is one more detail - traffic goes through a nginx-based https reverse proxy. I thought it could be the culprit but it causes no problems for other kinds of file transfers unrelated to BI, and, more importantly, there is no problem when I view cams through the BI Android app, and the traffic goes exactly the same way as with tinyCAM. In the end I thought it was tinyCAM's issue but I am surprised to read OP's familiar story. Maybe it is UI3 related? The streaming settings are max GOP: 15 frames and zero max B frames.

 

[bp2008]

I don't know how tinycam integrates with BI. If it indeed shows the orange clock icon in the corner, then you are probably right and it is loading UI3 in an embedded browser. But that seems unlikely.

Blue Iris's streaming performance is better with the longest possible GOP setting in the Streaming 0/1/2 profiles. That would be 300.

 

[pov2]

I am not sure how it integrates. It shows some extra info such as system name, CPU%, memory%, uptime, a list of BI devices. Each device is on a "channel" corresponding to the order of cams in BI. It looks to me like it's using UI3. I may be wrong.

Increasing GOP to 300 made it worse. I got a dropout after a few seconds. There is a diagnostic section in the tinyCAM app that shows displayed frames, received frames and dropped frames among other things. For cams streaming at 15 fps, when dropouts happen, somehow received frames become 30 fps and display frames - zero fps. That's strange. The app uses a "H264 Proprietary" protocol. Maybe after all it has something to do with tinyCAM. Pinging @alexvas ...

 

[Norcal]

Sorry for the late reply, work has become more hectic for IT folks like me now that everyone is working from home and needs more support.

@looney2ns:
It happens on both UI3 and on the local console of BI. The freezing also shows up on the continuous video recording.

@bp2008:
Thanks for providing the tool you developed. The server where the Blue Iris is running on has about 50GB of free RAM just 25% CPU Utilization of the 32 Logical Cores (dual Xeon Processors with 16 Cores and HT) and has an NVMe storage for both OS and Blue Iris Storage. The sender Bandwidth (Philippines) has 50Mbps upstream that is only 50% utilized while the receiver/Blue Iris here in California has a few hundred Mbps downstream bandwidth. Watching the camera stream directly on VLC Player does not show any issues whatsoever that is why I am very baffled with this issue.

Here is the result of running your tool for 10 minutes. The only variable I can think of is that there is a steady 170ms-180ms latency between the two sites but that is to be expected and does not seem to affect how the IP Camera video stream plays when directly viewed on VLC Player while with BI it freezes every minute.

 

[bp2008]

You are right. If there were any severe bandwidth issues or packet loss, we would see it in the graphs. Which we do not.

But that only verifies the quality of the link between your BI server and the camera site. As I said before, your symptoms indicate that the problem is between your Blue Iris server and your viewing device upon which you run UI3 (the web interface). You've basically ruled out server CPU / memory bottlenecks, which means the problem is most likely that you are running UI3 (Blue Iris web interface) on an incapable device or your connection between that device and Blue Iris is not capable of handling the traffic.

 

[Norcal]

Even without using UI3 and just viewing it directly on the Blue Iris Console the freezing still happens every minute. The Blue Iris video recordings also show the freezing where people "teleport" because frames were lost since the freezing causes BI to record nothing at that frozen period. Yet when I use the Amcrest PC-NVR to record and view or use VLC Player to directly view the cameras from where Blue Iris is installed here in California, there are no freezing whatsoever.

This is the 3rd iteration of the Blue Iris system for the cameras in the Philippines. The first one was on a new computer (i7, 8GB, SSD, tested with Intel Quicksync HA and without/disabled HA) behind a Comcast 300Mbps here at home on BI4 in 2018 and the same issue happens. Moved the Blue Iris system to the Datacenter on a much beefier host and larger 1000Mbps symmetrical bandwidth on BI4 and still the issue is there. Moved the Blue Iris to a much beefier host (Xeon 2Proc 32 Cores, 128GB RAM, all Enterprise NVMe storage) and still the same story - Blue Iris feed freezes every minute while VLC Player and Amcrest PC-NVR works without a glitch. This is what has been bothering me for years since the only thing I see different is the latency between the sites but why on earth does VLC Player and the Amcrest PC-NVR not have any issues? This has literally already caused the consumption of more than 10 bottles of good glenfiddich in the past 2 years!

 

[looney2ns]

Have you emailed support with your issue's.
He doesn't visit here.

Sorry if this has been mentioned, but you have excluded all BI folders from Windows defender and any other virus apps?
What happens if you test a local camera to the same system?

 

[bp2008]

Well that is interesting. If it has happened on all 3 BI systems then that really does suggest a problem with BI. But I can't think of any reason why high latency would do it. What kind of cameras are you using? Have you used that type and configuration with BI successfully before?

Some thoughts. How sure are you that the VPN capacity isn't maxed out? It can be quite CPU-intensive to handle all that encryption.

BI has a setting for the receive buffer size in the network ip camera configuration panel. Have you tried increasing it a bunch?

BI also has some settings related to RTSP that could be useful to toggle. Located in the same place as the receive buffer size.

Why can't you run the BI server where the cameras are?

Antivirus / antimalware services may interfere with video streams by intercepting them and trying to scan them before releasing them. I haven't heard of it happening to an RTSP stream before but nonetheless this is worth looking into.

 

[achalmersman]

Interesting problem. Not familiar with the armcrest stuff. Does the traffic for it or the app you mention utilize the same S2S tunnel or is that traffic going out to the web and your app connects to the web?

What are the details of the tunnel is it Cisco ASA or what? Have you done iperf tests or some kind of bandwidth and connection monitoring through the tunnel? How about any kind of weird QoS for the tunnel?

Sent from my SM-G965U using Tapatalk

 

[Norcal]

Its been a while but I figured I should give an update to this. What I discovered is that when the IPC video stream is travelling through a VPN tunnel with even the minimal encryption at that distance, the video stream timing goes haywire after a while then recovers thus the issue. The same issue happens regardless of any Layer-3 tunnel I used to connect the two locations. What worked to resolve the issue was to have the firewall in the Philippines do a Custom NAT for each IPC with unique TCP ports forwarding to tcp/80 E.g. PublicIP:9378 -> IPC1LocalIP:80 , PublicIP:9379 -> IPC2LocalIP:80 etc. then the Blue Iris here in California points to those PublicIPort addresses. The video streams still occasionally freezes but are rare and short and expected given the distance, hops and average latency between the two sites.

This is not an ideal solution because it now exposes the IPC to the public network although I have IP restrictions on what WAN IP addresses can connect to those ports on that firewall. Also, this was only possible due to the Public IP of the internet connection at the Philippines site was able to work in "Bridged Mode" with full transparency despite being a Dynamic WAN IP. Many ISPs do not allow bridged mode on their CPE Devices (i.e. your cable modem or the like) nor do they have the ability to do Port Forwarding/Custom NAT Rules.

Hopefully this information will help others that are trying to deploy similar transcontinental setups due to some unusual circumstances like mine. The next test I am going to do when I get the chance is to set up a Layer-2 S2S tunnel and see if that makes any difference with the previous Layer-3 tunnels i used.

 

[bp2008]

I'm glad you got that figured out.

Since you restricted the source addresses, your port forwarding should not open your cameras to attack. From the perspective of everyone else, those ports aren't even open.

 

[achalmersman]

Yea that's how I access my BI Ui3 or app. I port forward but only allow whitelisted WAN addresses.

Sent from my SM-G965U using Tapatalk

 

[Norcal]

You are right. It is just a pain in the behind to have to keep updating Blue Iris settings to use the new WAN IP address when the Philippines site gets issued with a new WAN IP since it is dynamic. There is also that short downtime when the IPCs cannot be seen by blue Iris anymore, good thing I have monitoring agents (uncles and aunts complaining LOL) alerting me when the video feed is down. I dont have any computer behind that network to install something like the no-ip agent nor is the firewall there able to talk to dyndns or something similar. Good thing the WAN IP changes just once a month then I update 8 cameras on BI.

The monthly cost of a SINGLE Static IP for that site costs about the same as what a family of 3 would have to spend in a month for food. I should just ask them to Please chop my arm off.

 

[bp2008]

@Norcal you know, a lot of IP cameras have DDNS clients built in...

 

[Prime]

Its been a while but I figured I should give an update to this. What I discovered is that when the IPC video stream is travelling through a VPN tunnel with even the minimal encryption at that distance, the video stream timing goes haywire after a while then recovers thus the issue. The same issue happens regardless of any Layer-3 tunnel I used to connect the two locations. What worked to resolve the issue was to have the firewall in the Philippines do a Custom NAT for each IPC with unique TCP ports forwarding to tcp/80 E.g. PublicIP:9378 -> IPC1LocalIP:80 , PublicIP:9379 -> IPC2LocalIP:80 etc. then the Blue Iris here in California points to those PublicIPort addresses. The video streams still occasionally freezes but are rare and short and expected given the distance, hops and average latency between the two sites.

This feels a lot like an MTU problem with the tunnel given that it works when going direct. I'm probably one of the few other people here that also has BI recording over a VPN - but in my case, the geographical distance is only a few hundred miles, so latency isn't an issue, and I have not observed the behavior you're describing.