Cameras losing connection - resolved by frequent reboots of Blueiris NOT the cameras themselves. What's going wrong?

[ccaru]

Hello,

I have a setup with 9 cams - mixed brands, mixed resolutions. A few months ago I had an issue with resources, which I have resolved successfully through the use of substreams.
This resulted in 15-20 % CPU usage, and it's very stable and responsive, BUT, from time to time, I start getting outages for several cameras at once - usually 3 or 4.
What I have found is that:
The cameras are still fine and they didn't crash. Accessing them via their separate web interfaces shows no issue
I have a few WIFI and a few wired cams. The problem does not seem to solely happen to wifi or lan. It's always a random mix of both.
Rebooting individual cameras (either restart or remote boot) does not solve the problem
A restart of Blue Iris does resolve the issue.
I can go for days/weeks without a single problem, but then suddenly I get an outage. It's almost never a single camera but several at the same time.
During these outages I see no signifcant change in CPU/RAM Useage.

Any ideas on what could be going on and whether this could in fact be related to the implementation of substreams or perhaps a coincidence? I cannot see myself going back from the use of substreams...

...and as a workaround, does anybody have any good ideas on how to remotely restart the Blue Iris service without needing to be at the console?

Cheers

 

[mikeynags]

Sounds more like some type of network issue. What’s your network layout look like? Also, WiFi is tough to run cams on in terms of it performing 100% of the time.

The windows Services MMC snap in can remotely stop and restart services on another machine.


Sent from my iPhone using Tapatalk

 

[wittaj]

What is your network topology like - are all the cameras connected and passing through the router or are they on a separate NIC or VLAN?

What is the computer CPU you are using? 9 cameras at 20% CPU is high and sounds like some more optimization is in order.

You can set up a Task Scheduler to restart BI, but that is a bandaid and something else is going on.

Post a screenshot of you BI camera status page that shows the FPS, MP, kB/s and MP/s at the bottom.

 

[ccaru]

Thanks for your quick reply. Prior to setting up substreams I was experiencing 50-60% usage but I never had this issue (granted.. I had other issues which perhaps superseded this one!)
The CPU is an I7 7200 / 2.5GHz (Kaby Lake) with 32GB Ram
Attached is the screenshot as requested.

 

[wittaj]

OK, something else is going on.

I am on a 4th generation with over triple the cameras as you and I am running 10% CPU. You should be sub 5% CPU with 9 cameras on a 7th generation.

I would revisit the optimization wiki and make sure that EVERY, and I mean EVERY optimization is done. Too many people come here complaining of high CPU usage and claim they have done every optimization in the wiki and once they post screenshots, we see they are not using the substreams and that is probably one of the biggest CPU savers. Do not skip one because you think it isn't important or won't make that big of a deal. Even dropping frame rate a few FPS can make a big difference. No reason to run more than 15FPS, and many us have cams running at 10 to 12 FPS.

If you do not understand what something does in BI, then ask. Too many people also change a setting not realizing what it does and actually makes their performance worse.

Are you doing anything else on that computer?

Are the cameras passing through the router?

Optimizing Blue Iris's CPU Usage

Blue Iris uses a lot of CPU time. In most Blue Iris installations, your CPU is the limiting factor which determines how many cameras you can have. As such, it is extremely important to optimize your configuration to ensure you can get the most out of your system while consuming the least...

ipcamtalk.com

 

[ccaru]

Thanks for the feedback. Indeed all optimizations feasible are implemented. I do not have an external Graphics card so acceleration is switched on but for Intel. Previously to switching on Substreams, i'd frequently get a CPU leak in that it would slowly rise up to 100%, but this stopped when I set substreams up.

The curious thing is that this is sporadic. I'd go for weeks without a single incident, then suddenly I find all these outages. Once it's corrected I might go again for several weeks or it might just happen after a few hours. Nothing in the resource usage states indicates an overload of anything. I do understand your point about 20% perhaps being too much, but it's still 20% and it will only exceed this marginally and return to regular operation (and actually that was perhaps even an exaggeration from my part, as I write this i am looking at the stats and it's currently at 12%) - I wouldn't expect an outage to happen with 20% useage.

Everything is going through the router. At one point i was fixating it was due to my wifi cameras so I actually passed some temp cables and connected everything through the LAN Ports, but after a while the same happened. Currently I have some cameras on LAN and a few on WIFI - in locations where it isn't feasable to pass a cable. However my understanding is that typically if there is a network outage, this gets resolved by a simple restart of the cameras. Even a reboot of the router itself doesn't require a Blue Iris service restart - it eventually comes back and all cameras are eventually found and go online. In this type of outage I am experiencing, only a restart of the Blue Iris service seems to resolve the issue.

The machine is totally dedicated to Blue Iris. I must also say that i've been testing Deepstack for a while too, but i think this occurred a few times before Deepstack was activated.

 

[wittaj]

OK, in addition to the wifi cameras (which I understand sometimes you gotta work with what you got), your other cameras passing through the router is probably the problem.

Wifi is problematic for surveillance cameras because they are always streaming and passing data. And the data demands go up with motion and then you lose signal. A lost packet and it has to resend. It can bring the whole network down if trying to use it through a wifi router. At the very least it can slow down your system.

Unlike Netflix and other streaming services that buffer a movie, these cameras do not buffer up part of the video, so drop outs are frequent. You would be amazed how much streaming services buffer - don't believe me, start watching something and unplug your router and watch how much longer you can watch NetFlix before it freezes - mine goes 45 seconds. Now do the same with a wifi camera and it is fairly instantaneous (within the latency of the stream itself)...

The same issue applies if it is hard-wired trying to send all this non-buffer video stream through a router. Most consumer grade wifi routers are not designed to pass the constant video stream data of cameras, and since they do not buffer, you get these issues.

You have 9 cameras basically each at 4MP resolution or higher and some as wifi trying to get routed through a wifi router. You are lucky it isn't more frequently.

Outages are probably happening around the same time that there is a lot of motion and the data requirements go up, you have a lot of activity going on other devices, or the cameras are phoning home. All of that makes for a dropped packet and BI sees that as loss of signal and goes offline for a period of time until BI tries to reconnect.

I would suggest trying to get those wired cameras off the router by either VLAN or what most of us do is add another internet card to your BI machine (they are under $20). Then connect all your cameras and BI machine to the same switch and put them on an ip address range that is different than your home internet ip address range.

 

[ccaru]

Thanks for that pointer. Very interesting. I must admit that one thing I omitted from mentioning (only because it was still half an idea formed in my head ) is that in these COVID times I'm tending to have a lot of Zoom/Teams meetings and was starting to wonder whether this was contributing to the issue. I will start by moving everything to a dedicated router. The BI machine already has 2 network cards, so I'll use one for this dedicated subnet. I'll update this thread with the results.

 

[sebastiantombs]

There is still something not right with utilization. I'm running 200MP/s on an i7-6700K and it sits around 20% all the time.

 

[Old Timer]

Also, look at your I frames. FPS on the main stream, and sub stream, including the I frame should be the same.

If you look at the pic you posted after the FPS, you will see the I frame rate. Most of your cameras are not close to 1.0.
See if the FPS will change to save CPU time and will help motion detect.

Some cameras even though you set them for the same will ignore it and send at .5 or twice the FPS rate. Replace with a good camera.

 

[looney2ns]

Thanks for that pointer. Very interesting. I must admit that one thing I omitted from mentioning (only because it was still half an idea formed in my head ) is that in these COVID times I'm tending to have a lot of Zoom/Teams meetings and was starting to wonder whether this was contributing to the issue. I will start by moving everything to a dedicated router. The BI machine already has 2 network cards, so I'll use one for this dedicated subnet. I'll update this thread with the results.

You don't need another router.
1-Connect all cameras to a POE gigabyte switch, if you have cams that are POE powered. Otherwise you just need a standard switch.
2-Connect the Bi computer to the same switch that is connected to the cameras.
3-Connect the switch with all the cameras to the router.

Be certain that you have excluded Blue Iris from any antivirus software as per the BI help file, this includes Windows Defender.

Avoid Wifi cams at all costs, there is almost always a way to get an ethernet cable to a camera.

 

[ccaru]

@looney2ns - thanks your message - just to be clear, as this gets a bit complicated for me : If I physically connect everything to the same "Box" then that Box to my main router, that's ok? Or should i go for a separate subnet?
I know it's not ideal but really I do have 3 cameras where I simply cannot use cabling - I've got an unused Netgear access point which I can dedicate to this - i guess it's always better than using the main WIFI from what I am reading.

I am however still interested to understand why Blue Iris doesn't automatically and eventually recover from these errors though and requires a restart. After all, if I reinitialize my network, Blue Iris does eventually find the cameras and go back online again.

 

[wittaj]

Personally I would put them on a separate subnet just to ensure they can't phone home and further isolate them from the potential of some weird nonsense happening of it still being managed by the router even if in theory it shouldn't. I am sure some router out there will overtake the routing if it can access it. Heck maybe your router disconnects a camera if it drops too many packets and that is why BI finds them again if you reboot the router... VLANs or dual NIC will ensure that doesn't happen.

Did you ever try the restart camera option in BI instead of restarting all of BI?

 

[sebastiantombs]

The cameras and the BI PC need to connect to the same "box", network switch. With the WiFi cameras you are stuck with them coming through the router and that can't be avoided unless you can use an access point on that, specific, network switch that supports all the other cameras. The camera traffic should not pass through a router, especially an ISP grade router, at all.

Maybe list your cameras, make and models, so we can understand their capabilities.

 

[TVille]

While it is best practice to physically separate the camera network from the internet, that is sometimes impractical. I don't do it due to my topology and having cameras in four buildings. But, you definitely have an issue. Your bandwidth is shown as around 2.5 MBps. I am running over 4 times that much through an i7-6700 with only 8 GB of ram, no GPU. With DeepStack running on four or so cameras, and Automatic License Plate Reader running on two, I run around 20% CPU. Without DS and ALPR, I would be below 10%.

I have only two wifi cameras at 640 and 1080 resolution on wifi. I agree with the other points on iFrame rate and keep it off the router. My routers only have one connection - everything else is connected to the GB switch connected to the router, including multiple other GB switches.

 

[looney2ns]

@looney2ns - thanks your message - just to be clear, as this gets a bit complicated for me : If I physically connect everything to the same "Box" then that Box to my main router, that's ok? Or should i go for a separate subnet?
I know it's not ideal but really I do have 3 cameras where I simply cannot use cabling - I've got an unused Netgear access point which I can dedicate to this - i guess it's always better than using the main WIFI from what I am reading.

I am however still interested to understand why Blue Iris doesn't automatically and eventually recover from these errors though and requires a restart. After all, if I reinitialize my network, Blue Iris does eventually find the cameras and go back online again.

Yes, whether you have a subnet or not, you still need everything connected to the same switch, Cameras, Bi, etc.
I would also add, connect that Netgear Access point to that same switch as well.

 

[ccaru]

Thank you all for your help with this one.

I think i've progressed another big step with my setup by moving everything into a subnet. I am just amazed at how the streams were bringing my router to its knees.

As a summary:

• I set up an unused router I had available and rewired all my wired cameras to it, as well as my Blue Iris server.
• I kept the router functionality and wired the WAN to my main router. This way, my Blue Iris server has internet access.
• I set up NAT to open the port for Blue Iris on the Asus router, and reconfigured the NAT of my main router to point to the Asus router

Some observations: I've always thought my regular router was very slow - particularly when accessing the management pages. This has now disappeared
On first installation of the Asus router, it was blazing fast - by the time I put the cameras on, it's inherited the above issue i had on the previous router
All my Zoom and Team issues I was experiencing when working from home seem to have disappeared.

So far, no further issues - although I will have to wait and see whether the problem as described originally in this post would reappear - as even on the previous setup it would sometimes take weeks to reappear.

I probably have a lot to learn on networking - I'm still at a loss why this happens. My cameras total 2.5 Mbps according to the Cameras Log in Blue Iris, which I thought was rather negligible - especially now on a dedicated router. Yet it seems like my router is working very hard to keep things running.

Once again - a big thank you for all your suggestions!

 

[wittaj]

Glad we could be of some help!

Yeah, the demands of the cameras cannot be discounted; the constant streaming of unbuffered data is what brings the routers to their knees. They were not designed for 24/7 data in that fashion.

Streaming devices like Netflix buffer and Zoom type video conferencing uses different coding to try to keep the demands down, but the cameras are full-blast nonstop data feeding machines.

 

[looney2ns]

Thank you all for your help with this one.

I think i've progressed another big step with my setup by moving everything into a subnet. I am just amazed at how the streams were bringing my router to its knees.

As a summary:

• I set up an unused router I had available and rewired all my wired cameras to it, as well as my Blue Iris server.
• I kept the router functionality and wired the WAN to my main router. This way, my Blue Iris server has internet access.
• I set up NAT to open the port for Blue Iris on the Asus router, and reconfigured the NAT of my main router to point to the Asus router

Some observations: I've always thought my regular router was very slow - particularly when accessing the management pages. This has now disappeared
On first installation of the Asus router, it was blazing fast - by the time I put the cameras on, it's inherited the above issue i had on the previous router
All my Zoom and Team issues I was experiencing when working from home seem to have disappeared.

So far, no further issues - although I will have to wait and see whether the problem as described originally in this post would reappear - as even on the previous setup it would sometimes take weeks to reappear.

I probably have a lot to learn on networking - I'm still at a loss why this happens. My cameras total 2.5 Mbps according to the Cameras Log in Blue Iris, which I thought was rather negligible - especially now on a dedicated router. Yet it seems like my router is working very hard to keep things running.

Once again - a big thank you for all your suggestions!

You really need a proper switch such as

https://www.amazon.com/NETGEAR-Unmanaged-Rackmount-Lifetime-Protection/dp/B07788WK5V/ref=sr_1_4?crid=3UV1EGWISFRPI&dchild=1&keywords=netgear+poe+switch&qid=1630000919&sprefix=netgear+poe,aps,203&sr=8-4

 

[ccaru]

Well, one more observation from my side: I now have dedicated a WIFI router for my WIFI cameras - 4 out of 9. Even then, the number of dropouts was very high for all four. I've taken the plunge and wired 2 of the 4 and the number of dropouts drastically decreased - not only for the wired ones, which are now close to zero, but also for the other two. Still however, the number of dropouts is just too large - I"ll be looking at wiring my other camera, with the last one being unfortunately a WIFI only model which i'll probably end up ditching at some point.