Weird Networking issue with Hikvision Camera system

[eangulus]

Hi,

First let me explain what I have.

2 x DS-2CD2532F-IS
4 x DS-2CD2032F-I

NVR is an i7 PC running BlueIris

Network is run with an RT-AC68U on Shibby Tomato, 1Gb to a PoE Switch, all cameras + nvr are on the PoE Switch.

My problem is this.

Every so often, my whole network dies. All devices stop getting internet access. The only thing that fixes it is rebooting the PoE switch (and therefore the cameras).

This isn't regular at all. It may be several months between incidents, sometimes it will happen a couple of times within hours of each other. Then it may work fine for 2-3 months and then crap out again.

All Network features on the cameras are turned off. Only Network related items that are on are TFTP, SSH, DHCP, TCP. Everything else is off.

All cameras are running 5.2.5 firmware. Rebooting the NVR doesn't fix the problem so I doubt it's that.

I am confused as to not only what is causing the problem but more what could cause a problem like this. There is no features on that could saturate the network to the point of crashing.

Rebooting the router never works either, only rebooting the PoE. I have yet to try rebooting individual cameras (will be trying that next time it craps out) to see if it may be 1 camera going rouge.


Any ideas from some networking experts?

Also, if it is handy, I have a similar setup with a client that uses 16 2032 Hikvisions, and when we have the 11 Cisco IP Phones & Cameras on the same network, the phones die.

For reference:
Client Internet is 6Mbps Down and 0.6Mbps up.
My connection is 100Mbps Down and 40Mbps Up. (but until 2 weeks ago, I had 16Mbps Down and 0.8Mbps Up.)

 

[nayr]

sounds like you might be maxing out the power celling on your switch, or its faulty.. you neglected to share the switch make/model#

is it just the PoE devices that die, can you still contact your NVR?

 

[eangulus]

Will get back to you on the PoE Brand/Specs (not at home).

But the cameras are still functioning/recording. Its more like something is saturating the network to the point that other devices no longer can see the internet. Even NFS and Samaba connections from a server still work. Only way I find out there is a problem is when internet drops out, but the router says its still connected and after a reboot of the PoE and cameras internet is working again without touching the router or the NTD.

 

[nayr]

you could be having an arp flood, or something simuliar.. its definitely something wrong with that switch I suspect.

Next time its all dead, plug into the router and see if it has internet.. if it does then your switch has gone stupid for some reason or another, will be hard to tell.. is anything exposed to the internet, could your cameras be compromised or being brute forced? sounds like you access this alot remotely, block your cameras from any internet access on your router then VPN past your router when you need remote access.

It could be a device is compromised and zapping your internet, then when you reboot the switch you kill whatever it was doing.. thus freeing your internet.. try removing one device from the network at a time while keeping a continuous ping open to see if anything causes it to come back when removed.

Scan all your computers for malware, they can bust switches trying to find other computers to infect..

 

[eangulus]

My whole network is secure. Due to the nature of my clients, I actually get my network tested every 4 months.

The cameras don't have access to the Internet. I remote into the BlueIris software on the NVR. This is also done over a VPN connection.

No other devices are affected other than loosing internet. The only items plugged into the PoE Switch is the Cameras, NVR and a link directly back to my Router.

Cone to think of it, my devices don't just loose internet, I remember the first time it happened, while I was tearing my hair out try to fix it, I remember that after disabling the Ethernet card on the NVR, it wouldn't pickup a DHCP address. This also happened with my mobile WiFi.

So far the ARP flood seems to fit the bill. What can cause this and is there a way to mitigate it?

Next time it happens (have no idea when thou), I will try disconnecting individual cameras instead of the whole switch to see if it is a single device. Also was going to say I can ping from my Router out directly but remembered that while it happens, I can't log into the router either. (Rebooting router doesn't fix it either, have tried that).

 

[nayr]

good info, this helps.. are your cameras and stuff configured for dhcp? perhaps next time this happens configure a static ip on a device and see if it regains internet.

anything else that could be running a dhcp server such as the NVR or access points? i seen behavior similar to this from rouge dhcp servers hijacking the gateway/default route.. any network monitoring/scanning services that could be misconfigured?

arp tables can be poisoned or corrupted if you have various network issues, from mac address collisions, spanned trees (branches of switches with a loop), malware trying to detect hidden services, etc.. it usually requires analyzing traffic on the network to figure out the source of the problem.

 

[eangulus]

Definitely no rouge DHCP Servers, single Wifi Router, no AP's no DHCP from NVR (PC with BlueIris).

No network monitoring (is now but didn't have any when problems occurred). No spanning trees.

Based on your help (and much appreciated, even thou I have check most of it, it helps get some fresh eyes and some double checking), what I will do if and when it happens again:

1. Run Wireshark (if I can get network access) and hopefully record what's going on.
2. Unplug each Camera then other devices on the PoE switch to see if it is a device causing it.

Anything else to add to the list? I just want to have a setup and list ready for when it happens again so I am prepared to get some info.

 

[ManuMCoupe]

I have the same problem with a Asus AC66U with Shibby Tomato and a Hikvision Cam

Did you solve it?

 

[glenm]

Could heat be an issue? Where is your PoE switch physically located? I found one in a very poorly conditioned wiring closet once, well over 100 degrees in there. I note it is summer time in the northern hemisphere, and there have been heat waves lately in a few geos.

 

[ManuMCoupe]

In my case, it's on a closed box but ventilated with two BeQuiet 140 fans. It was working perfecty before my router upgrade to shibby tomato firmware :-S
It's surely a Shibby Tomato option to activate/desactivate, but which one? :-S

Everything else is working perfectly :-(

 

[ManuMCoupe]

Hard reset done, but still having the problem :-S The camera works great during 8-10h, and suddenly stops. The cam is still working, but impossible to access to it. The only way to restore it, it's powering it off :-(

 

[ManuMCoupe]

Problem solved!

Hard reset + configuration from scratch.

Hard reset and config import didn't solved it before, very strange.

24h working now

Enviado desde mi MI NOTE LTE mediante Tapatalk

 

[NickIzk]

I have this issue as well. There seems to be a very bad networking bug with these HikVision cameras (using a 2xx2 device). Not only does it flood the network with UPNP broadcasts (even when that is unchecked) but it also appears it will constantly flood to devices even when they disconnect. For example I had my router setup to port-forward to the camera to allow viewing from the Internet. With my Android phone I could peek in and see whats going on. Later on (maybe a week or two) I notice my router is just constantly lit up when nothing is accessing the internet. Sniffed the traffic and turned out the camera is just continually sending traffic to the last connection (my phone) even though the app was turned off/phone rebooted just to be sure. I've also seen this issue just using the web viewer. Hard to reproduce but when it happens, the only fix is rebooting the camera.

Ended up SSH'ing into the camera and manually setting up the iptables rules so 1) it stops using ipv6, 2) it stops all multicast traffic, 3) it can only transmit to certain IP ranges (to limit the flooding issue). Using a NVR now to access the camera and set it so the camera can only broadcast to that device to limit this bug since I can't babysit my network every time I want to view the camera.

I was on firmware 5.20 when this happened, tried 5.23 and same issue, then tried 5.3 (big mistake) and same issue but now I can't change the ipfilters because they blocked SSH / shell access. About to throw this camera away and go for another brand, these things are a complete pain in the ass to upgrade and that network bug makes me very nervous.

 

[ManuMCoupe]

I have this issue as well. There seems to be a very bad networking bug with these HikVision cameras (using a 2xx2 device). Not only does it flood the network with UPNP broadcasts (even when that is unchecked) but it also appears it will constantly flood to devices even when they disconnect. For example I had my router setup to port-forward to the camera to allow viewing from the Internet. With my Android phone I could peek in and see whats going on. Later on (maybe a week or two) I notice my router is just constantly lit up when nothing is accessing the internet. Sniffed the traffic and turned out the camera is just continually sending traffic to the last connection (my phone) even though the app was turned off/phone rebooted just to be sure. I've also seen this issue just using the web viewer. Hard to reproduce but when it happens, the only fix is rebooting the camera.

Ended up SSH'ing into the camera and manually setting up the iptables rules so 1) it stops using ipv6, 2) it stops all multicast traffic, 3) it can only transmit to certain IP ranges (to limit the flooding issue). Using a NVR now to access the camera and set it so the camera can only broadcast to that device to limit this bug since I can't babysit my network every time I want to view the camera.

I was on firmware 5.20 when this happened, tried 5.23 and same issue, then tried 5.3 (big mistake) and same issue but now I can't change the ipfilters because they blocked SSH / shell access. About to throw this camera away and go for another brand, these things are a complete pain in the ass to upgrade and that network bug makes me very nervous.

Oh s***! In my case, the problem disappeared but it came back a week ago. I'm trying if it's a heat problem, because the Poe feeder is in a closed cabinet with my 4disk Nas and 8 ports switch (ventilated with 2x140 fans on side to outside).

I'll maintain you informed.

Thanks

Enviado desde mi MI NOTE LTE mediante Tapatalk

 

[ManuMCoupe]

The issue is back. Don't know what to do to solve it

Enviado desde mi MI NOTE LTE mediante Tapatalk

 

[ManuMCoupe]

Problem still there. I'm making some tests (changing cables, without pie,etc...). If I don't get good results, I will contact support this week

Enviado desde mi MI NOTE LTE mediante Tapatalk

 

[eangulus]

I am having similar issue again on new cameras.

Recently put in 6 2085 4K bullets and saw that the every activity light on 8 48 Port switches was going nuts. Having dealing with a similar issue that started this thread I was determined to solve it.

Thee cameras are flooding the network with multicast or upnp packets.

I have multicast and upnp off. Still doing it.

I then checked the camera logs and config file, in there it indicated that multicast was on even thou the option is off in the GUI.

I then disabled dhcp and set it to static... done.

That fixes the problem.

This started on 5.5.0 firmware and just updated today to the newest firmware 5.5.3 or 4... Wherever the newest is. Tried turning on dhcp again, but problem still there.

Now I want to test if the problem with the 2032's was/is the same thing.

Sent from my Pixel XL using Tapatalk