7 cameras now cycling on/off

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
Guys,
I have a problem. It appears that 7 of 11 cameras went bad within a day. They are all different (brand, models) and have been purchased over various time periods. They are all Dahua and HikVision. The oldest is a cheap Dahua, 10 years old and as I got more and more into this I did spend more money - so some are higher quality than the first ones. I can provide a list if this helps troubleshooting.

I did recently upgrade to UNIFI UDM and monitored switches 4 weeks ago. The first three weeks all was flawless. Then I was notified by UNIFI about an intrusion that came through my daughters laptop that went after a handful of cameras (some NVR command was recognized). I am not sure anymore if at that time I had just thread monitoring on or the thread prevention. The attack went after some cameras but not all of them.
From there on we noticed occasional drop offs in BlueIris but I am not sure if this was related.

A few days ago all 7 cameras are now constantly cycling on/off. BlueIris reports this and it is up to 100 times in one hour.

What is strange is that some of the cameras that were touched by the intrusion attempt are not the cycling on/off. So I do not believe this was the problem. The only common thing I can see is that these 7 were all on one POE (TP Link Switch 8 60W). One could assume that something happened through that POE which caused this. The only strange thing is that one HikVision that was on the same POE and this one is not affected. So 7 of the 8 POE ports were affected.

I also see high network traffic now and everything slows down.

My trouble shooting steps so far:
  • tried two different POEs for these cameras - no difference
  • I tried disconnecting all of them except one at the time - no difference, still high network traffic
  • brought cameras closer to the router (from workshop to house) - no difference
  • placed the old router back into the network to see if UNIFI Thread management (IPS) made a difference - no difference
  • I factory reset two cameras but no difference
It is my assessment that the POE must have had a power surge or something that caused 7 of 8 cameras to go bad. Is this something that you have seen before?

My questions:
  1. Is there a way to test the POE to see its stability before I connect new cameras to it?
  2. Is there a way to test the camera separately in a small network setup? Any suggestions? I still have my old router that I can setup but I have to drag the other components (i.e. BlueIris) from the house to the workshop as I have only one Lan line between the two.
  3. Has anyone experience with UNIFI? I read that you should not put the cameras on a virtual network due to UDM having issues with high traffic across Virtual Network. So I had all cameras on the main LAN but firewall protected to only talk to BlueIris IP, also no internet access for any camera either way. Is there someone that may guide me with the best setup as I start building this again. I am thinking of complete factory reset my UDM and start from scratch.
  4. Also - I need to start buying cameras again. You can imagine that buying 7 will strain everyone budget. I dont need the latest and greatest,
    1. two for the horse barn (indoor but very dusty, no need for high end),
    2. two for the workshop (indoor, no need for expensive ones),
    3. two high quality outdoor watching the house,
    4. one outdoor (not high quality needed). Any recommendations?
  5. Any thing that I should test before I embark on a complete reset & purchase spree?
 

samplenhold

Known around here
Joined
Aug 8, 2018
Messages
4,813
Reaction score
14,699
Location
Spring, Texas
I know nothing about UNIFI, but why are you going through a router for your cam traffic?

Have you tried WIRESHARK to trace the high network traffic?
 

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
@samplenhold - I am not sure if I understand. The router has the DHCP and I thought I need this to arrange the traffic. I also believe that I need this because of the following:

House has the modem and the Router.
From there I basically have two branches - one in the house and one that goes to the workshop.
In the house I have all kinds of wired and wireless devices. But only one camera and I also have the BlueIris Server there.
In the workshop I have a switch and from there branch out to an indoor access point and outdoor access point. Again, I have a bunch of devices attached. From the switch the described POE switch with all cameras is branched off.

Unfortunately I have only one Cat5 connection between the Workshop and the House. I could see this nicely segregated if I had run two Cat5 at the time.

In the house, workshop and outdoors I have IoT devices that I have segregated into one virtual network that can be done in UNIFI.
I also have a segregated Guest network and then the main LAN. All are separated by strict firewall rules.

Do I understand your question on the CAM traffic right? You have this physically separated from the any other LAN and the router? What device is arranging then the IP assignments and traffic? Can you manage this all without a router and only have static IP through a simple switch? In that case I need BlueIris on that network too but also need BlueIris connected to the main LAN. Wouldn't this require a Router to manage?

BTW - I have just tried my old router (not connected to the internet) and a quick BlueIris Laptop install with the cameras in question. It is still happening in this simple network setup, although some cameras show less cycles now.
 

samplenhold

Known around here
Joined
Aug 8, 2018
Messages
4,813
Reaction score
14,699
Location
Spring, Texas
Again I know nothing about UNIFI and even less about VLANs. I have my cams physically separated (192.168.2.xxx) from my LAN that has internet access (192.168.1.xxx) and that sub-net is the only one with a router. None of my cams run through that router. They only run through POE switches. I do not use DHCP for IP address assignments for my cams or the BI PC. I use fixed IP assignments. Having the cams on their own sub-net simplifies troubleshooting. The BI machine has two NICs, one on each sub-net.

Having things set up this way stops all of the IoT stuff I have from interfering with my cams.
 

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
@samplenhold - thanks, that makes a lot of sense and is probably the most secure setup. But it requires me to run a second Cat5 line to the workshop because I still have this one camera at the house that I need to tie in. That is not going to happen. They said, use the biggest empty pipe you can possible run, I ran a 1" pipe and have it full now with separate runs for network, security, cable and telephone....
I should have run 2"...
I have to check if I can convert the telephone line (not used anymore) into the security panel line and that will free up one Cat5.

On the other questions - does anyone has some recommendations on question 4) for basic cameras?
 

samplenhold

Known around here
Joined
Aug 8, 2018
Messages
4,813
Reaction score
14,699
Location
Spring, Texas
I would bet that with two managed switches, one at each end, and VLANs you could pump all that needed through the one Cat5e. But I do not know that much about networking. Maybe @mat200 @Holbs @sebastiantombs or others could help on that.

As far as cams, the 5442 series is the go to cam right now. They come in bullet, turret, and domes. Varifocal and fixed lens. They are 4MP on a 1/1.8" sensor and do very well in color at night if there is some light. Otherwise they do very good with IR. If those a too expensive ($155-$185) then there are other cams. I have one T2231T-ZS-S2 varifocal turret which is a 2MP on a 1/2.8" sensor ($125) and does fine in night color mode with a good amount of light. I also has two T2431T-AS-S2 turrets ($80) that are 4MP on 1/3" sensor. I have them in my garage and they do OK but are not great in color low light. The IR mode is fine though. All of these cams have been reviewed here.

Edit: I only follow the Dahua cams. But the Hik cams are just a s good. I just do not know anything about them.
 
Joined
May 1, 2019
Messages
1,897
Reaction score
2,656
Location
Reno, NV
set daughters laptop (or any one who is not comp security conscious) to their own VLAN and mostly locked down. They will download screensavers and all bunch of malware/virus stuff. Might as well put them in on the IOT VLAN.
10 years and now all of sudden Blue Iris drop offs from camera's? Best take the safe route and lock everything down. I'm not virus/malware/security dude but I would treat your entire network as compromised and work backwards from 100% safe to.... comfortably safe.
Firewall Rule the heck out of all VLAN's and monitor. WAN OUT rules, galore and between VLAN's.
Update all virus/malware software, use duck DNS maybe as well?
 
Joined
May 1, 2019
Messages
1,897
Reaction score
2,656
Location
Reno, NV
Your single Cat5 cable out to workshop. It may work for great bandwidth, or it may not. Only way to know is to certify/test. I've seen standard Cat5 cables pass as Cat5e and sometimes Cat6. Could you use the Cat5 cable as a pull string so you can pull in a new Cat6?
I would look into wiresharking some network stuff. It's advanced but that's the name of the game for IT troubleshooting.
 

IAmATeaf

Known around here
Joined
Jan 13, 2019
Messages
2,611
Reaction score
2,040
Location
United Kingdom
What you could do is add an extra card to your BI PC, then connect the new network card to a POE switch and connect the camera on the same side as the PC to that switch, then connect the single uplink to that switch. So you’ll have effectively separated the cams using the extra card. For the new network card and cams, assign them all static IPs in their own subnetwork.

Doing the above will ensure that your cams aren’t accessible from the internet NAS will keep the cam traffic separated.

The single cable I would also test it to ensure that it is capable of running at a gigabit and then ensure that the switches at either end have a gig uplink capability.
 

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
I figure I give you and update as I go.
I realized this morning that the telephone cable that I had run to the workshop was actually a Cat5. So I repurposed the telephone line and now have no landline anymore in the shop. No loss as everyone uses cell phones anyway.
With that line freed up I tested it and have now all cameras on a separate network. I use an old wireless router (not connected to the internet) to manage this stand alone network. I have now all but 2 cameras back in operation and it seems to be 'normal'. Two of them are probably dead but I need to further investigate this. BlueIris sits on this network now with its standard NIC. The router is set to 192.168.10.1.
I will try later to install a wireless USB stick into BlueIris and connect this to the main network which will be on the main router serving 192.168.2.1 with guest access on 192.168.3.1 (virtual) and IoT devices on 192.168.5.1 (virtual). I just need to figure out a logic on what sits where and where I connect the second BlueIris network access to? Any recommendation? I only have one port open to the internet for the remote view of the cameras pointing to the BlueIris webserver.
My initial preference is to place it on the IoT network as I had started to use MQTT to trigger things on the home automation. But the IoT network is the most vulnerable. I may regulate this with firewall settings.
And yes - I started placing all kids devices on the guest network.
 

samplenhold

Known around here
Joined
Aug 8, 2018
Messages
4,813
Reaction score
14,699
Location
Spring, Texas
You really should not run cams through a router. The stand alone subnet for your cams do not need a router.

Opening ports, even only one, will give you nothing but trouble. Use a VPN. See:
 

Shockwave199

Known around here
Joined
Mar 13, 2014
Messages
1,020
Reaction score
539
Location
New York
Check and make certain a bot didn't open up ports when there was a breech. Delete any open ports. High traffic and odd and intermittent network spells a bot having opened ports on you. I have been stung twice by that. No more port forwarding. You can get away with it sometimes for long stretches of time, sometimes even years. But it'll happen eventually. I kinda doubt it was a hack through your daughters laptop. More likely through an open port. I don't need remote viewing any longer so I just don't need to mess with it. If you really don't need remote viewing don't even mess with any of it. You're better off.
 

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
Great advice guys. I start getting things under control on the network side. What I am baffled with is still the camera side. Status today:

Cameras are on a separate network now, only BlueIris is on there with the original NIC. I also added a wireless NIC to the computer that is currently blocked on my network and I can switch this on/off if I need this. Kids are banned to the GUEST network and IoT are all on IoT network.

I noticed that last night and the night before BlueIris crashed at exactly the same time - 12:00:05. So something is happening here and I need to find this out. Any advice on how to proceed?

Also, I thought I'd install a fresh Windows on it but had to upgrade from Win7 first. I now have a registered Win10 version and will completely reinstall Windows from scratch. Before I do this should I try Zoneminder and see if this works?
 
Last edited:

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
One more piece of info:

I completed a new install of Windows 10, reinstalled BlueIris and still have this happening. I then installed Wireshark but this is a lot of stuff to take in. I monitored the NIC interface and see a lot of things happening. I would appreciate a few directions on what I need to look for. I believe that I see lot more traffic then on my entire router network with all the devices attached.

It is my unsubstantiated and unprofessional (n3wb) conclusion that somehow multiple cameras were "affected" either by an attack or by a bad POE and now their networking capabilities are severely hindered require them the reaffirm connections and traffic.

I would like to see if you have some pointers on what to look for in Wireshark. I am not concerned to make a log available as this is now a complete segregated network and I will change it ones its all sorted out anyway. What log/filter would you need to review?
 

heitjer

n3wb
Joined
Sep 10, 2014
Messages
28
Reaction score
4
Guys,
I have been monitoring wireshark and the re-connect issue has not gone away. Today I started to reset my main camera and only test this one. I captured some data and maybe you see something that is unusual. I noticed the black blocks earlier when I had a lot more cameras on there and had hoped that after resetting and only having on this would go away but its still there.

5.200 is Blue Iris Server
5.207 is Dahua (IPC-HFW8232EP-Z) Camera
Both are connected through an older Netgear Gigabit Router that has only the two devices connected.

Let me know if you see something obvious, also let me know what else I could capture.
 

Attachments

Top