Very strange - 4 Dahua/QSee cameras drop connections

[BeachCam]

I've got an old QSee NVR with ten IP cameras circa 2014. Six of the cameras are Hikvision and four are the original QSee (Dahua) cameras that came with the NVR. All the cameras are POE, hardwired, and routed through a half dozen hubs/switches on a gigabit network in my house. The whole system has worked solid for six years. I am recording all ten channels 24x7 and not doing anything significant at the camera level. No motion detection. no local recording or snapshots. Very simple 24x7 NVR.

Until last week. I noticed that my four QSee cameras were offline. So, I went around pulling their POE plugs to reboot them. They came back up online and I was happy, but then noticed the next day all four were out again. Huh? So I reset a few of them and then realized they went offline about an hour later. Only the QSee and not the Hikvision cameras, many/most plugged into the same POE switches and routing through the same hubs.

So, today I dug into the issue hard. I have a network engineer background and computer expert, so I wasn't going to let a little camera defeat me. but, here I am.

I focused on one of the cameras. I changed all the passwords. Removed it from the DVR list. ensured every possible function was turned off on the camera, etc. Confirmed all files deleted. Ensured TCPIP only and a static IP address. When I power cycle it, it stays online and visible on the network for 30-60 minutes then disappears from the network (no ping), but the power stays on (IR lights visible). Oddly, it only happens to the four QSee cameras equally and consistently. I looked for obvious things like conflicting IP addresses or other network anomalies, but nothing stands out. The logs on the camera also don't indicate much. Just the startup and login/logouts.

My sense is there has to be something in the firmware or unique to the cameras, such that all four are failing in the same way at the same time.

Any insights, clues, or additional things to check would be appreciated. --besides the obvious answer to throw out those four cameras and get new ones. The QSee cameras were never well supported OEM Dahua's, but I think the model is QCN8026B and the firmware is 2.400QS00.0R build 2015-05-23.

 

[samplenhold]

I had a similar issue. Are your cams using/writing to SD cards? If so, they may have gotten full. I had this issue and reformatted the cards and no more problem.

Another thing that I see here on this forum is that when someone reports of wired cams being intermittent or going offline, about 90% of the time it is due to connectivity, generally the cable or connector. Are these outside cams? Did you properly weatherproof the connections? Have you tried pulling the cam down and using a new, short, pre-made cable to connect it directly to your NVR as a test to see if it continues to go offline?

 

[BeachCam]

Thanks for the reply @samplenhold

This one is really odd. The cams do not have SD cards and there is no local recording or image capture. All of my cams are external, but I've got at least 5-6 different hubs/switches for different sides/floors of the house. The closest one to my main router is only a six foot cable run and goes directly to my main router. The others all go through multiple switches, but there is no one common run or switch to all of them (except virtually through the same main router). I could see one switch or cable failing or even two failing, but the odd thing is all four of the same cameras are failing with the same behavior. They stay online for 20-60 minutes then disappear from the network. I have turned everything off on the cameras, changed users/passwords, and power cycled every part of the network, the DVR, router, switches, etc. multiple times. My next step is to isolate one camera on one switch with no other connections.

At one point I thought it might be time related. DST kicked in early due to settings and the cameras are using a NTP server that connects every 20 minutes. That seemed logical if the NTP server was bad, but I've disabled all NTP, DST, and other time settings and they still fail after about 30 minutes. Again, six other Hikvision cameras on the same network (many on same switches) have no issues. I'll keep trying to isolate the issue. These cameras won't defeat me.

 

[samplenhold]

So you have cam traffic running through a router? That is generally not advised. Is your router assigning IP addresses to the cams or are they static IP addresses?

Do the cams have access to the internet? We generally state that cams should be isolated from the internet. Is it possible that the qsee cams have been hacked?

 

[BeachCam]

No. No cam traffic is running through the router to the internet, but the router is the primary router for the house, so by definition, all traffic needs to flow back through it to get routed to the NVR. All IPs on the cameras are on reserved static addresses. I've tried switching a problem camera over to DHCP, also tried a different static address, but no difference.

My first (and second) thought was that they might have been hacked, even though they are not exposed to the outside. Only the NVR is exposed to the outside. I've got a few other devices that are port mapped to the outside, so if someone did hack the QSee cams it would have to be indirectly through a different primary hack (not impossible, but hard).

 

[wittaj]

Get the cameras off the router as consumer grade routers are not known for being efficient at passing data so you are probably hitting an issue with data bandwidth regardless of what the router is rated for, especially if it is a wifi router. Except for high end routers, most are not made to pass through heavy data hog continuous streaming of video cameras 24/7. Unlike Netflix and other streaming services that buffer video, these cameras do not so if there is a dropped packet or loss of signal, it can wreak havoc on the entire system.

As a general rule, surveillance cameras should not pass through a router. Most of us here use a switch to connect all the cameras to.

How are you sure the cameras are not exposed to the internet if they are passing through the router?

 

[IAmATeaf]

Sounds me more like tired internal electronics within the cams, possibly. Crack one open and have a look at the main board, check voltages when it is working, then wait for it to stop and then check again to see if anything has changed.

 

[BeachCam]

Thanks for all the ideas. Here is an update with progress....

I isolated one of the cameras on a single POE switch. Power cycled it and then checked in on it at 60 min and 120 min with no issue. So the problem seemed to be external to the camera. On this specific camera, I had already changed all the user names and protocol ports, so I assumed it was 100% disconnected from the NVR. I confirmed this on the NVR side, as it just showed "loss of signal".

But... when I did a hard delete of the camera out of the NVR list (did for two cams) the off-line disconnect resolved itself. So, something in the NVR was causing the four cameras to disconnect after about 30 minutes. Previously, the four cameras were all connected to the NVR in a "managed" capacity using UDP.

So, I deleted all the See cams from the NVR and I readied them using HTTP with onvif enabled on each camera. So far, they are all back working well and no disconnects in four hours.

The question still remains... what changed? I've had no hardware or firmware changes on the network, no change on the NVR in years, and nothing else I can think of unusual. All of the cameras are connected through switches, although some of the switches to hub through the lan side of my router. I've also got a 50 port managed Cisco switch, so I can generally see traffic and network utilization in the house. Every device is gigabit and there is nothing close to saturation or latency that I can see.

Thanks!

 

[samplenhold]

Sometimes shit happens. Things get old. Electronics need to be rebooted. Maybe in certain instances, cams need to be deleted and re-added.

 

[BeachCam]

Maybe in certain instances, cams need to be deleted and re-added.

Yeah, I think this was the cams way of telling me it is their time to be physically deleted and re-added with new ones

 

[BeachCam]

So... the mystery with my four QSee cams continues. 40 days after resolving the connection issue and getting them back online, all four of the cams shut down again. But... again... only the four QSee cams on my (vast) home network and not the other six IP cams. The QSee cams are all on different segments, going through different switches, and share segments with the other cameras in the house. I have a managed Cisco switch and network health monitoring on all the segments, so I know the network is fine.

All four cameras dropped off the network (and stopped recording on the NVR) at 2:46am on 4/21. The other six cameras on the network didn't miss a frame, nor were there any other anomalies on my network at the time.

When the cameras drop they appear to be on (IR LEDs at night), but they have no network connectivity (no ping response), although other cameras on their local POE switch have network response just fine, so it is not switch related.

I have a strange suspicion that the cams dropping is somehow related to QSee going out of business and shutting down their servers at the end of January. My first outage of these cameras was 40 days after QSee shut down. That said, there is no setting on the cameras or on the DVR that should connect back to QSee servers, so I cannot think of a technical reason that QSee shutting down servers would impact these cameras. I also think there would be more reports of similar issues if that was the case. I have DDNS, Network time, and all other "outside" network features turned off on the cameras. Also, there is no way all four cameras are perfectly synchronized in time, so it must be something external that is triggering them to drop or crash.

My first suspicion would be the QSee NVR, since it is the only device that is (supposed) to be talking to the four cameras.

Is anyone aware of any settings or features that would want to "phone home" (e.g., points to QSee shutting down servers) or something in the NVR that would send a shutdown signal? It is clearly something unique to these specific cams/firmware, as they are all failing at the same instant.

Thanks for any ideas.

Mike