Inherited a Camera System

[-CapsLock-]

Hello everyone! Long time lurker and I appreciate the throve of knowledge that people have in this forum. New member here who is learning more about IP camera systems and I am in a pickle.

I work as an IT MSP (managed service provider) and recently got a new client who has a camera system which was managed by their previous MSP. Previous provider does not want to work with me so I am stuck figuring things out on my own.

The camera system currently has an issue wherein when the NVR looses power/improper shutdown, most of the IP cameras when the system goes back up, will have the motion detection turned off which then causes the cameras to not record. I find this odd since Motion Detection is turned on if I select the default setting on the camera. This has now happened twice and I had to turn it on on all cameras. To compound the issue I can't seem to find any information about the make and model of these cameras nor the NVR. I can probably check the sticker on the devices but I manage this client remotely and they are not close to me so I will have to schedule a trip. Has anyone experienced this issue or have any information as to what is causing this? I have attached some screenshots for reference.

One of the 30 cameras, they all have the prefix AM5L on the serial number which means they are all from the same manufacturer.



This is the NVR



Pardon for the lengthy first post and please move this if this is not the appropriate location to post. TIA!

 

[tangent]

Put the NVR on a UPS to reduce incidents of power loss.
Switch cameras to continuous recording.
Don't post pictures of full serial numbers of cameras / NVRs, the serial number is often used as part of a NAT traversal remote access system (typically called P2P or easy4ip). Previous MSP could even still have access to the cameras.
RTFM:

https://us.dahuasecurity.com/wp-content/uploads/2023/08/Network-Video-Recorder_Users-Manual_V2.3.1_20230210-Eng.pdf

 

[-CapsLock-]

Thank you for the quick response. I have redacted portion of the serial number. That is my plan for the onsite trip, they have UPS there in their server room so I was a bit baffled that the previous provider did not use it although this is more like a band aid solution since there is still a possibility that power on the UPS will run out and the NVR will shut down eventually. I really need to figure out the root cause and fix it the right way. I was thinking of updating firmware but not knowing the make and model of the hardware in the system makes it a no go. Continuous record is not an option at the moment since it will shorten the playback period available.

Credentials for the NVR and cameras were all changed as well as login that the employees use. I have also looked at the logs and there were no config changes, the only thing that stands out is after they system boots back up the cameras do not record.

 

[wittaj]

Based on that GUI you are looking at Dahua OEM equipment.

Changing the passwords alone may not be enough if the other provider has access to it via a backdoor. So maybe it is being hit from the backdoor vulnerability that will not show up in the logs and is causing the reboots and the issues you have.

You don't want to hear this LOL, but the best solution is to factory reset and turn P2P off.

Also, make all of these changes in the camera GUI, not the NVR GUI.

 

[-CapsLock-]

Thank you for the reply @wittaj

Dahua was my guess as well since the firmware versions loaded on the devices matches firmware from Dahua. I owuld like to update the firmware but not knowing the models make it complicated since I can't cross check which firmware will work for the particular camera.

Where exactly would the backdoor be? I have changed the credentials on the cameras, NVR, and the camera software (SmartPSS). This includes updating the credentials for the users who access it internally.

Unfortunately turning off P2P might not be an option since people internally wants to access the feeds remotely. Are there any other setup/protocols to use that would allow remote viewing of the feeds?

Both time that I had to turn on motion capture was done on the camera level.

 

[wittaj]

Unless the release notes specifically mention it fixing a problem you are experiencing, more than likely it won't fix an issue and may make the camera or NVR worse by removing functionality or worse brick it.

Another thing to consider is that the same model could have different firmware for different chipsets used during the life of that model. So you run the risk of bricking if you do not know what chipset you have.

Here is an example - the HFW3549T1-AS-PV is available in 2 versions of firmware across 3 different chipsets for the same model camera:

• HFW3549T1-AS-PV-S4 uses the HX3XXX-Taurus firmware.
• HFW3549T1-AS-PV and IPC-HFW3549T1-AS-PV-S3 use the HX5XXX-Volt firmware

In this instance, Dahua added an S# designation after the model number (while Hikvision adds a C), but many do not and then you try to update with a firmware not compatible with your chipset and potentially brick it.


Further, it is best to obtain any firmware updates from the vendor you purchased it from so that you do not run into issues. Any firmware you find here or elsewhere is obviously proceed at your own risk. We have many threads here where someone tried an update with a firmware they found on the internet and bricked their unit.

Many units being sold are Chinese hacked units into English that will either brick or go into Chinese upon updating. Some vendors will be upfront and tell consumers that as part of their website, but many do not or the consumer forgets...here is one such example....

It is well documented here the vulnerabilities in any manufacturer NVR and Cameras. And they are rarely updated to close them. And many are backdoor that allows someone to bypass all authentication and/or by using P2P initially sends the user/pw unencrypted.

Most here isolate the system from the internet and not give it access via P2P or UPnP or port forwarding and instead rely on setting up a free VPN that you host or something like zerotier

How to Secure Your Network (Don't Get Hacked!)

Many camera networks are unsecure, even those installed by professionals. This guide gives basic instruction in how to secure a camera network from the most common types of attacks. Perhaps the most important rule of securing a computer network is to not forward ports to unsecure services...

ipcamtalk.com

VPN Primer for Noobs

The internet is a force of nature; no video surveillance system made was designed to be exposed to those forces.. NEVER FORWARD PORTS to your NVR or Cameras, doing such things not only exposes you to severe security problems, but everyone else on the internet too.. Hackers dont want your video...

ipcamtalk.com

 

[tangent]

Continuous record is not an option at the moment since it will shorten the playback period available.

Consider a larger HDD when you connect it to the UPS (do check if the VNR firmware has limits on how big the drive can be.

 

[-CapsLock-]

Thanks for the very informative links. So I guess there are no other other options for remote feeds over the internet then?

I would still like to explore if there are any settings that I am missing that is causing the issue.

The NVR drive usage sits at about 75% so there is plenty of headroom.

 

[tangent]

The camera system currently has an issue wherein when the NVR looses power/improper shutdown, most of the IP cameras when the system goes back up, will have the motion detection turned off which then causes the cameras to not record.

When the power dips below a certain threshold you can get erratic behavior out of many electronic devices. Bits in the RAM can flip. If it dipped just a little lower the camera probably would have properly reset reloading parameters from flash. Short of a UPS the best you can probably do is generate some kind of alert when this happens and then go in an manually reboot the system.

 

[tangent]

Thanks for the very informative links. So I guess there are no other other options for remote feeds over the internet then?

Lots of options, they just involve tunneling into the network somehow, do they have a VPN server?

 

[tangent]

The NVR drive usage sits at about 75% so there is plenty of headroom.

if you switched to continuous then you'd need a bigger drive.

 

[-CapsLock-]

The camera's setting default is that motion detection is on. This is what stumps me, if the cameras reboots and looses its config and loads default settings then it should stay on. I change the camera setting's motion detection to off, save it, then click defaults to load factory default settings and it turns it on.

They do not have a VPN at the moment, and I can try and look into that.

Also to add, config changes do show up on the logs , I can see the log entries when I enable the motion detection or when I try and load the default settings but there are no log entries for when they were "turned" off after the power loss. So I can say with a bit little certainty that it is not caused by someone intentionally turning it off but then again the logs on these systems are not very granular.

 

[tangent]

if the cameras reboots and looses its config and loads default settings then it should stay on.

That's not what's happening.

in a brown out, erratic things can happen and bits can flip. It's not loading defaults.

When it reboots properly, it loads the user settings that are saved to flash. If you do a factory default, it overwrites the user flash.

 

[-CapsLock-]

When I get onsite I will try and properly shut the system down to see if the behavior is the same and if it is not then the UPS will be a sufficient band aid solution.

Any other inputs are much appreciated.

 

[tangent]

When I get onsite I will try and properly shut the system down to see if the behavior is the same and if it is not then the UPS will be a sufficient band aid solution.

You should be able to reboot the NVR remotely if it's still functioning and likely restore operation.

 

[SpacemanSpiff]

The camera's setting default is that motion detection is on. This is what stumps me, if the cameras reboots and looses its config and loads default settings then it should stay on. I change the camera setting's motion detection to off, save it, then click defaults to load factory default settings and it turns it on.

They do not have a VPN at the moment, and I can try and look into that.

Also to add, config changes do show up on the logs , I can see the log entries when I enable the motion detection or when I try and load the default settings but there are no log entries for when they were "turned" off after the power loss. So I can say with a bit little certainty that it is not caused by someone intentionally turning it off but then again the logs on these systems are not very granular.

Often times, the web browser you use can have mixed results in getting the changes to 'stick'. IE is recommended, but many do not have it available. Pale moon web browser seems to be the 2nd choice for many

 

[IAmATeaf]

It might be as simple as a bug in the firmware. For my Dahua 5442 cams there is/was a bug where if the cam was simply restarted it would disable substream 2 and reenable substream 1. This was an issue for me as I wanted/needed substream 2. This had nothing to do with factory defaults but simply was the behaviour as a result of a firmware issue.

 

[-CapsLock-]

Thanks @SpacemanSpiff and @IAmATeaf for the additional info. I will try and make the changes on a different browser just as a precaution.

That is my thoughts too is that the behavior is due to the frimware. The previous provider not being cooperative makes it difficult since I can't get the history of the system, like was this happening before, As far as the users are concerned they said it happened a few times before but cannot provide anymore context as to what the cause of those incidents are.

@tangent I know I can reboot remotely, I just wanted to do it when I am onsite since I have scheduled downtime for my visit and would rather do it in one go rather that setting two different maintenance window.

 

[-CapsLock-]

Just an update, the NVR has battery backup now and when rebooting the NVR the issue does happen. I tried removing power for the cameras and the settings stays but when the NVR reboots random cameras stop recording coz the motions detection is off. There is no consistency either, there are 31 cameras in total and any of the cameras will randomly have motion detection turned off when the NVR reboots. So it is safe to say that the issue is with the NVR. To complicate things I don't see any stickers on it help identify make or model and the only information I have is the one I can see from SmartPSS.