Seeing non public IP addresses that don't exist connecting to my DVRs via admin login

[InnerSanctum]

Hey guys

I am seeing some very weird successful admin login connections to the DVR via the admin account. The DVR is showing these connections are coming from NON routable, internal IP addresses, which of course is impossible. Please note: The internal network is 10.1.10.x 255.255.255.0

Examples:
Successful admin logins from IP addresses and subnets that dont exist on the local network

192.168.1.70
10.108.147.45

Any ideas whats going on? Ill have a network sniffer and bandwidth monitor installed later in the week to get to the bottom of this. I was hoping you guys might have some ideas. Please note, the admin login/password has been changed friday night 3/13 when i found people from external IP addresses connecting from all over the world.

thanks

 

[bp2008]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

Seems to me that either those subnets do exist on your local network or the DVR's logging can't be trusted.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

Its a very small network. Ill try to re-ip myself to one of those subnets and try to ping those IP addresses but i can tell you for sure the only DHCP server on that network is for the 10.1.10.x subnet. The network itself has 2 computers, a couple of tablets, and wireless access points (running WPA2).

They are having issues connecting to the DVRs remotely from their home tablets with constant timed out connections and I'm troubleshooting.

 

[alastairstevenson]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

What other devices are on your local network that may have been compromised by the external access that you created?
It's quite normal when malicious activity is in play that source addresses are spoofed.

As a quick check, do a 'netstat -a' on as many active devices as you can on your local network, and check for any suspicious destinations.

*Edit* Traffic congestion for no obvious cause can be an indication that a device on the network has been compromised and is operating as a scanbot. It's been very common in *nix environments for some months following several easily-exploited high-severity vulnerabilities.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

What other devices are on your local network that may have been compromised by the external access that you created?
It's quite normal when malicious activity is in play that source addresses are spoofed.

As a quick check, do a 'netstat -a' on as many active devices as you can on your local network, and check for any suspicious destinations.

*Edit* Traffic congestion for no obvious cause can be an indication that a device on the network has been compromised and is operating as a scanbot. It's been very common in *nix environments for some months following several easily-exploited high-severity vulnerabilities.

Excellent point. There are 3 devices on the network. I will check the network connections running off all 3 to make sure there are no crazy connections. Still doesnt really explain how the logs show non existant IP addresses successfully connecting to the admin console (i can see failed admin logins from the same non existent subnets). i mean those IPs shouldnt even be able to communicate with the DVR since the DVR is on the 10.1.10.x subnet to begin with and there is no router on the network that would route traffic to the DVR from a different subnet.

The two DVRs were operated with default admin login and password for a while before i found outside connections to it. I immediately changed the admin password and rebooted the devices.

Is there a possibility that once they got in via default login/password, they were able to mess with the linux OS on the DVR? I mean having the admin/pass for the web console shouldnt allow them to mess with the OS would it?

Heres my guess. They were able to use the admin login/pass and get into the Linux OS on the box. You're definately right about the spoofed addresses. Interestingly enough, when the admin login/pass was default, id see external IPs that were controlled by the Department of Defense among other weird IPs connecting to it.

I will be replacing their residential firewall with a juniper firewall so I'll have SNMP bandwidth monitoring and also sniffing. Ill get to the bottom of this soon and post here.

Any suggestions are welcome.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

I have checked all 3 computers on the network and dont see any crazy network connections (i used tcpview instead of netstat, its prettier!)

Once i get that sniffer between the local network and the firewall, im sure ill nail it down.

I have a feeling that the DVR firmware is installed with a version of Linux with a known vulnerability that gets exploited the second ports are forwarded to it. I cant see knowing the DVR admin login/pass is enough to screw with the OS.

Ill find out more to this mystery soon enough and post here.

 

[code2]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

I have checked all 3 computers on the network and dont see any crazy network connections (i used tcpview instead of netstat, its prettier!)

Once i get that sniffer between the local network and the firewall, im sure ill nail it down.

I have a feeling that the DVR firmware is installed with a version of Linux with a known vulnerability that gets exploited the second ports are forwarded to it. I cant see knowing the DVR admin login/pass is enough to screw with the OS.

Ill find out more to this mystery soon enough and post here.

I would be removing a the ethernet cable till you figure out who and what is sneaking in the back door. Just to prevent anything from being snatched up

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

I would be removing a the ethernet cable till you figure out who and what is sneaking in the back door. Just to prevent anything from being snatched up

My friend, you are preaching to the choir. Tell that to the small business owner who dropped thousands of dollars to have it installed. I'm going to need definitive proof of what is going on and with the current residential firewall they have, i get NOTHING from it. Ill be installing a juniper firewall there for SNMP and attaching a hub between the firewall and internal switch so i can sniff all traffic.

Im sure what I find will be an eye opener. The installer is probably going to have to re-flash the OS and reconfigure once i prove that its been compromised. Im just worried this bottom basement DVR has an OS with a known vulnerability that exists on it which makes this entire process useless.

I can't wait to find out whats going on. Ill post here once i learn more.

 

[code2]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

My friend, you are preaching to the choir. Tell that to the small business owner who dropped thousands of dollars to have it installed. I'm going to need definitive proof of what is going on and with the current residential firewall they have, i get NOTHING from it. Ill be installing a juniper firewall there for SNMP and attaching a hub between the firewall and internal switch so i can sniff all traffic.

Im sure what I find will be an eye opener. The installer is probably going to have to re-flash the OS and reconfigure once i prove that its been compromised. Im just worried this bottom basement DVR has an OS with a known vulnerability that exists on it which makes this entire process useless.

I can't wait to find out whats going on. Ill post here once i learn more.

So what ever happened with this

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

So what ever happened with this

I have not forgotten about this guys. I am deeply appreciative of your help and suggestions.

I am still seeing weird spoofed addresses appear in the remote login logs for "admin" but they are SIGNIFICANTLY reduced since I changed the admin password. As in 10 to 1.

I am in the process of installing a Juniper firewall on their network to allow for SNMP bandwidth monitoring and also sniffing.

You will get a full report once I get it all setup.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

I have not forgotten about this guys. I am deeply appreciative of your help and suggestions.

I am still seeing weird spoofed addresses appear in the remote login logs for "admin" but they are SIGNIFICANTLY reduced since I changed the admin password. As in 10 to 1.

I am in the process of installing a Juniper firewall on their network to allow for SNMP bandwidth monitoring and also sniffing.

You will get a full report once I get it all setup.

So I was able to install a Juniper firewall at the location that gave me full bandwidth monitoring via SNMP. From what I can see on that end, the bandwidth at the location is barely touched. This means I dont have strangers connecting to my feed and blowing my outgoing bandwidth (we have a 100/20mbps comcast line @ the site). So now for the next part.

I've installed a 10/100mbps hub between the main switch and the firewall internal port. I've connected a PC to that hub and started to sniff all incoming and outgoing internet traffic. If anyone is interested, the software I used and like to use for sniffing and network stats is Colasoft Capsa 7. Great software. Parses out the traffic info a lot nicer graphical representation then a nakked wireshark capture. You can grab the latest version via bittorrent if your company is to broke to buy it.

So I check the DVR logs from the time i turned on the sniffing software till now. I am still seeing "weird" IP addresses in the DVR log. Mostly weird internal/non routable IP addresses. When I check my sniffer for ALL connections that access the DVR from the internet, I am seeing nothing but normal connections from the area that the DVR is installed in (florida).

There is no way my sniffer would be missing traffic coming from the internet to the DVR or vise versa. There is also no other way out of the network other then my firewall....

Im forced to come to the conclusion that this DVR is a massive piece of shit and entering garbage into its own logs. This doesnt surprise me considering other experiences that I've had with this DVR and the company that makes it (or rebrands it from hikvision which is the case)

I'm not sure if ive mentioned it yet but I have 2 LTD8316T-FT manuf by LTS
Platinum Advanced Level 16 Channel HD-TVI DVR 1U
LTD8316T-FT LTS H.264 Dual-stream Video Compression 16CH TVI DVR Support Both HD-TVI and Analog Cameras- Directron

Thats what you get when you install basement DVRs I guess. Thank you all for your time and input.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

I would like to note something here. When I first started troubleshooting the DVRs at this location, in addition to the weird internal IP addresses I was seeing in the remote admin logs, i also saw the following:

29.2.7.203 Ohio Columbus Dod Network Information Center
30.3.55.65 Ohio Columbus Dod Network Information Center
29.2.128.221 Ohio Columbus Dod Network Information Center
30.2.58.127 Ohio Columbus Dod Network Information Center
29.2.184.7 Ohio Columbus Dod Network Information Center
51.1.83.188 England London Uk Government Department For Work And Pensions
100.90.226.167 California Los Angeles (westchester) Internet Assigned Numbers Authority

After I changed the admin login/password from the default to something un-bruteforcable, those connections have completely stopped. At this point I see legitimate local traffic (florida) and a crap load of random internal non/routable IP addresses.

 

[fenderman]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

So I was able to install a Juniper firewall at the location that gave me full bandwidth monitoring via SNMP. From what I can see on that end, the bandwidth at the location is barely touched. This means I dont have strangers connecting to my feed and blowing my outgoing bandwidth (we have a 100/20mbps comcast line @ the site). So now for the next part.

I've installed a 10/100mbps hub between the main switch and the firewall internal port. I've connected a PC to that hub and started to sniff all incoming and outgoing internet traffic. If anyone is interested, the software I used and like to use for sniffing and network stats is Colasoft Capsa 7. Great software. Parses out the traffic info a lot nicer graphical representation then a nakked wireshark capture. You can grab the latest version via bittorrent if your company is to broke to buy it.

So I check the DVR logs from the time i turned on the sniffing software till now. I am still seeing "weird" IP addresses in the DVR log. Mostly weird internal/non routable IP addresses. When I check my sniffer for ALL connections that access the DVR from the internet, I am seeing nothing but normal connections from the area that the DVR is installed in (florida).

There is no way my sniffer would be missing traffic coming from the internet to the DVR or vise versa. There is also no other way out of the network other then my firewall....

Im forced to come to the conclusion that this DVR is a massive piece of shit and entering garbage into its own logs. This doesnt surprise me considering other experiences that I've had with this DVR and the company that makes it (or rebrands it from hikvision which is the case)

I'm not sure if ive mentioned it yet but I have 2 LTD8316T-FT manuf by LTS
Platinum Advanced Level 16 Channel HD-TVI DVR 1U
LTD8316T-FT LTS H.264 Dual-stream Video Compression 16CH TVI DVR Support Both HD-TVI and Analog Cameras- Directron

Thats what you get when you install basement DVRs I guess. Thank you all for your time and input.

Those are not manufactured by LTS, lts just rebrands hikvision...they are generally good NVR's...never used a hybrid/trybrid model..

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

So I've never worked with DVRs before but I now know a lot more then I ever wanted to. I know that LTS will not respond to any tech questions sent to them and I've never received a call back from them after leaving a message. I also know that their SNMP settings doesnt return any useful info because it was never setup to. When I point my SNMP software at a piece of network hardware, i will get back all types of great information. CPU/Memory/Network card bandwidth usage. The info that this hardware give via SNMP is absolutely useless. You can see my thread here. Working with this hardware has been a study in frustration.

http://www.ipcamtalk.com/nvr-s-dvr-s-and-computers/3090-snmp-ltsecurityinc-com-dvrs.html?posted=1#post26562

 

[fenderman]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

So I've never worked with DVRs before but I now know a lot more then I ever wanted to. I know that LTS will not respond to any tech questions sent to them and I've never received a call back from them after leaving a message. I also know that their SNMP settings doesnt return any useful info because it was never setup to. When I point my SNMP software at a piece of network hardware, i will get back all types of great information. CPU/Memory/Network card bandwidth usage. The info that this hardware give via SNMP is absolutely useless. You can see my thread here. Working with this hardware has been a study in frustration.

http://www.ipcamtalk.com/nvr-s-dvr-s-and-computers/3090-snmp-ltsecurityinc-com-dvrs.html?posted=1#post26562

If you want tons of options you need a pc based solution...Did you buy the NVR from LTS? if not they probably dont support it...just like hikvision will not support products unless purchased from an authorized retailer.

 

[alastairstevenson]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

Leaving aside the private non-publically-routed internet addresses:
So you've had successful admin logons from public internet IP addresses showing up in the DVR log in abundance.
You've changed the passwords to strong versions. Were they originally at defaults / weak while directly accessible from the internet?
The volume of admin attempted logons from not-known-to-you public internet addresses has gone to zero.

What makes you think the internals haven't been seriously messed with by very smart people after you exposed it to the internet?
Maybe that's the cause now of the strange log entries. Some are out to do bot stuff with other's equipment, some are out for the lulz.
This forum is packed with people using Hik NVRs who are interested in what the devices do and how they work internally.
I believe your symptoms of admin logons from illogical IP addresses haven't been mentioned by anyone else, so the cause is unlikely to be the original as-supplied firmware.
Apart from your packet sniffing and log searching, did you ever do a 'netstat -a' to look for any suspicious connections?

What I would do to close off the issue is to obtain a legitimate copy of the system firmware, bootloader and all (normal firmware update / replacement would not touch the bootloader) and get the machine back to an as-manufactured state.
I'd be willing to bet it would then behave quite normally.
I have 2 Hikvision NVRs, and have done a fair bit of looking around internally, and sniffing the traffic, and they are both absolutely fine, no worries, even with my IT Security hat on.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

FENDERMAN: Just so you know, I was brought into the situation to figure out remote iphone/ipad/android time out issues. The installer walked away from the project and is refusing to speak to me after being put on the spot for bad design decisions and my challenging his claims that we dont have enough bandwidth at the site to handle the traffic (we have 100/20 and rarely is the upload maxed out now that I have bandwidth monitoring setup). So pretty much I have no information on how the units were purchased and where they were purchased which does suck indeed. From what I have seen, the software LTS says to install on windows PCs (NVMS7000) doesnt work what so ever on windows 8.1. It will constantly crash. I have also found that these NVRs are not compatible with the ipad/iphone/android software that LTS says on their website to use (also called NVMS7000) (the feed will consistently time out). Its been a really horrible and frustrating experience all around. Luckily I found some good replies on this forum. Wish LTS had their own forum where users can discuss issues...

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

Leaving aside the private non-publically-routed internet addresses:
So you've had successful admin logons from public internet IP addresses showing up in the DVR log in abundance.
You've changed the passwords to strong versions. Were they originally at defaults / weak while directly accessible from the internet?
The volume of admin attempted logons from not-known-to-you public internet addresses has gone to zero.

What makes you think the internals haven't been seriously messed with by very smart people after you exposed it to the internet?
Maybe that's the cause now of the strange log entries. Some are out to do bot stuff with other's equipment, some are out for the lulz.
This forum is packed with people using Hik NVRs who are interested in what the devices do and how they work internally.
I believe your symptoms of admin logons from illogical IP addresses haven't been mentioned by anyone else, so the cause is unlikely to be the original as-supplied firmware.
Apart from your packet sniffing and log searching, did you ever do a 'netstat -a' to look for any suspicious connections?

What I would do to close off the issue is to obtain a legitimate copy of the system firmware, bootloader and all (normal firmware update / replacement would not touch the bootloader) and get the machine back to an as-manufactured state.
I'd be willing to bet it would then behave quite normally.
I have 2 Hikvision NVRs, and have done a fair bit of looking around internally, and sniffing the traffic, and they are both absolutely fine, no worries, even with my IT Security hat on.

Thank for putting time into replying to me. The weird DOD external IP addresses (and others) are no longer showing up in the logs after I changed the admin password. The weird local connections have persisted. Now is there is a possibility that something was done to the operating system while the admin password was default? Absolutely! However, there is no way to hide traffic from me passing through the firewall to the internal switch. I have a hub installed with my PC installed on it sniffing ALL traffic passing each way. I would see a high volume of total traffic going to easily seen external IP addresses with my sniffer sniffing. In other words, even though i see weird internal IP addresses showing remote admin logins, im not seeing the associated traffic reflecting in my sniffer. The traffic just doesnt exist. Even if it was encrypted, i still would be able to see large amounts of data flowing to outside of florida external IP addresses.

I would love to somehow obtain a legitimate copy of the system firmware, bootloader and all, however, the installer walked away from the project and I have no idea where the units were purchased and what my support options are. To tell you the truth, im not even being paid for this work, im just helping a friend who owns the business that got the DVR installed on.

At this point I am not worried about strangers getting access to the feeds. My sniffer just doesnt show me thats happening. Its just super weird to see it in the DVR logs.

 

[alastairstevenson]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

I would see a high volume of total traffic going to easily seen external IP addresses with my sniffer sniffing.

Yes, understood. Just thinking that the weird behaviour in the logs is now an isolated remnant of something yet to be discovered.
It's amazing what the bad guys can do even with automated methods. Look at the rash of fairly recent Linux high-severity vulnerabilities that affected so many users that in ignorance of the threat, port-forward their NAS boxes to the internet.
Some very clever hacks, that persist over initialise / reinstall.

I had forgotten - or didn't read - that you've come in 'after the event' on this to sort it out.

 

[InnerSanctum]

Re: Seeing non public IP addresses that don't exist connecting to my DVRs via admin l

Yes, understood. Just thinking that the weird behaviour in the logs is now an isolated remnant of something yet to be discovered.
It's amazing what the bad guys can do even with automated methods. Look at the rash of fairly recent Linux high-severity vulnerabilities that affected so many users that in ignorance of the threat, port-forward their NAS boxes to the internet.
Some very clever hacks, that persist over initialise / reinstall.

I had forgotten - or didn't read - that you've come in 'after the event' on this to sort it out.

Yes sir! I came in "after the event". And you're right, these compromises can be pretty advanced. I have sniffing turned on and check everyday to see where a majority of the traffic is going just to make sure the feeds arent going outside of florida. I also have the ability (the sniffing software is pretty damned cool) to see all incoming connections to the DVR during the sniff to see what is connecting or trying to connect to the DVR. Unfortunately, I have no way of "flashing" the hardware with a new OS or would even know where to start. Pretty difficult situation. I'm going to post a new post here to discuss the time outs Im seeing in the DVR ipad/iphone/android software.