Menu
What's new

Multiple Amcrest cams went offline with same error at same time today?!?

C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
Cameras in question are P5M-T1179EW-28MM. I have multiple BI systems setup using these same cameras and no problems until today.

Around 16:00 three cameras go offline a few minutes apart. They give the "no signal" error in BI; won't respond to restart.

Upon http access they present the standard Amcrest login screen, minus the "IP Camera Web Access" text above the credential fields, and the password field populates in plaintext. The login button does nothing however (just in case this is an evil portal I try an incorrect password)

amcrest.png

Changing the credentials in BI results in the error message changing from the "no signal" screen to the "invalid username/password" screen, so the camera is clearly still online, just not with all services running.

The fact that all three went offline a few minutes apart would lead me to believe it was a firmware issue related to a communication problem with their C&C servers, however I'm pretty sure they are all running the same FW and the other five cameras did not go offline.

I was able to restore the feeds by checking "Use RTP/UDP ports" in the BI advanced options, and picking some random port (that shows up nowhere in the camera's config page- iirc the UDP port is set to 37778), so I got the feed back anyway, I guess the cameras allow all incoming RTP/UDP ports to be open? however the http UI still won't allow me to login.

I can't do a hard reboot at this time fyi.

Eventually I was able to login to only one of the three via the http page, and only once, and I was greeted with the usual looking live screen but now it is asking me to download a chrome extension in order to view the live feed, as if this is some grey market HikVision camera from ten years ago using an IE-only activeX plugin! This extension however appears to be a legit one from Amcrest.

Questions:

WTF is going on here? I have been running these cameras for over a year elsewhere with no such issues, then three go down from a weird firmware glitch within minutes of each other?

Do these cameras have some default mode that their webserver switches to if HTML5 is not working or something? That's why it asked me to download an extension to view live video when it knows damn well an extension is not supposed to be needed?

And why was I able to restore my feeds by picking a random RTP/UDP port in BI advanced settings? AFAIK the only UDP port setting in the camera config is 37778. Why can I tell BI to use 37772 and I get my feed back? Do these cameras keep all incoming ports open just in case someone needs to use RTP/UDP?


Any ideas?
 
Last edited:
wittaj

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,848
Reaction score
48,465
Location
USA
Are they isolated from the internet or can they talk to the internet and maybe got a firmware update that messed stuff up?

And yeah, even in 2022, Dahua cams still come with needing to do the IE plug-in LOL, so with Amcrest being Dahua OEM, no surprise LOL.
 
C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
wittaj said:
Are they isolated from the internet or can they talk to the internet and maybe got a firmware update that messed stuff up?

And yeah, even in 2022, Dahua cams still come with needing to do the IE plug-in LOL, so with Amcrest being Dahua OEM, no surprise LOL.
Click to expand...
They are on a separate subnet with a dedicated router, the only thing that gets through that is the BI webserver port which is forwarded from the main router.
 
wittaj

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,848
Reaction score
48,465
Location
USA
Did BI update and take exception to the cameras (although that doesn't explain the camera UI)? One of my cams did a no signal on a BI update and doing the find/inspect brought it back.
 
C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
wittaj said:
Did BI update and take exception to the cameras (although that doesn't explain the camera UI)? One of my cams did a no signal on a BI update and doing the find/inspect brought it back.
Click to expand...
No; using an old v4 and no auto update.
 
S

SpacemanSpiff

Known around here
Joined
Apr 15, 2021
Messages
1,467
Reaction score
2,470
Location
USA
cds333 said:
...
WTF is going on here? I have been running these cameras for over a year elsewhere with no such issues, then three go down from a weird firmware glitch within minutes of each other?
...
Click to expand...
date/time triggered "glitch"?
 
C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
So now it has happened to all the cameras, at two completely different locations. The one thing I did at both locations was set all the cameras to use the default NTP time server and automatically adjust for DST. I did this two days ago.
 
wittaj

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,848
Reaction score
48,465
Location
USA
Sounds like you may have found the root cause! Switch them back and see if it corrects itself!
 
C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
wittaj said:
Sounds like you may have found the root cause! Switch them back and see if it corrects itself!
Click to expand...
Unfortunately I cannot revert the setting because I cannot login. Does anyone know how to remotely reboot these cameras without using the web UI? I did not detect any open telnet port but perhaps there is a way using the RTSP/ONVIF/TCP port?
 
E

EMPIRETECANDY

IPCT Vendor
Joined
Nov 8, 2016
Messages
8,293
Reaction score
23,843
Location
HONGKONG
wittaj said:
Are they isolated from the internet or can they talk to the internet and maybe got a firmware update that messed stuff up?

And yeah, even in 2022, Dahua cams still come with needing to do the IE plug-in LOL, so with Amcrest being Dahua OEM, no surprise LOL.
Click to expand...
There will have a big update this year, the 7 series cameras are all move to the new fw now.
 
J

jack7

Getting comfortable
Joined
Mar 21, 2019
Messages
323
Reaction score
250
Location
USA
Last edited:
bp2008

bp2008

Staff member
IPCT+ Member
Joined
Mar 10, 2014
Messages
12,672
Reaction score
14,015
Location
USA
If this is a bug triggered by a date integer overflowing, you should be able to work around it by changing the date on your NTP server to 2012 or something. Of course that can only be done if you control the NTP server. Hopefully the cameras NTP clients are still working so they can get the changed date.

I was not aware of any such date overflow happening any time soon though (the next one that is likely to affect anything is 2038, January 19th).
 
C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
jack7 said:
You could try using the Amcrest API http command called Reboot.
Click to expand...
Excellent find. Thank you. Now I just have to replicate the malfunction again so I can test whether the API is still accessible.


bp2008 said:
If this is a bug triggered by a date integer overflowing, you should be able to work around it by changing the date on your NTP server to 2012 or something. Of course that can only be done if you control the NTP server. Hopefully the cameras NTP clients are still working so they can get the changed date.
Click to expand...
It was the ntp server and settings that populate the field by default. (clock.isc.org)

I have tried to replicate the malfunction on my own cameras with no success.

SpacemanSpiff said:
Disconnect power source, re-connect power source.
Click to expand...
Again, "remote" is the problem. I suppose I could fail to pay the electric bill for a month or two, let them disconnect the power, and then have it turned back on.

But next time I visit the site I will do just that- install a remote switch on the PoE injector.
 
C

cds333

Young grasshopper
Joined
Jan 5, 2016
Messages
41
Reaction score
6
You must log in or register to reply here.
Top