Cam intermittently goes off-line. Why?

[Fastb]

All,

I have a Dahua IPC-HFW4431M-I2 bullet that intermittently goes off line. This happens every 3 to 7 days. Today, there was a bad spell of off-line episodes, so I decided to figger it out (with your help)

At 11 am today, the cam disappeared for two minutes.. Then at 11:46, it really got wiggy. See pic.


The camera operated fine otherwise for several days.

Note: I don't use Motion Detect. I do get the yellow when the cam is acting up, ie: after it comes back, and sometimes just before the cam goes away.

I'm using POE. The cam is connected with ~140ft of Cat 6 cable. My 5 cams are powered by the POE switch - TP-LINK TL-SG1008PE 8-Port Gigabit PoE Switch,IEEE 802.3at/af, Max Output 124W

Only this one cam experiences the off-line episodes.

Any hunches are appreciated!
Fastb

 

[nayr]

Open a non stop ping til it drops again, likely a cabling issue

Sent from my HTC One_M8 using Tapatalk

 

[Fastb]

nayr,

Thanks for the quick response and the suggestion.
I ran a ping for a short while. There is a bit of variation - puzzling. Pings range from 2 to 8ms. See pic
(the cam with dropouts is at 192.168.128.113)



I checked another cam (that doesn't have dropout issues), and found similar results. Pings range from 1 to 8ms.



But another cam had stranger results (tho it doesn't suffer from dropouts)
Of my 5 cams, this one is the worst, where pings range from 1ms to 91ms:


Q1: Is this range of ping times a problem?
Q2: The results seem to eliminate the dropout-camera-specific-cabling as root cause, since the ping results are similar to the non-drop-out cams.
Q3: You mentioned "Open a non stop ping til it drops again". The drop out may take days to occur again. And when the dropout occurs, I'd have to get to the DOS screen quickly to Ctrl-C out before the ping results scroll out of the DOS window.... Maybe there's a better way? Or is there an error in my thinking/conclusions?

Thanks again,
Fastb

 

[nayr]

leave the dos window open with pings, and after the next time you encounter a dropout on your NVR cancel the ping and look at the results.. 0% Loss is the key, if you get any packet loss then it was either dropping packets or rebooting.. both can be caused by poor terminations/crimps on your cabling and appear sporadically.

cabling issues come/go.. were trying to ensure the problem is not with the recorder but instead something specific to the camera's network or hardware.. After this we'll have you move camera to a bench or with another camera and try again to see if the problem follows the camera or cabling.

 

[Fastb]

Good idea and t-shoot plan.
I get an email alert when the cam goes off-line, IF the snapshot reaches my phone.
Too bad gDMSS doesn't support a push notification for cam off-line. Push is so much more dependable....

 

[cor35vet]

To check if the camera reboots telnet into it and run "cat /proc/uptime", the first number is the number of seconds the camera has been online.

 

[Fastb]

Cam went off line for 3 minutes today. I caught it.

The constant ping test didn’t show any problems. Pings continued while cam was reportedly off-line.

I received several emails from my NVR, reporting the cam was off line. The NVR log recorded when the cam went off line, and when it came back 3 minutes later.

I haven’t delved into learning how to telnet into the cam, to see how long the cam thinks it’s been on line. Thanks for the suggestion @cor35vet!

Interestingly, the cam missed a tripwire event 30 minutes earlier. It’s usually pretty reliable. The event was my wife driving down the driveway, so it’s an event that is common, and commonly caught by the cam tripwire. This may be unrelated, or may indicate the cam was going into the weeds. 30 minutes later, it was reported off line.

Another clue: In playback, the cam’s video dropped out, returned, dropped, returned, then dropped for good. (for 3 minutes). Maybe the cam is struggling? It’s recording main & sub stream. Settings below.



An additional clue for “struggling cam”: Sometimes when I surf into the cam, I get “Resources are Limited” error, and no image. 5 secs later, the image comes up.

Yet another clue: using the browser to reach the NVR, sometimes when I switch tabs to call up the problem cam (to check encode settings, IVS settings, etc), I get an error message “Channel 5 failed to get configuration”. This is intermittent.

My T-Shoot Plans:

- Reduce resolution of main stream. I have it at max, to read license plates.
- Learn how to telnet to the cam, and check the “uptime timer”
- Keep the continuous ping test running
- Maybe bring the cam inside, and use a different shorter cable than mmy 100ft Monoprice Cat6 cable.

Any ideas? Any other tests?

Thanks in advance!

Fastb

 

[cor35vet]

Set your framerate to 10, bitrate to 6kbps, codec to H265, disable substreams and everything else you don't need.
Just try to reduce load on the cam - the CPUs in them are shit. (Hi3516 800MHz ARM Cortex A9)

 

[Fastb]

cor35vet,

So you're thinking the cam is struggling. Thanks for confirming my hunch. Testing will really confirm it!

This cam, w/ 12mm lens, was intended to get good detail so I could read license plates as cars pulled into the driveway.
Moving cars, hence;
- 20 FPS frame rate (max is 25)
- high resolution
- high bitrate (assuming that provides a better recorded image)

Q: Knowing what I want to accomplish with license plates, would you tweak your recommendations any?

Re: substream
I find it very useful to view my cams remotely, on my cell phone. So I'll likely keep that. But I could;
- lower FPS from 10 to 8
- lower I Frame interval 20 to 10

Thanks again,
Fastb

 

[nayr]

if your doing LPR you dont really need a high framerate unless your shooting for a small mask w/high speed traffic..

I tested pretty good resuts with as low as 5-6fps for manual reading w/my setup.. depends on how sharp of an angle you have too I suppose.

disable the 3rd substream, any features your not really needing and might be enabled.. try to keep web usage down and use SmartPSS for viewing.

You mainly want to get the bitrate down, and to maintain quality you drop the FPS

 

[Fastb]

Yesterday, I made changes based on guidance from @cor35vet (before I saw nayr's guidance.)
I set the Bit Rate to 6144, selected from the pull down menu. Today, Bit Rate shows as "customized" with a Bit Rate of 2816. Odd. Maybe this changed after the cam went off-line today....


So the cam went off line today.



Some findings:
h.265 encoding on IPC-HFW4431M-I2
- h.265 Recorded video (main or substream) is not visible in the web interface. (Playback mode)
- h.265 Recorded video (main and substream) is visible in Smart PSS (Playback mode)

Tripwire (IVS)
- 7:34 am event did not cause a tripwire event.
- 8:35 am event did cause a tripwire event.
- Unusual, the tripwire usually works pretty consistently


When cam was reported as Off-line
- Main stream was visible in SmartPSS and web interface.
- Substream not visible in SmartPSS or web interface.
- In the Live View, using web interface, the substream isn’t visible, but data is coming in, judging by the changing kbps figures that change regularly (just as when there is an image)

When the cam went off-line:
-At the time, the NVR web interface was open, but in “Playback” mode. And no playback was underway
- The running ping test caught a problem. Longest ping was 2.8 seconds! I bet this is when the camera went away.
- The "ping statistics" summary shows 9 packets were lost. 6 packets were lost just before the "off line" event. (see jpeg)
- I assume 3 packets were lost earlier, but the evidence scrolled off the DOS window.


After the cam came back on-line:
Main video stream appears fine. Substream has problems.
Hunch: The substream being produced by the cam is corrupted.
Clue: Substream can’t be displayed “Live” by the web interface, SmartPSS, or gDMSS.
Clue: Substream in gDMSS shows a greenish, scrambled view for 2 seconds, then the image stops (black image). gDMSS chokes, and returns to “Live Preview” screen, as if I hadn’t chosen to view the ca at all.
Clue: I "nudged" the camera by changing substream compression from H.264H to H.264. The substream was then fixed, and could be viewed!

Next Steps:
- Avoid H.265. This model was an early camera to use H.265. It might be buggy.
- Investigate a f/w upgrade. I got the cam on ali, so I'll have to be careful.
- Run the test again, w/o web interface open, or Smart PSS running.
- I tweaked the encoding parameters, using guidance from both cor35vet and nayr. See below.

Any suggestions?
Thanks again for your awesome help!
Fastb

 

[nayr]

@cor35vet has firmware in the mod kit thread for this camera thats been hacked to a multi-language.

the camera's dsp is very weak, being a chinese camera they come with a comparatively weak CPU and that accounts for why they are so much cheaper than the international variants.

trying to do both IVS and H265 I suspect is going to be too much for this hardware; pick one or the other.. I think you want to drop h265

 

[Fastb]

nayr,
And presumably, that multi-language f/w started with a newer base.....
Cuz I don't need multi-language.....

@cor35vet,
Do you have newer f/w by any chance?

Fastb

 

[nayr]

Dahua Firmware Mod Kit + Modded Dahua Firmware

 

[cor35vet]

nayr,
And presumably, that multi-language f/w started with a newer base.....
Cuz I don't need multi-language.....

@cor35vet,
Do you have newer f/w by any chance?

Fastb

View attachment 13767

Yeah, latest one here: Dahua Firmware Mod Kit + Modded Dahua Firmware

The US models are the same hardware though lol. Hi3516.
Hi3516 also has native HEVC encoder, so it doesn't matter if you pick H264 or H265, should be about the same speed.

 

[Fastb]

nayr,

Thanks. And thanks for the link to the thread.

My plan:
- Learn how to telnet in to cam
- Learn how to find the latest f/w image from cor35net's site. (github didn't have a solution jump out to me)
- I've been an EE involved in the h/w side of things, so I'll have to do a crash course. We did use Putty and github, but I was in the peanut gallery and not immersed in that area.
- Find a suitable image for my cam. Fortunately, cor35vet worked on the same cam (IPC-HFW4431M-I2), though a while ago.
- Telnet appears to let you see if the re-flash was succesful. If not, you can revert to the previous version without bricking the cam. Very attractive. I don't want to deal with unbricking a cam.

Of my 5 cams, this is the first I bought from ali. I'm getting an appreciation for the "save money" versus "no safety net by a US vendor". No regrets at this point. I enjoy steep learning curves (esp if it save $$)

No response needed. I will do more reading and learning.

Thanks again,
Fastb