IPC-Color4K-T180 troubleshooting

[xcore]

Hi,

I have a pilot camera installation: IPC-Color4K-T180 (f/w 3.120.0000000.36.R, Build Date:2023-08-28) on the PFA122 fixture.

Since it's a pilot, I'm not looking on my ZoneMinder feed very often, basically my goal was to establish a stability baseline, and, well, the camera has failed me already.

This morning I had no feed since about 4am. Camera was unresponsible to ping/web UI.
Further digging shown that it was not the first time. Please see attached graph. The intermittent access is very much concerning.



It's a CPU load graph gathered by SNMP from the camera. I can tell my Zabbix installation is not a reson you see the blanks.

Steps I performed to troubleshoot:

1. Interface status on the switch was showing the same downtime as Zabbix was, about 4 hours
2. Interface was enabled, but not UP
3. PoE draw was present on the interface, usual ~5W
4. Resetting the PoE on the port helped - camera booted up and become responsible again with feed present in both VLC and zoneminder

Logs on the cameta (I use syslog) are not informative at all. Last message from the camera was before it stopped working.

I'm positive its not a power outage or anything else unrelated to the camera.

Camera was purchased back in June-ish and only installed mid-Nov/Dec.
Fixture is installed on the brick wall, incoming non-metallic conduit is water-tight, fixture plugs are sealed, so it's not a water issue.
PoE is supplied from a Juniper EX3300 switch, so both of PoE part and switching part are fine.

It's in Texas, so before recent snow and current cooler than usual weather we might have had a couple of rains and some sun, nothing unusual or very hard for the camera enclusure (supposedly).

Now to the bad stuff: since it's a pilot project, I'm very concerned. In my view, it's not something I should expect from anything I put outside for years to work seamlessly.

Did anyone had same experience with this/other EmpireTech camera? Was it an isolated incident? How did you solve it?

I have to admit, SNMP and Syslog information camera provides are not what I used to. It would be great if I could take a look into the logs and figure out it by myself, but as I said, there's nothing there. Probably I should lower the syslog severity to something around "debug", and that's what I'm doing right now.

Sorry for a rather long message that might not be very well structured.

 

[xcore]

For the record, I just lost camera feed/access again (it was streaming when I was writing the post above).

If there is a way to troubleshoot it without removing from the outside wall, it would be very nice.

Last syslog message:

<182>1 2025-01-15T13:07:14-06:00 172.30.4.52 "SetCurrentTime"
{
- - - "Address" : "172.30.4.1",
"Before" : "2025-01-15 13:07:13",
"Type" : "NTP",
"User" : "System"
}

Also PoE draw looks normal right now (as it used to be when camera was accessible/streaming):

xcore@SW-01> show poe interface ge-0/0/32
PoE interface status:
PoE interface : ge-0/0/32
Administrative status : Enabled
Operational status : ON
Power limit on the interface : 30.0W
Priority : Low
Power consumed : 5.1W
Class of power device : 4
PoE Mode : 802.3at

{master:0}

I edited original message above to correct myself (PoE power draw is not changing between "operational" and "non-operational" conditions).

 

[alastairstevenson]

1. Interface status on the switch was showing the same downtime as Zabbix was, about 4 hours
2. Interface was enabled, but not UP
3. PoE draw was present on the interface, usual ~5W
4. Resetting the PoE on the port helped - camera booted up and become responsible again with feed present in both VLC and zoneminder

I'd guess it might be a cabling issue.
Did you terminate the cable yourself?
What termination standard did you use?
What test method did you use?
Did you test the camera before mounting it?

edit What does the switch port log show for traffic stats & errors?

 

[xcore]

I'd guess it might be a cabling issue.
Did you terminate the cable yourself?
What termination standard did you use?
What test method did you use?
Did you test the camera before mounting it?

Hi,

Yes, I terminated the cable myself. It's RJ-45 plug on the camera side and a type-110 keystone on the patch panel near the switch. Patch-cords from patch-panel to the switch are factory made. It's TIA-568B on the both sides since modern switches always do auto MDI/MDIX.

I use the simple off-the-store cable tester to make sure 1) there's a connection per pair and 2) pairs are correct. I don't own complex expensive machines since I don't do cabling for a living.
That's what I have, including the crimper:

https://www.amazon.com/Klein-Tools-VDV501-851-VDV226-110-Ratcheting/dp/B09C279SYP

I'm a network engineer myself, so I hope my cabling is not extremely terrible.

I set up the camera on my desk before installing it, and it was working fine for a month or so (as I can tell from my monitoring).

 

[alastairstevenson]

I'm a network engineer myself, so I hope my cabling is not extremely terrible.

OK, that probably lets out one potential cause of the problem!
Long shot - swap the patch cord, re-punch the keystone ... but you've probably already done that.
Do the port stats show non-zero error counts?

 

[wittaj]

As you know, the cheap testers only test continuity not the ability to pass POE.

It could be a kink/bend in the cable, maybe a loose terminal connection that weather fluctuation hot/cold shrink/swell has impacted it.

What did you do in terms of waterproofing? If you did not use dielectric grease on the connection, it is really easy for it to get just enough build-up of corrosion not even visible to wreak havoc. Humidity in the air/time on install and hold/cold will create just enough. The waterproof connectors supplied with the camera is not enough. We have seen that happen to many people here and even though they didn't see any corrosion, when they used the trusted dioxit on the connections, guess what the camera started working again.

Do you experience this loss in the camera GUI or just ZoneMinder. We have seen with other non-Dahua VMS systems that this camera can struggle with other VMS systems with some of the different codecs and AI smart stuff. So it ends up looking like a signal loss but it is really some issue between the two talking. Try general codec, H264, and nothing else on - no acupick, no AI SSA and see if that fixes it.

 

[xcore]

OK, that probably lets out one potential cause of the problem!
Long shot - swap the patch cord, re-punch the keystone ... but you've probably already done that.
Do the port stats show non-zero error counts?

I didn't perform re-patching yet.

That's the stats from the camera port (I'm pasting only part of it for brevity):

Last flapped : 2025-01-15 13:50:00 CST (00:25:19 ago)
Statistics last cleared: Never
Traffic statistics:
Input bytes : 3197561416051 7592800 bps
Output bytes : 118693523697 341560 bps
Input packets: 2209414806 650 pps
Output packets: 1694124904 609 pps
IPv6 transit statistics:
Input bytes : 0
Output bytes : 0
Input packets: 0
Output packets: 0
Input errors:
Errors: 2621, Drops: 0, Framing errors: 2621, Runts: 0, Policed discards: 0, L3 incompletes: 0,
L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
Output errors:
Carrier transitions: 4181, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0,
HS link CRC errors: 0, MTU errors: 0, Resource errors: 0


These 2621 non-growing errors most likely related to 4181 carrier transitions.

Looks weird, cleared counters for now to monitor them next day or two.

 

[xcore]

As you know, the cheap testers only test continuity not the ability to pass POE.

It could be a kink/bend in the cable, maybe a loose terminal connection that weather fluctuation hot/cold shrink/swell has impacted it.

What did you do in terms of waterproofing? If you did not use dielectric grease on the connection, it is really easy for it to get just enough build-up of corrosion not even visible to wreak havoc. Humidity in the air/time on install and hold/cold will create just enough. The waterproof connectors supplied with the camera is not enough. We have seen that happen to many people here and even though they didn't see any corrosion, when they used the trusted dioxit on the connections, guess what the camera started working again.

Do you experience this loss in the camera GUI or just ZoneMinder. We have seen with other non-Dahua VMS systems that this camera can struggle with other VMS systems with some of the different codecs and AI smart stuff. Try general codec, H264, and nothing else on - no acupick, no AI SSA and see if that fixes it.

Hi wittaj,

Sure, I only trust my tester to show continuity. Cable was intalled by me mostly going over the attic space, so no kinks, almost no bends, and the bends are reasonable (well beyound 8-10 cable diameters).

I do not remember putting dielectrical grease into the connector, so it's likely as you're saying. Sad to hear supplied connectors are not enough.
While I can just re-crimp the RJ45 male plug, what should I do with female socket? I mean, I can gently rub it with fine file, but it might be not what you guys would do. I don't have any special chemistly to clean the oxidation.

As for the dielectric grease, I have a jar of clean non-abrasive silicon, so it's pretty straight-forward unless it's not what I should use.

About the camera access, when it's gone, it's gone. I use zoneminder as a smoke test and always check the stream with VLC player. I only use H264 codec and do not (and have no plans to) use any features of the camera behind the main stream+1-2 substreams with H264 encoding.

For waterproofing extra camera connectors I use the folllwing product:

https://www.amazon.com/gp/product/B08N4WBXL1

It fits on connectors really well (snug), but I frankly don't know if it will help in the long run.

 

[wittaj]

Many here swear by deoxit. I guess you could try the file first and see if it comes back.

But you would be surprised how little corrosion can wreak havoc on these things.

https://www.amazon.com/DeoxITLiquid-squeeze-tube-100-solution/dp/B0015A5AAY/ref=sr_1_8?crid=29U2D4Q1OKM92&dib=eyJ2IjoiMSJ9.WWYV76T4o0F4opwdowjH0-0SXEWKBucENaJWil9E6wnysRo0CuMiqtlKtSF9jZfDBVurab9EdzYbylCOl7Q-0BQQa0iSuLpcwVsIF6d7CEX72650kXoJoc7EFnBPZgNLNEe5M4lwYPhYYctBw7vdRh7XGY9tVgNVxag5vsrus8COOgXTQw_LIXlJ3L5pA_8-gXbEifqiuvdo6eg7cEIU0EfatUAJ_JFG3BgUntytY5YcFErEUf7LcMdh-tOKutDwgG-JE4OBZbNw2Q8toeRce_O1qHVqcJ1l2ZIIbZmg6mrRknhzcglgilvqu-r38HTLo0pHXN6StG8YZOls3l_dwYHzfwE1ejeU_dkkTl8LQDrG9mTNjFCd0bFXo09JTzBa17xFt6L3pHP-VPBK75BkwEP_OvTePkeie8y-D3S2H0QZlKcnMdcoDQrSG2UFsBER.hQEcw7ZHtQEbZ4WPk6htFjdu_0mFXWkNES55KhRQ45A&dib_tag=se&keywords=deoxit&qid=1736974461&sprefix=deoxit%2Caps%2C131&sr=8-8

If your silicone is like this, I guess it would be fine. Most here fill the female side, put a dab on the male end and then when the excess squeezes out, rub around the connector to kinda make a seal around the connector.

https://www.amazon.com/CRC-INDUSTRIES-ELECTRIC-GREASE-3-3-05113/dp/B01DOEJZTK/ref=sr_1_6?crid=2QPWDFBXCPR7M&dib=eyJ2IjoiMSJ9.THzS2hBpvC-nOwcJMHmtq3bT42t3Q_45q9CTKMrxAXZaxIYK-1lKOtRU7ki-brUI4C5NLBh223xqXbDJdKyprrQW4MFOYtT9aW7m8fEmNEETRCS4xtkWay5aXu9_n_hsVf214XuKFqlMCuPEa06DyIdsdJfMvp82xF_A2bme6x0U1aJWvVf8Ya1OcxwfH1FWcs8FfZJGzmFNszHJWYbrLPHjNYOVUmUEz1YojK4VmVQ.Xol9ZYQJRu7Uo5HyLlEptboPascKqEoHLgFOf8ooJug&dib_tag=se&keywords=dielectric+grease&qid=1736974610&sprefix=dielectric+grease%2Caps%2C103&sr=8-6

 

[tigerwillow1]

Another thing for the list of suspects is the possibility of the switch cutting off power to the port. I've had this happen when the load on the port exceeded the port limit. It could also happen if the total switch limit is exceeded.

 

[xcore]

Another thing for the list of suspects is the possibility of the switch cutting off power to the port. I've had this happen when the load on the port exceeded the port limit. It could also happen if the total switch limit is exceeded.

Hi tigerwillow1,

This was on the menu today. I have 1 camera, 3 wifi access points and another tiny device on PoE feed, and switch is giving about 28W out of total 750W available, so it should be fine for now. I was initially worried not knowing how much these cameras draw, and was expecting full 30W consumption (don't laugh too hard, it's my first interaction with IPCAMs), so I was pleasantly surprised when saw "5.1W" in the port usage output.

 

[xcore]

Thank you very much y'all, I'm going to get deoxit or something similar, open up the camera fixture and re-seal the connection (re-crimping the cable-side).
I will report here as soon as it's done, it might happen early next week.

 

[alastairstevenson]

These 2621 non-growing errors most likely related to 4181 carrier transitions.

Looks weird, cleared counters for now to monitor them next day or two.

Good idea.
Though not a huge number of errors, the expectation would be zero.
It will be interesting to see if there is a low-level buildup.

 

[alastairstevenson]

For waterproofing extra camera connectors I use the folllwing product:
https://www.amazon.com/gp/product/B08N4WBXL1
It fits on connectors really well (snug), but I frankly don't know if it will help in the long run.

A dumb question, for which apologies :
Would that bear down on the 'tang' that's depressed on the male RJ45 that might leave it unclipped and the connector maybe loose?
I know when I tightly wrap an RJ45 male to female joint with self-amalgamating tape I put something under the tang so it doesn't get squeezed down and no longer clicked in.
If that make sense.

 

[xcore]

A dumb question, for which apologies :
Would that bear down on the 'tang' that's depressed on the male RJ45 that might leave it unclipped and the connector maybe loose?
I know when I tightly wrap an RJ45 male to female joint with self-amalgamating tape I put something under the tang so it doesn't get squeezed down and no longer clicked in.
If that make sense.

I only use these protectors on free-hanging connectors I don't use (A/V, power and alarm dry-contacts). RJ45 goes unobstructed into the camera end and being protected by that thin cap with rubberized bushing and O-ring.
I usually try to hear/feel that click every time.

It wasn't a dumb question for sure given I provided no details on how I use them.

 

[rdxny]

I don't know if this is relevant, but just in case there's some carryover: I have the same camera and had video drops, often, though not long ones like you see. Cam was perfect with short cable to my switch, but the 150 foot run had the errors. The cable tested fine like yours. At someone's suggestion, I changed a Blue Iris setting (checked the "decoder compatibility" option) which stopped the drops for good. I realize you are not using BI, but maybe something can be learned from my experience that points to some help for you.

 

[Flintstone61]

^+1!
I have a color4k T180 and some people on the forum had a setting change that helped reduce / eliminate drops out.
Now if I could only remember what the hell it was.
Scour the forum you'll find it. ( oh its in the post above lol- that cleared it up for me.
Try running the camera cable as a Home run direct to switch port and see how it behaves for 12- hours or so.
Mine acted funny because i was feeding it 802.3 AF power over a 150 ft( 50 meter) run. I added a 5 port 802.3AT switch and then it behaved.
I pulled all home runs into my rack to avoid any extra points of failure.
the camera 180 in the log is my color4k.
my ad110 doorbell is on motion only with wifi with 18 no signals
I haven't reset the logs for about 10 days so that's acceptable for me.
its a hot running camera. so try to run a lower frame rate as to not overtax it in the heat.

 

[Gimmons]

I’ve had a couple of cameras go bad that started good but began failing intermittently. The culprit both times was corroded rj45s.

 

[bigredfish]

I’ve had a couple of cameras go bad that started good but began failing intermittently. The culprit both times was corroded rj45s.

There you go again with the common sense...geesh

 

[FlipNJ]

Sorry to jump in here but, about the RJ45 burnout. I have had 3 poe plus cameras (that have been up a couple years) have chernobyl incidents only on the camera ends. So, my question is, dielectric grease or not? I've read about it causing issues. All 3 connections were the weather-sealed ones and of course installed outside. All cameras are Dahua but don't think that matters. I believe it is moisture intrusion but figured I'd see what you think.