Cameras keep dropping offline and coming back seconds later

[Ispy]

I just replaced my old hardware with a new system. Basically a rip/replace with same IP, network config etc. Everything worked great except that my cameras keep randomly dropping out all at once for about 5 seconds and then reconnecting. It is a dedicated hardwired network (access via wireless adapter on server) so nothing else is on this network besides cameras. I ran continuous pings to two of the cameras that are dropping (all 5 drop at the same time) and pings never drop which tells me the network is at least not dropping. The switch is a gigabit switch and interface is gig as well. stats on BI seem fine except disk. CPU 9%, mem 21%, disk 98% constant. I had all cameras on continuous recording and dropped them to triggered to see if that did anything but it made no difference. Is there any kind of logging that could help me here before I dig out wireshark? I am at a bit of a loss. Everything has been rebooted to no avail.

New hardware is as follows

Kingdel NC860 Mini Desktop Computer, Fanless Mini PC with Intel i7-10510U 10th Gen. CPU, 16GB RAM, 512GB SSD, 4K 4096x2304, HD Port, DP, Full Metal Case, Windows 10 Pro

-scott

 

[th182]

To clarify... You are running hardwired for your cameras and into the BI box. And wireless is to connect the BI box to the home network (separate from cameras)?


Sent from my iPhone using Tapatalk

 

[sebastiantombs]

Have a look at the CPU temperature. Fanless is not the best choice for constant video even at low CPU utilization.

 

[Ispy]

correct, hardwired for cameras only. as for cpu, i will check it but i do NOT use local console. i ONLY use web interface from remote pc unless i am doing camera changes.

 

[looney2ns]

correct, hardwired for cameras only. as for cpu, i will check it but i do NOT use local console. i ONLY use web interface from remote pc unless i am doing camera changes.

Doesn't matter, the CPU is under considerable load all the time BI is running.

 

[Ispy]

Doesn't matter, the CPU is under considerable load all the time BI is running.

Ok. did you see my CPU load? 9% is not considerable in my opinion. and that was with an RDP session open too.

 

[sebastiantombs]

It doesn't cost money to check the CPU temperature. Fanless minis are notorious for running hot and that could easily be causing your problem.

 

[Ispy]

sure. package/cores running about 55c on average. compared to my system that is working which is 82c on average. don't think this is an issue. running one on wifi side works fine. something is flaky on that LAN side.

 

[th182]

Does the switch have any QOS features? Could it be throttling the cameras? Or is there an issue with the computers NIC? For some reason my BI PC kept dropping cameras when I used the built in NIC for them. Swapped so the cameras were on the add on card and all was well.


Sent from my iPhone using Tapatalk

 

[Edcfish]

Sounds like a challenging mystery.

The fact that your disk usage is excessive and that the cameras all drop simultaneously are the red flags i would start chasing.
Are the drops timed consistently? ie: every 40 minutes all cameras drop. Or are the drops more random in nature? ie: the cameras dropped 3 hours ago, 1 hour ago, and 20 minutes ago.

Have you tried viewing the cameras a different way (ie: web browser, ONVIF Device Manager, etc.) Do they all drop simultaneously when viewing this way?

Does your disk usage ever drop to a reasonable level?
In task manager, what processes does it show using the disk the most?
Do you have antivirus disabled for the BI machine? Also make sure no optimizations or defragging of the drive are enabled.

How are you recording? Not at all? On Triggers? Continuously?
What device/drive are you saving the recordings to? i would not recommend using a SSD for recording storage. A surveillance grade spinning drive would be best.

Are the camera's POE from the switch (not injectors)?
If so, i would suspect issues with the power provided by the switch (either cameras are over-drawing and causing the switch to 'reset' power distribution or the switch POE is starting to 'flake out'. I have had both occur and spent the better part of 3 days trying to figure them each out. For each of my problem sites, a new power source stabilized things; for one i installed a separate 12 port power injector between the switch and the cameras and for the other a installed a new POE+ switch.
I would start by taking all the cameras offline except for one, let it run for a while and see what happens, if nothing bad, then add back 1 camera at a time to the switch.)

Are there any traffic shaping/QOS options on the switch that could be interfering with camera traffic?
Does your switch have monitoring tools built in? If so I would monitor port traffic and POE and look for any anomalies.

Lastly, what type of NIC is in the BI PC? I have had problems with el-cheapo on-board NICs in the past (they would overhead, drop packets, etc...but not with the on-board Intel NICs) and needed to add a separate NIC card (I went dual NIC card and like the results). Admitedly, since you are able to continuously ping, i doubt if it is the NIC, but it still could be since pings take almost no bandwidth when compared to video.


EDIT: fixed typos

 

[Ispy]

OK, first off I would like to thank you all for the information you have given and asked for. I was working to prep a large response as I tried other steps for troubleshooting and in the process have come across what appears to be a fix however I don't know the true issue. As I did some testing I had a wifi camera that never dropped so I converted that camera to its hardwire connection and it worked fine. I added another on a small hub as well and had no issues. At that point I decided to use that hub as an interim connection point between this new hardware and my camera switch (dumb switch). Since I did that I have had no failures. This unit has no ability to add additional NIC (something I had thought of as well) but I do suspect there is some kind of congestion avoidance or flow control that are not jiving between this NIC and that switch. This NIC is a Realtek PCIe so not an uncommon chipset. I did try tweaking flow control on it to no avail. Drivers were as up to date as I could see. That switch is POE and I have seen before as edcfish noted can cause issues (deal with it in my profession occasionally). I really do need to address my disk usage at some point however it is not causing my any problems I can see right now. It was no different on my other system. I do run continuous recording (which I did turn off during testing but did nothing). So for now my fix is to run this small hub/switch inline to resolve my issue. I needed an extra port on this network where my server is for another camera anyways. Other switch is across the house in a closet.
Thanks everyone again!

 

[IAmATeaf]

What make/model is the POE switch?

 

[Ispy]

just an intellinet unmanaged poe gig switch.