Delay on one camera

[TLR]

I eluded in another post of some delay issues that I am having and did not want to add this to that thread. But as I was working on the problem listed on that post, I have more data on my second issue. I have a 12 camera setup as follows.


A single camera will display and record with a 90s delay. The camera in question is the courtyard camera. All other cameras are in synch but that one delays recording and display of approx 90 seconds. If I log into the camera, it is fine. It displays live. But when viewing in BI5, I have a 90 second delay against all the other cameras. When I go back to view all cameras, it records delayed so it is like someplace between the camera and recording there is a buffer. Like in my other post, I from BI just open camera settings and exit out to force BI to drop and add and I am back in synch. I do not see how this could be a cpu issue as I would expect everything to be slow. CPU is ususally around 40-50%. running on and i7-8700 with 16GB of ram so should have plenty of headroom. When I log into camera, the camera is showing live video ( meaning no delay).

Any ideas?

on version 5.2.8.2 but this has happened since original install on earlier versions.

As I mentioned in other post, issues seem to coincide with move to new machine and BI5.

 

[fenderman]

I eluded in another post of some delay issues that I am having and did not want to add this to that thread. But as I was working on the problem listed on that post, I have more data on my second issue. I have a 12 camera setup as follows.
View attachment 62351

A single camera will display and record with a 90s delay. The camera in question is the courtyard camera. All other cameras are in synch but that one delays recording and display of approx 90 seconds. If I log into the camera, it is fine. It displays live. But when viewing in BI5, I have a 90 second delay against all the other cameras. When I go back to view all cameras, it records delayed so it is like someplace between the camera and recording there is a buffer. Like in my other post, I from BI just open camera settings and exit out to force BI to drop and add and I am back in synch. I do not see how this could be a cpu issue as I would expect everything to be slow. CPU is ususally around 40-50%. running on and i7-8700 with 16GB of ram so should have plenty of headroom. When I log into camera, the camera is showing live video ( meaning no delay).

Any ideas?

on version 5.2.8.2 but this has happened since original install on earlier versions.

As I mentioned in other post, issues seem to coincide with move to new machine and BI5.

how is your network setup? is the blue iris pc connected to the same switch as the camera or a switch cascading from the camera switch or is it connected to the router?

 

[pozzello]

sure you're not just seeing out-of-sync timestamps from that camera? 90 sec is a really long time!...

edit: NM. you said "If I log into the camera, it is fine. It displays live ", so that can't be it...

 

[TLR]

how is your network setup? is the blue iris pc connected to the same switch as the camera or a switch cascading from the camera switch or is it connected to the router?

cameras are on PoE switch. gig link to core switch. Server on core.

 

[pozzello]

what brand/model of camera is the slow one? maybe there's something about its stream BI doesn't like...

soem things to try (if not already) ?

• disable 'use RTSP/Stream timecode' in BI's camera/video/configure pane,
• reduce FPS. 25 is overkill, try 15 or even 10 FPS in the cam's UI, adjust keyframe interval as needed.
• reduce max bitrate.
• try VBR vs CBR (or vice-versa)...

 

[pozzello]

have you tried putting the BI server on the same switch as the cams? one less hop for cam traffic...

 

[TLR]

sure you're not just seeing out-of-sync timestamps from that camera? 90 sec is a really long time!...

edit: NM. you said "If I log into the camera, it is fine. It displays live ", so that can't be it...

Its not a time stamp issue. I can track a person around the house. Did an experiment and had my wife walk around. So lets say there are 3 cams in line with some overlap on the linear path. Cam 1 picks up somebody walking, as the person progresses you would expect to leave the view of cam 1 and cam 2 picks up. Then cam 2 to cam 3. In my case, you see in cam 1. Nothing in cam 2 then you see cam 3. Wait 60-90 seconds, you get the frames from cam 2. It is wild. Network traffic is a fairly steady 50mbps at the server. Everything is 1gig connections . This particular camera is a 8MP camera (IPC-HDW5831R-ZE) but I have 2 others that are just fine. Same firmware versions and setup. Agree that I need to drop the FPS on the 3 cams. Just an oversight on my part.

I can move the server but I would think if there was a network issue, I would be having other issues. Why this cam. Now I have seen this before. My wife noticed it and I just reset things thinking it was something one-off. I think it was on a different camera but I cannot be sure. This is a random issue and have not figured out if it is a drift ( meaning each day there is a few seconds of additional delay) or if it just all of a sudden delays. I just am looking for ideas as I cannot come up with anything that would cause this. This system has been up for over a year and never had this.

 

[pozzello]

ok, so it's not some off-brand crap cam. how very strange... have you tried re-crimping the ends of the 'slow' cam's ethernet? one end at a time...

 

[TLR]

ok, so it's not some off-brand crap cam. how very strange... have you tried re-crimping the ends of the 'slow' cam's ethernet? one end at a time...

I could it just does not track to the issue. A bad crimp would caused packet drops. Data is destroyed in these cases. This is being buffered somewhere. does anyone know if BI creates a process per camera to record? Thinking way out of the box here

 

[pozzello]

do you have the newer version of BI with substream support? if so, have you tried accessing the substream on that cam and is it OK?

I'm thinking when you view live via the cam's web gui, it's serving a different stream than BI is pulling.

15FPS should not be too much for the main stream of an 8MP cam, but i see your other 8MP cams are at 10 FPS, so try that if not already.

Maybe the cam has issues keeping a 15FPS 8MP stream under 750kbps for some reason, and the streams you see working live (not delayed) are actually the substream?

Maybe you did already, but try lowering the FPS, and separately (change one thing at a time!) try RAISING the bitrate on the stream BI is pulling.

As Sherlock Holmes said: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth "...

 

[TLR]

I lowered the FPS last night but as state, so I will wait and see. Unfortunately this does not happen immediately so each change could be days in between. I will keep an eye out on a daily basis for variance building. As of this morning, everything normal. Thanks for the comments and feedback. Will come back on status either way as maybe it will help someone else. But I have to believe something is up beyond FPS. If I take this and add it to my other problem post I did yesterday where a random camera will drop and display what looks like a 4 color image, it has to be related somehow. The game is afoot - sticking to the Sherlock Holmes theme.

 

[Tro]

A few things I would try - I have a similar set up (9 cameras - i7 9700k)
Save the cam config and set up as a new cam - with minimal change - set fps same as other 2 X 8PM cams
Swap location with another camera
I did my own wiring and did find 1 cable that had 1 pair down recently - although tested fine at the time
check the cable from the switches/back-haul
although machine shows little RAM usage - i had better performance going from 16gig to 32gig - most likely due to additional RAM slots used?

Sometimes the connections in sockets (wall and switch) might need re-seating - in humid climates

Tro

 

[TLR]

Lowered camera to 15FPS. Delay happened again. Have a little more detail on how the symptom develops. What I am about to describe is how it is recorded. The first time is the playback timeline value, the second is the camera value.

at 13:35:47 the camera timer value freezes.
at 13:36:15 the camera counter resumes at 13:36:14 so 1 sec behind. Usually the camera value is 1 second ahead of the timeline value
at 13:37:01 the camera value is 13:36:53 - so lagging 8 sec
at 13:37:42 the camera value is 13:37:13 - so lagging 29 sec
at 13:38:35 the camera value is 13:37.31 - so lagging 64 sec
at 13:40:03 the camera value is 13:38:27 - so lagging 96 sec
at 13:42:01 the camera value is 13:40:01 - so lagging 90 sec
It remains at a constant 90 sec lag until I reset the camera.

So, at this time I was doing a Ubiquiti firmware update. So that explains the freezing. All the other cameras froze at the same time for the same duration. All cameras recovered at the same time ( +/- a sec) but only this one 8MB camera experienced this time delay/slowdown. It is visible in the on screen counter. Over 6 mins it looses 96 secs then stabilizes and runs 90 sec behind. What is buffering the data. It seems a network issue could trigger whatever this issue is but I am stumped at what could hold 90 secs of video. It is like the camera is buffering it.

Does this trigger any ideas with anyone?

 

[Tro]

Strange,
When i recently set up my doorbell I had issues that were similar -delay (freezes) but catches up
Had to play with the settings a little on the main add camera config page

Assuming set up the same as other 2 X 8mp cameras... check to see if all the tick boxes and (main) feed string entries are the same
As above, try the config on another camera - or another camera config in the troubled camera

That might eliminate any possible settings or network issue
Personally I would set it up at another location (swap it out) and try it - if it still plays up - try the other cameras config (that way if its OK you can leave it there)

Have you tried to log in directly and stream for hours to see if the same happens?

Tro

 

[samplenhold]

There is a receive buffer in BI. In the camera settings, video tab, under Configure... Network Options, Receive buffer (MB).

Compare the settings on that widget for the offending camera to the others.
I do not know if this could be the problem, but worth a try.

Another thing to try would be to delete the camera and the add it again.

 

[TLR]

There is a receive buffer in BI. In the camera settings, video tab, under Configure... Network Options, Receive buffer (MB).

Compare the settings on that widget for the offending camera to the others.
I do not know if this could be the problem, but worth a try.

Another thing to try would be to delete the camera and the add it again.

Well the buffer on the offending camera was half the other two. The offending was at 6mb and other two were at 15mb. So I raised it to match the others. If this does not work, I will drop the camera and add it back in just a next step. Lets see.......

 

[ExtremeSkiBum]

Are you still having these problems? I'm having a similar problem as you described ever since I upgraded my hard drive. So last weekend I removed my boot up HDD and replaced it with a SSD, lost all my configuration so I had to setup everything from scratch. Ever since then I'm getting a delays on a few cameras, I never had this problem on the old drive.

 

[SouthernYankee]

BY Pass the Ubiquiti switch. Use a simple dumb POE switch to the camera.
Did you increase the buffer size for the camera in BI, I do not thing a buffer in BI problem would cause a 90 second slip.

Another test, do a long running, large packet PING test to the camera from the BI machine, Look at the time and for any drop packets. There are better programs than the standard windows ping
PING -n 20000 -l 32000 -w 200 192.168.1.210

Are you using Static IP address ?
Make sure that the camera traffic is not running through the router>
Verify that the virus protection software running on the BI machine is not causing a problem.
Do not know about Ubiquiti switch, turn off virus protection and fire wall on the switch

 

[TLR]

Are you still having these problems? I'm having a similar problem as you described ever since I upgraded my hard drive. So last weekend I removed my boot up HDD and replaced it with a SSD, lost all my configuration so I had to setup everything from scratch. Ever since then I'm getting a delays on a few cameras, I never had this problem on the old drive.

very sporadically. it has actually behaved itself for the last few weeks. I just cannot find a pattern.

 

[TLR]

BY Pass the Ubiquiti switch. Use a simple dumb POE switch to the camera.
Did you increase the buffer size for the camera in BI, I do not thing a buffer in BI problem would cause a 90 second slip.

Another test, do a long running, large packet PING test to the camera from the BI machine, Look at the time and for any drop packets. There are better programs than the standard windows ping
PING -n 20000 -l 32000 -w 200 192.168.1.210

Are you using Static IP address ?
Make sure that the camera traffic is not running through the router>
Verify that the virus protection software running on the BI machine is not causing a problem.
Do not know about Ubiquiti switch, turn off virus protection and fire wall on the switch

The cameras are all on a ipcampower switch. THis feeds a core ubiquiti switch that goes to a IDF switch where the BI server is. it never passes through the FW and is all hardwired. I ran a ping test for a couple hours using the ping tool from colasoft and had no packet drops. In fairness I do not remember if I used large or small but the default is 64bytes with a 1 second delay in pings. I can re-run just for grins but as I stated earlier it has been behaving lately. I did have a situation that 2 cameras at one time delayed for different delay times. Both were the 8M cameras so I am confident I am having some resource constraint but have not caught it yet. The cameras do not drop frames just seem to slow down. It records that way. When it happens you can see the timestamp slow down over the course of 5-7 mins then just run normal but approx 90secs behind. Reset the camera in BI and back in synch. Network monitoring shows i never come close to saturating the Gig uplink from the camera switch to the central nor the central to IDF. So I do not believe I am network constrained.

All addresses are static in the cameras and BI Server. Thought about the Virus scanner on the BI box but how would it target only a specific camera.

As stated in my previous post, last few weeks it is all calm so tough to continue to troubleshoot.