BI recordings are skipping

[Corellon]

I'm running into an issue on my system where BI is recording properly but appears to "Lag" in the recordings until it falls behind and every 5 minutes "Skips" ahead to the current view and I'm trying to troubleshoot why.

System:
Xeon L5639 x2 CPU's
48GB of RAM
K2200 GPU
2x 1GB Ethernet bonded as a team

Camera's:
6x Lorex's converted to IPC-HFW4830E-S Dahua frimware

Streams:
6x 24/7 4k Recording Streams H264H VBR 10kbps Encoded, Direct to Disk enabled
6X Motion Triggered 4k Recording Streams H264H VBR 10kbps Encoded, Direct to Disk enabled

As far as I can tell the CPU and GPU are not being taxed
CPU was at 36% prior to installing the GPU, Now runs 18-20%
GPU runs 20-30%
Memory is running at 48%
Camera's are set to NVENC for the decoding

I can confirm the feed from the camera doesn't skip, it will display constantly and without issue so the issue lies within the BI setup. I'm just not sure if it's the encoding, motion detection or system that is the issue. Even the streams that don't have motion encoding setup still experience the issue, one thing that did lessen the impact but not eliminate it was increasing the memory buffer from 0.6MB to 20MB on each camera, now it appears to go longer without "catching up" but still jumps.

Thanks for any suggestions or insight.

 

[SouthernYankee]

Is the jumping occuring when viewing a video file in playback in the BI application program.
Are you running headless ?
Does it skip in the same place when viewing the camera in UI3 ?
What OS are you running ?
Are you running a VM ?

A side note duel core xeon processeors do not work very well on a single multi threaded program, that is not designed for two independent CPU. The memory management and memory delays cause problems in thread switching. Also that is a power pig, it is not very green

 

[Corellon]

The jumping occurs when watching live view or recorded clips no matter how many times watched both through BI and UI3 as well as when the clips are exported and I watch on the desktop.
Server isn't completely headless as there is a monitor connected to it however it is off most of the time and I view through UI3 or RDP (RDP setting is unrestricted)
Server is running windows server, and it's not running on a VM.

System it's running on is a hand-me-down right now until I can put something better together, while not very green core utilization is utilized across both processors (12 cores total).

It may be related but BI also can't seem to figure out what frame rate the camera's are showing at, they should be putting out 15fps with an I frame every 15 frames (1/sec) but BI detects fps fluctuating on the feed between .5fps and 17 fps.
Again I can confirm the feed is consistent but something about how BI detects the frame rate is getting different results.

 

[fenderman]

Again I can confirm the feed is consistent but something about how BI detects the frame rate is getting different results.

Test the rtsp feed in vlc.

 

[Corellon]

Test the rtsp feed in vlc.

Can confirm that RTSP is being received without error as well, When I Change the decoder on the camera it works perfectly for about 10 seconds then starts to stutter again.

I do have one camera (The first camera I've added) that is working perfectly, recording and receiving all frames and I have confirmed all the other camera's are identical in configuration but are not working.

There are a few things that I noticed that may be related but not quite sure:
When setting up VLC without hardware decoding some frames would be lost, but when enabling hardware decoding all frames rendered properly
When running the resource monitor the network activity is not as expected, only 192.168.xxx.15 (the working camera) is pulling a constant 2MB/sec, all the other cameras go through a cycle of 0MB/Sec then .5MB/sec, 1MB/sec then down to .5MB/sec and then drop off the list for lack of activity. The cycle of activity also seems to match the cycle of recording issues. I know it is not network related as on the same computer VLC can pull the full 2MB/Sec.
DXVA 2 decoding seems to be limited to 5 streams at one time, if I try to enable a 6th one of the others reverts to "no" however NVENC does stick when all camera's are set to it. I'm also seeing the GPU usage be more "Bursty" then I would expect but never exceeds 30% though will at times drop as low as 7%

 

[fenderman]

Can confirm that RTSP is being received without error as well, When I Change the decoder on the camera it works perfectly for about 10 seconds then starts to stutter again.

I do have one camera (The first camera I've added) that is working perfectly, recording and receiving all frames and I have confirmed all the other camera's are identical in configuration but are not working.

There are a few things that I noticed that may be related but not quite sure:
When setting up VLC without hardware decoding some frames would be lost, but when enabling hardware decoding all frames rendered properly
When running the resource monitor the network activity is not as expected, only 192.168.xxx.15 (the working camera) is pulling a constant 2MB/sec, all the other cameras go through a cycle of 0MB/Sec then .5MB/sec, 1MB/sec then down to .5MB/sec and then drop off the list for lack of activity. The cycle of activity also seems to match the cycle of recording issues. I know it is not network related as on the same computer VLC can pull the full 2MB/Sec.
DXVA 2 decoding seems to be limited to 5 streams at one time, if I try to enable a 6th one of the others reverts to "no" however NVENC does stick when all camera's are set to it. I'm also seeing the GPU usage be more "Bursty" then I would expect but never exceeds 30% though will at times drop as low as 7%

Does this problem occur when you disable all hardware acceleration? are you using a smart or + codec?

 

[Corellon]

Problem still occurs with acceleration disabled, it was one of the reasons I added the GPU in thinking it was a CPU load issue.

Codec is the H.264H codec, not with smart codec turned off.

Between the working camera and the non working ones I checked again and all settings are identical both in BI as well as on the camera's side.

I've been watching the resource manager and other tools for longer and closer and a few additional details I can add:
GPU Memory is at 50%, each stream is 350MB of usage (2.1GB total) of the 4GB available
GPU is not throttling based on temp, power or steam limitations
When the recordings "lag" all camera's except for 1 "lag" as well and "catchup" at the same time.
Since adjusting a few camera settings the "Normal" camera is now a different camera, it was the last one adjusted and now does not seem to "lag" but the old camera that was "normal" is now "lagging"
Average latency to write to disk is 13-18ms, load is peaked at 2.2MB/s however I would expect disk utilization to be constant as it is recording 6 streams 24/7 however it drops to zero, then writes then drops
When the Camera's "Lag" disk writes, network usage and GPU usage almost zero out (except for the "normal" camera) which would indicate to me it's not an overload on the decoding engine but a bottleneck in something feeding/pulling to the engine.

As far as I can tell at this time everything points to BI not being able to receive/pull the feeds consistently, from a networking standpoint I can view all the feeds through the browser or VLC without issue but BI appears to be bottlenecking somewhere.

 

[Corellon]

So I think I'm onto something here:

I increased the buffer to 40MB, no real change
Changed profile from Genertic RTSP/ONVIF to Dahua Mainstream, no change
Changed the main 24/7 recording to "Limit decoding unless necessary" and now the streams are pulling ~15fps

So there seems to be an issue with running parallel decodes on cloned streams even if there is the hardware capacity to do so, something in the software creates a bottleneck/loopback/deadlock.

It's not perfect still since when I view one of the 24/7 camera's live through BI or UI3 it unsyncs everything (As it appears that triggers decoding)

I'm going to see if I switch decoding off completely for those streams helps with the matter but at least I have a direction on where to go from here. The odd thing is that One camera had "Limit decoding" on both the 24/7 and motion based recording streams and it was affected as well and another camera had it unchecked on both and it was working fine so I suspect it's not specifically a stream issue but an internal BI issue where it deadlocks/loops itself (would explain why one camera would work, always the one last adjusted)

 

[fenderman]

So I think I'm onto something here:

I increased the buffer to 40MB, no real change
Changed profile from Genertic RTSP/ONVIF to Dahua Mainstream, no change
Changed the main 24/7 recording to "Limit decoding unless necessary" and now the streams are pulling ~15fps

So there seems to be an issue with running parallel decodes on cloned streams even if there is the hardware capacity to do so, something in the software creates a bottleneck/loopback/deadlock.

It's not perfect still since when I view one of the 24/7 camera's live through BI or UI3 it unsyncs everything (As it appears that triggers decoding)

I'm going to see if I switch decoding off completely for those streams helps with the matter but at least I have a direction on where to go from here. The odd thing is that One camera had "Limit decoding" on both the 24/7 and motion based recording streams and it was affected as well and another camera had it unchecked on both and it was working fine so I suspect it's not specifically a stream issue but an internal BI issue where it deadlocks/loops itself (would explain why one camera would work, always the one last adjusted)

When you properly clone a camera it will not pull an additional stream from the camera unless you select both as masters OR use different login credentials and/or paths to get the streams. You can see this in the status tab as the bitrate for the cloned cameras should be zero. If your camera is sending multiple streams it can be choking as they have limited cpu resources.

 

[Corellon]

When you properly clone a camera it will not pull an additional stream from the camera unless you select both as masters OR use different login credentials and/or paths to get the streams. You can see this in the status tab as the bitrate for the cloned cameras should be zero. If your camera is sending multiple streams it can be choking as they have limited cpu resources.

I can confirm it was properly cloned, the clones have/had 0 bitrate and the same FPS as the master. When I had "limit decoding unless required" unchecked on both the master and clone it would create the above issue, when I have it only unchecked on the clone (Since I believe it's required for motion detection) the issue goes away (at least until I view the live stream of the master).

Motion detection is still working with the setup and when viewing the recordings I only get 7-8fps though UI3 (Player delays are in excess of 17000ms) but exporting gives the full 15fps @ 4k.

 

[SouthernYankee]

I think you may be having a network problem with the cameras or a CPU problem.
1) What OS are you running ?
2) Unbind the network, you do not need a bind network. Use one Nic only.
3) if you can disable one of the CPUs, do it.

 

[Corellon]

Right now it's on a evaluation copy of Server 2019.

Issue existed before I bound the network nic's, binding them was one of the steps I did during troubleshooting. Since it's working when the master is set to limit decoding and the clone set to decode for motion I doubt it's network related.

I can't pull one of the CPU's but I have set BI's affinity on what cores and CPU's it can run on and it didn't seem to make a difference, when I look at the task manager it liked to only run on one of the CPU's anyways even when both are enabled.

The issue seems related to the simultaneous decoding of the master and clone streams at the same time.