1 x 4mp hikvision camera - stuttering and freezing. HyperV issue [Solved]

[Rex]

I have one hikvision 4mp camera DS-2CD2542FWD-IWS monitoring the front of house. While the inbuilt motion detection is reasonable, I'm getting a lot of false motion events with tree shadows. I've purchased BI because it is much more flexible with detections. BI version is 4.5 x64

With the iVMS app or web view, I have no issues with live view or replaying recorded footage. With BI I’m having issues with video stuttering, blurring and at worst, freezing up.

Machine is a HP ML10v2 server upgraded with a E3 1230v3 CPU, 16GB ram. It’s running windows server 2016 as a host/hypervisor. BI is running in a server 2016 VM with 8 vCPUs and 6GB ram.

I know running it inside a VM is not as great as a dedicated machine and there is no hardware acceleration either. However, the server was fairly cheap and I needed something to replace my aging microserver and thought it would be good to run an all in one machine that has VMs for fileserver, linux, home automation etc. In hindsight, should have spent the Xeon money and got a dedicated BI box instead. An refurbished optiplex 9020 is around the same price as the Xeon. Since I’m stuck with this setup for the moment, I’m trying to make the best of it. Strangely the CPU usage is only ~10% (with no activity). Monitored both within the VM and through the HyperV performance monitor counter on the host machine.

The camera is running on wifi to the router only a couple of meters away. Ideally, it should be on POE but it will involve drilling through floorboards and walls. I’ll give it a go when I have more free time. Note, another hikvision camera also utilises the 2.4Ghz wifi but it is monitoring a storage room with hardly any activity and is set to record only on motion. It is not monitored through BI right now.

I had to duplicate the front camera for a total of 3 cameras because I need:

• A catchall setup that does my overall motion detection with average sensitivity, together with audio trigger
• a high sensitively level for one area
• line crossing detection for one area

2 cameras are hidden. I can confirm that BI is only pulling 1 stream via the status page. BI is run as admin and minimised normally.

The camera is setup up as below in BI:

• Trigger is set to on motion and record only on motion
  
• direct-to-disc recording
  
• The new clips are saved to a SSD (shared with the VM OS)

The camera configure in hikvision setup page:


When there are no events, the live view is not too horrible, a little lag and stuttering. This is evident when watching the timestamp ticking over and counting the seconds. It's never as smooth as iVMS. When motion is detected, (sometime more than 1 camera is triggered), the whole thing becomes very temperamental. In the worst case, I get a 'signal lost' message and it will take a minute for the video feed to be restored.

Note I currently have it set on H264+. This has worked well to conserve wifi bandwidth. The camera does ~ 130kB/s in BI status page. With the stock ivms app or the web app, I also had no real issues with H264 off. This increases the bitrate to ~500 kB/s (measured via my router). Thinking it could be a H264+ issue, I’ve switched it off but BI struggled, with live view with 2 minutes of lag. It feels like the machine is struggling to catchup. Strangely the CPU usage is only ~20% still.

Things I haven’t tried but might help:

• Turn off motion detection and recording on the camera. Currently its on and record on motion since I don’t want to miss anything due to the BI issue. Could it be because camera is using CPU on motion detecting and recording that it's struggling with the video feed to BI?
• Try BI inside a Win10 VM as opposed to a Server 2016
• Try install on the host. Really don’t want to do this as it defeats the whole point of having a hypervisor.

I've setup performance monitoring over time for the moment. There could be the chance that on motion the CPU struggles to catchup.

I’ll continue to do more tinkering but I’m wondering if more experienced BI users can point me in the right direction?

Thanks

 

[fenderman]

@Rex, disable h.264+ and set the iframe interval to match the FPS...
If you are not using your cameras motion detection, uncheck get events with pull point
If that doesnt help (it should)..try selecting hikvision rtsp or 2cdXXXX rtsp from the dropdown...

 

[Rex]

just spent the last 4 hours on trying to find out the root cause. observations so far:

I turned off pullpoint straight after so all observations made with this.

1. I noticed that having a live preview made a huge impact, even if I limited it to 1fps. I can verify this 100% of the time by opening up the status screen and monitoring the fps. with BI minimised I get ~20fps, as soon as I open up the BI window, fps drops to as low as 10fps. I'm not really sure why since CPU never goes above 50% in the VM and in the performance monitor in the host machine. I thought it could be due to the fact that the camera is struggling to send but point 3 below doesn't align with this. This is an issue since this means I can't actually use BI to review my footages without impacting recording.

2. I tried playing back with the web ActiveX plugin and videos were much smoother. There were still tiny micro freezes but generally ok. When viewing the clips, CPU in the VM goes up by ~20%. I then copied the BVR file onto my main PC, opened it in Media Player Class (MPC) and it was butter smooth. This is with the H264+ streams. However, there were a couple of clips where the freezes were real and I missed >10sec of footage and the timestamps struggled to stay constant after.

3. I thought the issue could be due to the camera getting over worked. ie sending an rtsp stream (for BI), processing motion detection, saving recording and sending email all at the same time. I turned BI off, opened up RTSP stream in VLC. opened up the browser and logined for a live view. I then walked around to trigger the detection. This didn't really do that much and both feeds were really smooth.

4. not sure if turning off h264+ made too much of a difference since the footages in the morning were in H264+ and most were smooth apart from the few that froze badly (point 2). The footages with H264 off and iframe =20 were about the same. had a few that were smooth and then a couple that froze badly.

5. also tried changing the Make to hikvision and 2cdxxxx rtsp. don't think it did that much either.

6. this is perhaps a most strange one. I'm keeping the status window open at all times and Bi minimised. I noticed that randomly the fps would drop gradually to almost 0 and then pickup again. at the same wifi on my router would show a drop in kbps. first image had me opening up some replays in iVMS as well so had a few spikes but the red circles is when it all dropped. 2nd image was everything running idle and fine and then a drop randomly. I have a iVMS 4200 window open (on my main PC) with a substream and this was not affected at all. I think these are causing the stuttering and major freezes in some of my clips.


7. i also noticed that in the VM, if I did anything like drag a window frame around and open some apps, this appeared to affect the fps at times

8. just when i thought i had it figured out i'm throwing out some of my theories out the door again. its late here now and camera is in B&W mode. kb/s is ~370vs 560 during the day, h264 off. I just tried dragging windows around like in point 7 and that made no difference. also opened BI, turn on live view for about 10sec, fps dropped some what and then recovered. in my router, I dont see a big drop at all in activity. it's just a constant smooth stream at ~ 380kbs! in BI status window though, fps is struggling to go back up. I had to minimise the BI window and fps then slowly recovered. Not sure how accurate the fps is since according to the router, it is sending the rtsp stream with no issues.

All in all, i'm thinking there might be some wifi interference during the day from neighbours/other devices, which explains why at 10.30pm it is smoother. it doesnt explain why iVMS never has an issue though and rtsp stream seems to be ok in VLC. Also the VM might be struggling even though cpu utilisation is low for some reason. it could be a hyperV issue, perhaps with how video is implemented, which explains why with live view on, it struggles even though CPU utilisation is low.

I guess one good way to rule out wifi is to deactivate licence on the VM and try BI for a few days on my main PC which should have quicksync as well. I didn't want to try incase activation buggers up. will there be issues if all the PC have the same WAN ip?

I might just try BI on the host OS instead of the guest VM, at least that will rule out a VM issue.

another test I can try is to unmount the camera for the storage room. its a 3mp hikvision on an older firmware. I can up the bitrate and quality so it reaches the 570kb/s of the problem camera. I can then test this on wifi and then Ethernet cable.

 

[Rex]

I solved it! hopefully others can learn from this!

This morning I was monitoring the wifi and it appears in the day, more people had wifi on (router scanned >20APs!) and although I changed the channel to an uncongested channel it still didn't like it. Changed to a new channel and 4mp live view and rtsp stream became pretty steady. Looks like i'll need to do the wired lan cable install asap.

When I got home, I installed BI on my main PC. haswell i5. ran without a sweat. I then installed BI on the server host instead of the VM and it was still horrible. Since the Xeon is almost an equivalent of my i5, I was scratching my head. I almost thought BI didn't like server 2016!

Thinking it through, i checked powersettings and realised I set the server to balanced instead of performance. Balanced allowed the CPU to drop to 5% min. I changed it to performance and instantly everything became buttery smooth! Reactivated the BI in the VM and it was the same. Super smooth. I can record with live preview on and also playback my videos!

Note my main PC is on balanced and I never had any issue with setting it on balanced since windows is smart enough to assess the load. However I'm guessing it's either the ml10v2 machine's integrated power saving features and/or the combination with server 2016 (there are some issues with 2016 such as vt-d not supported where as server 2012r2 is fine) that caused this massive headache.