HikVision DS-7608NI-SE/P - Abnormal Shutdown?!

[DrBo]

Hi guys,

I am still trying to figure out what's wrong with my NVR, as it frequently enters a coma.

Anyway, I tried extracting information from the NVR log, searching for Major Type "Operation", and Minor Type "Local: Abnormal Shutdown". This is what I found (NVR was installed 15 November, with firmware 2.3.7 - this was then upgraded per advice from HikVision to 2.3.9 using tftp 22 December - after event #3 below):

Time Major Type Minor Type Channel No. Local/Remote User Remote Host IP
1 2014-12-10 15:02:30 Operation Local: Abnormal Shutdown 0.0.0.0
2 2014-12-16 19:22:45 Operation Local: Abnormal Shutdown 0.0.0.0
3 2014-12-22 04:57:11 Operation Local: Abnormal Shutdown 0.0.0.0
4 2014-12-23 01:33:41 Operation Local: Abnormal Shutdown 0.0.0.0
5 2014-12-28 12:29:35 Operation Local: Abnormal Shutdown 0.0.0.0
6 2015-01-01 16:14:10 Operation Local: Abnormal Shutdown 0.0.0.0
7 2015-01-02 12:34:30 Operation Local: Abnormal Shutdown 0.0.0.0

The NVR does NOT recover from this situation; either re-installation of firmware OR a hard reset (toggling power) is required.

Anyone with similar experiences?! :wondering:

BR Bo

 

[DaveP]

Few thoughts...

Reduce the load by removing a few cams and see what happens.

Could be the HHD, try a replacement as a test.

May be an idea to telnet into it and keep an eye on the cpu ( top )

 

[DrBo]

Few thoughts...

Reduce the load by removing a few cams and see what happens.

Could be the HHD, try a replacement as a test.

May be an idea to telnet into it and keep an eye on the cpu ( top )

Thanks for the suggestions - my comments:

* Got 5 3 Mp cams (DS-2CD2132-I) connected - should be "within operating limits"??
* The HDD is a brand-new 1 Tb purple ("24/7") WDC drive - so should(!) not be an issue. I might try to switch it just to see what happens..
* Telnet - hmm... that will require some studying from my side (never done that before!)

BR Bo

 

[DaveP]

* Got 5 3 Mp cams (DS-2CD2132-I) connected - should be "within operating limits"??

Once you have proved that the hard drive is ok, then just for a laugh ... (indulge me, lol) .. reduce the number of cameras or lower them all to 720p / 1080p and see if the lighter load helps....

If you have a monitor connected to the vga or hdmi output, then you can see the bandwidth your using by going to... Maintenance / Net Detect / Traffic

 

[thestooge]

mine does this everynow and then, figure it was from the heat cos it is located in the pantry. might add some vents up there.

coincidentally, it mostly happens when i'm blasting through the menu options too fast, it's like the system can't keep up. that said, I was having dinner the other day and I heard the unit reset of its own accord, shrug.

 

[DrBo]

Once you have proved that the hard drive is ok, then just for a laugh ... (indulge me, lol) .. reduce the number of cameras or lower them all to 720p / 1080p and see if the lighter load helps....

If you have a monitor connected to the vga or hdmi output, then you can see the bandwidth your using by going to... Maintenance / Net Detect / Traffic

The HDD (WD Purple 10PURX) is the successor to a 2011 WD drive approved by HikVision - so don't believe this is an issue (even though the distributor is also pushing for me to replace the HDD - would hate to shell out another $100,- for a new drive, though ).

I have now lowered the res for 3 of the 5 cams to 720p (remaining 2 most critical ones so far stays at 3 Mp). Using iVMS-4200, I noticed an "idle" (= very static scenes) CPU load of ~60% (mem load ~20%; network load <2%). Did however see CPU load jump to ~90%(!!) when motion (trees affected by wind) was detected.

I am now leaning more and more towards the problem being caused by overload of the NVR CPU - it simply does not seem capable of handling 5 x 3 Mp cams, when "a lot is going on" (= scenes being very dynamic). I am still waiting for some information from HikVision on what exactly triggers "Abnormal Shutdown" - there must be a programmer who put this in as a result of one or more scenarios happening.

 

[DrBo]

mine does this everynow and then, figure it was from the heat cos it is located in the pantry. might add some vents up there.

coincidentally, it mostly happens when i'm blasting through the menu options too fast, it's like the system can't keep up. that said, I was having dinner the other day and I heard the unit reset of its own accord, shrug.

Does it "only" reboot - or does it actually go into a coma (i.e., becomes non-contactable)? Have you tried looking in the Log to see what it has reported (think you would need to search for "Major Type: Operation" around the time of the problem)? What is your setup (i.e., how many cams at what res and frame rate?) Do you see any correlation between how dynamic the scenes are and when the reboots occur? For a period of time, it was quite windy here, so a lot of motion detected (even with sensitivity set to low) - and judging from the log, it seems the NVR was overloaded..

 

[thestooge]

Does it "only" reboot - or does it actually go into a coma (i.e., becomes non-contactable)? Have you tried looking in the Log to see what it has reported (think you would need to search for "Major Type: Operation" around the time of the problem)? What is your setup (i.e., how many cams at what res and frame rate?) Do you see any correlation between how dynamic the scenes are and when the reboots occur? For a period of time, it was quite windy here, so a lot of motion detected (even with sensitivity set to low) - and judging from the log, it seems the NVR was overloaded..

same WD purpx (3TB) drive and 3MP cams connected to all 8 inputs here, but no real correlation to what is happening in the videos. under "operation" the log shows "abnormal shutdown" but the system never goes into a coma as such. it stops responding, and then half a minute later it is back up again (after beeping and rebooting itself), and there is the "abnormal shutdown" log recorded at the same time. as mentioned, only seems to be when hammering the menus too fast.

 

[DrBo]

same WD purpx (3TB) drive and 3MP cams connected to all 8 inputs here, but no real correlation to what is happening in the videos. under "operation" the log shows "abnormal shutdown" but the system never goes into a coma as such. it stops responding, and then half a minute later it is back up again (after beeping and rebooting itself), and there is the "abnormal shutdown" log recorded at the same time. as mentioned, only seems to be when hammering the menus too fast.

Hmm... what resolution & frame rate are you running the 8 cams with?

 

[DaveP]

I am now leaning more and more towards the problem being caused by overload of the NVR CPU - it simply does not seem capable of handling 5 x 3 Mp cams, when "a lot is going on"

I think your correct, there is advice on the internet that would support your leanings

The best way of keeping a watch on the nvr load is with telnet, I'll make a dummys guide post in a separate thread

 

[DrBo]

I think your correct, there is advice on the internet that would support your leanings

The best way of keeping a watch on the nvr load is with telnet, I'll make a dummys guide post in a separate thread

Thanks - I'll fit in the dummys category in the telnet-regard, I think..

 

[DaveP]

Enjoy

http://www.ipcamtalk.com/showthread.php?2071-Telnet-Hikvision-NVR-a-simple-guide

 

[DrBo]

Enjoy

http://www.ipcamtalk.com/showthread.php?2071-Telnet-Hikvision-NVR-a-simple-guide

Super-cool - thanks!
I'm just a little surprised about your apparent low-single-digit percentage CPU load? :wondering: What do you have on the NVR (no. of cams; resolution; frame rate)?? As mentioned above, I run at ~60% CPU load with my 5 x 3 Mp cams...

 

[DrBo]

I think your correct, there is advice on the internet that would support your leanings

Just did a bandwidth calculation using http://www.jvsg.com/ip-camera-calculator/ - and my 5 cams should "only" be able to provide a total of ~37 Mbit/s running full 3 Mp; 24 fps; H.264-10 compression (high quality). The NVR should be able to handle 40 Mbit/s "incoming"...?!?! :notrust:

 

[DaveP]

The only way to see the true real time bandwidth... connect a monitor and a mouse to the nvr

Maintenance / Net Detect / Traffic

 

[thestooge]

Hmm... what resolution & frame rate are you running the 8 cams with?

5 at 1080, 3 at 720 from memory. Quite extensive motion detection so it's not like all of them are recording constantly. i would say at any given time 2-3 might be recording something.

 

[scn101]

I had the abnormal shutdown issue too. It turned out to be my hard drive. I was using a regular hard drive and switched it to a Seagate video drive and then a WD purple. The abnormal shutdowns ceased to occur with either drive. I know you're running a WD purple but if you have another drive to test it may be worth a shot.

 

[psicobelico]

Same problem here...

NVR: DS-7608NI-E2
HDD: Seagate SV35 Series ST1000VX000 1TB 6Gb/s Internal Surveillance Hard Drive
For now running only 2 cameras DS-2CD2032F-I (20mts away from NVR)
All the hardware were bought brand new 2 weeks ago..

12016-02-07 12:32:07 OperationLocal: Abnormal Shutdown

22016-02-07 14:21:35 OperationLocal: Abnormal Shutdown

32016-02-07 18:54:41 OperationLocal: Abnormal Shutdown

42016-02-07 20:51:32 OperationLocal: Abnormal Shutdown

52016-02-07 22:11:46 OperationLocal: Abnormal Shutdown

62016-02-07 23:20:40 OperationLocal: Abnormal Shutdown

 

[DrBo]

I was through all the (probably well-meant) recommendations from my supplier - changing HDD (from a WDC Purple drive to a Seagate SV35); playing with number of cams; playing with resolution and Motion Detection sensitivity (even though I was well below the limit for "incoming bandwidth" (max. 40 Mb/s); upgrading/downgrading firmware - and just about everything else - to absolutely no avail. :sad2: My ordeal went on for months!

After a lot of investigation, I realized that the problem might be because of the NVR + some or all of the cams being parallel-imported China-versions (i.e., products made for the domestic Chinese market and not meant for export).

I finally gave up on solving the problem doing the things mentioned above - and decided for a simple but as it turned out very efficient solution: I simply fitted a digital timer with 1-minute resolution between the power outlet and the NVR, and had it turn off power for 1 minute every day at a given time. Thereby the NVR was cold-booted once a day - and hey Presto! The infamous Abnormal Shutdown issues gone!

The bottom line: I believe the problems with my frequent Abnormal Shutdowns originate from bad programming somewhere (i.e., some piece of program not cleaning up after itself resulting in stack overflow or the like); maybe in combination with NVR and cams being incompatible to some extent. By hard resetting the NVR, all memory is cleared and whatever mess left by the program is also gone. Only negative thing is that I have a 1-minute slot (+ the time it takes the NVR to boot) every day without the system working - but it is still MUCH better than the alternative with a randomly non-working system!

 

[psicobelico]

DS-7608NI-E2 V3.4.1
2 or 3 "abnormal shutdown" events daily.

Downgraded from V3.4.1 to V3.3.4 a week ago.
Zero "abnormal shutdown" events after that.[h=2][/h]