Dahua NVR Reboots during the day

[Tygunn]

And the verdict is: dahua_backtrace_callback 11

[0mgetIntelliFrameInfo success !!!!!!!!!!!
[33;40m07:52:48|[NetApp-310260] warn tid:1160 CWlanConfigInterface::WLanConnChk. no wlan devices !
[0m[0;32;32m[libvo] Resize blit cnt:114688
[mLIBHD_MSG : poll timeout.
[37;40m07:52:48|trace CConfigManager::getConfig 'DisplaySource', but default config is not set yet!
[0mgetIntelliFrameInfo success !!!!!!!!!!!
[37;40m07:52:49|trace CDriverManagerImp::OnHotPlugProc
[0m[37;40m07:52:49|trace hdd_mask = 0
[0m[36;40m07:52:49|debug tracepoint: Storage/DriverManagerImp.cpp, 2636.
[0m[33;40m07:52:49|[Unknown] warn EventManagerClient.cpp 162 tid:1216 ------EventMagRevCallBack---------AttachHandle:13096960987817388012
[0m[37;40m07:52:49|trace CConfigManager::getConfig 'VideoInAnalyseRule', but default config is not set yet!
[0m[37;40m07:52:49|[NetApp-310260] trace tid:1208 Src/PushNotification/ProviderManager.cpp, 362, code_type:CrossLineDetection is not been subscribed!
[0mgetIntelliFrameInfo success !!!!!!!!!!!
dahua_backtrace_callback 11
Time : Tue May 9 07:52:49 2017

=========================== TRACE START ===================================
Tid:1164, ExceptAuto login as root ...
login[8102]: root login on 'ttyS000'




=== Next failure ===

[0m[33;40m08:40:22|[Unknown] warn EventsDataBase.cpp 253 tid:1207 SaveToDataBase----------EventType:0 eventTime:2017-05-09 08:40:21
[0m[33;40m08:40:22|[Unknown] warn EventsDataBase.cpp 256 tid:1207 SaveToDataBase----------EventType:0 systemTime:2017-05-09 08:40:22
[0m[32;40m08:40:22|[libDatabase] info tid:1207 CDBCustom::Insert bRet : 1 sqlString : insert into EventsDataBase(EventTime, SystemTime, Channel, EventType) values(1494319221.000000, 1494319222.000000, 3, 0);, line : 79
[0m[33;40m08:40:22|[NetApp-310260] warn tid:1308 CWlanConfigInterface::WLanConnChk. no wlan devices !
[0mgetIntelliFrameInfo success !!!!!!!!!!!
getIntelliFrameInfo success !!!!!!!!!!!
[37;40m08:40:23|trace CDriverManagerImp::OnHotPlugProc
[0m[37;40m08:40:23|trace hdd_mask = 0
[0m[36;40m08:40:23|debug tracepoint: Storage/DriverManagerImp.cpp, 2636.
[0mgetIntelliFrameInfo success !!!!!!!!!!!
[33;40m08:40:23|[Unknown] warn SnapManagerClient.cpp 127 tid:1216 ---------------AttachHandle:13052219657665575050
[0m[37;40m08:40:23|[Unknown] trace SnapManagerRev.cpp 112 tid:1216 CIntelliEventsReceiveClient:ushRecvData -------------Ch[3]dwJsonLen[835]Length[721766]
[0m[33;40m08:40:23|[NetApp-310260] warn tid:1308 CWlanConfigInterface::WLanConnChk. no wlan devices !
[0m[33;40m08:40:23|[Unknown] warn EventsDataBase.cpp 253 tid:1207 SaveToDataBase----------EventType:0 eventTime:2017-05-09 08:40:21
[0m[33;40m08:40:23|[Unknown] warn EventsDataBase.cpp 256 tid:1207 SaveToDataBase----------EventType:0 systemTime:2017-05-09 08:40:23
[0m[32;40m08:40:23|[libDatabase] info tid:1207 CDBCustom::Insert bRet : 1 sqlString : insert into EventsDataBase(EventTime, SystemTime, Channel, EventType) values(1494319221.000000, 1494319223.000000, 3, 0);, line : 79
[0m[37;40m08:40:23|trace CConfigManager::getConfig 'DisplaySource', but default config is not set yet!
[0m[33;40m08:40:23|[Unknown] warn EventManagerClient.cpp 162 tid:1216 ------EventMagRevCallBack---------AttachHandle:13053405584116015660
[0mgetIntelliFrameInfo success !!!!!!!!!!!
dahua_backtrace_callback 11
Time : Tue May 9 08:40:23 2017

=========================== TRACE START Auto login as root ...
login[3712]: root login on 'ttyS000'



BusyBox v1.18.4 (2016-07-21 18:30:16 CST) built-in shell (ash)
Revision: 11799
Enter 'help' for a list of built-in commands.




=== Next failure ===

[0m[36;40m09:18:04|debug tracepoint: Storage/DriverManagerImp.cpp, 2636.
[0mgetIntelliFrameInfo success !!!!!!!!!!!
[33;40m09:18:05|[NetApp-310260] warn tid:1140 CWlanConfigInterface::WLanConnChk. no wlan devices !
[0mgetIntelliFrameInfo success !!!!!!!!!!!
[37;40m09:18:05|trace CDriverManagerImp::OnHotPlugProc
[0m[37;40m09:18:05|trace hdd_mask = 0
[0m[36;40m09:18:05|debug tracepoint: Storage/DriverManagerImp.cpp, 2636.
[0mgetIntelliFrameInfo success !!!!!!!!!!!
getIntelliFrameInfo success !!!!!!!!!!!
getIntelliFrameInfo success !!!!!!!!!!!
[33;40m09:18:06|[Unknown] warn EventManagerClient.cpp 162 tid:1216 ------EventMagRevCallBack---------AttachHandle:13099108230976948252
[0m[37;40m09:18:06|trace CConfigManager::getConfig 'VideoInAnalyseRule', but default config is not set yet!
[0m[37;40m09:18:06|[NetApp-310260] trace tid:1208 Src/PushNotification/ProviderManager.cpp, 362, code_type:CrossLineDetection is not been subscribed!
[0mgetIntelliFrameInfo success !!!!!!!!!!!
dahua_backtrace_callback 11
Time : Tue May 9 09:18:06 2017

=========================== TRACE START ================Auto login as root ...
login[8388]: root login on 'ttyS000'



BusyBox v1.18.4 (2016-07-21 18:30:16 CST) built-in shell (ash)
Revision: 11799
Enter 'help' for a list of built-in commands.

~ # [ 2083.526114] MMB LEAK(pid=1017): 0x7B526000, 4149248 bytes, 'tde_user'
[ 2083.532585] mmz_userdev_release: mmb<0x7B526000> mapped to userspace 0xa6f55000 will be force unmaped!
[ 2083.542007] MMB LEAK(pid=1017): 0x7B91B000, 4149248 bytes, 'tde_user'
[ 2083.548465] mmz_userdev_release: mmb<0x7B91B000> mapped to userspace 0xa6b60000 will be force unmaped!
[ 2083.557793] MMB LEAK(pid=1017): 0x7BD10000, 4149248 bytes, 'tde_user'
[ 2083.564264] mmz_userdev_release: mmb<0x7BD10000> mapped to userspace 0xa676b000 will be force unmaped!
[ 2083.573624] MMB LEAK(pid=1017): 0x7C105000, 65536 bytes, 'AO(1,0) CirBuf'
[ 2083.580450] mmz_userdev_release: mmb<0x7C105000> mapped to userspace 0xb5d17000 will be force unmaped!
[ 2083.589792] MMB LEAK(pid=1017): 0x7C115000, 65536 bytes, 'AO(1,1) CirBuf'
[ 2083.596586] mmz_userdev_release: mmb<0x7C115000> mapped to userspace 0xb5d07000 will be force unmaped!
[ 2083.605928] MMB LEAK(pid=1017): 0x7C125000, 65536 bytes, 'AO(0,0) CirBuf'
[ 2083.612730] mmz_userdev_release: mmb<0x7C125000> mapped to userspace 0xa675b000 will be force unmaped!
[ 2083.622063] MMB LEAK(pid=1017): 0x7C135000, 65536 bytes, 'AO(0,1) CirBuf'
[ 2083.628895] mmz_userdev_release: mmb<0x7C135000> mapped to userspace 0xa674b000 will be force unmaped!

Special NAND id table Version 1.36
Nand ID: 0xEF 0xF1 0x80 0x95 0x00 0x00 0x00 0x00
stmmac_init,550,0x70431,3
MAC: 00-00-23-34-45-66

get_update_state,266,55,ff
get_update_state,266,aa,ff
get_update_state,266,5a,ff
*** INFO *** dul-backup state: NORMAL BOOT fail=0.


U-Boot 2010.06-svn2362 (Jul 21 2016 - 16:46:32)

 

[DavidDavid]

So what does that mean for those of us that don't speak robot?

 

[HMS]

So what does that mean for those of us that don't speak robot?

+1

 

[CaliGirl]

Following...

 

[aristobrat]

So what does that mean for those of us that don't speak robot?

IMO, it should give some specifics for the Dahua engineers to look at, vs the random "try this..." stuff they've suggested so far.

 

[Tygunn]

+1

I'm sure that is what maps to 0x06? Hahah. I'll send to Andy none the less.

 

[Tygunn]

I've sent Andy the log snippets as well as the full logs. Hopefully the software folks at Dahua can figure something out.
For now, my NVR is set to only record continuous and to ignore events completely (motion and IVS). I'll see if that increases stability.

 

[Del Boy]

I've sent Andy the log snippets as well as the full logs. Hopefully the software folks at Dahua can figure something out.
For now, my NVR is set to only record continuous and to ignore events completely (motion and IVS). I'll see if that increases stability.

Good news.

Stating the obvious maybe but when you are in and can have no recording - you have tried no HDD connected? Common problem in Hikvision NVRs with reboots is the HDD and not the NVR.

 

[CaliGirl]

During playback on my 4k Dahua NVR, if I move the cursor around too much to "srub" the footage the NVR crashes at times. It doesn't come back up on its own unless I power cycle it. Luckily I have a remote way to power cycle it or it would be dead for months.

It is certainly possible to overload the NVR and cause it to crash at times

 

[weigle2]

@Tygunn - Excellent work getting console messages and being able to send them to Andy. Hopefully, the engineers can review this and come up with a possible firmware update. If not, at least make suggestions on how to eliminate/minimize the reboots. Thanks, again.

 

[Tygunn]

Good news.

Stating the obvious maybe but when you are in and can have no recording - you have tried no HDD connected? Common problem in Hikvision NVRs with reboots is the HDD and not the NVR.

These are brand new hard drives, both used for less than a couple months.
The SMART diagnostics for the drives shows no issues at all. However who knows.
I wouldn't be surprised if the firmware on these NVRs is over sensitive to slight glitches in drives. I've NEVER had an issue in ~20 yrs using hard drives of a drive causing a crash like this though. So these NVRs would really be flaky snowflakes.

 

[HMS]

@Tygunn Where these new recorded reboots with the Hik camera online withe the NVR or off (think i remember you said you might put it on your BI machine)

 

[Tygunn]

@Tygunn Where these new recorded reboots with the Hik camera online withe the NVR or off (think i remember you said you might put it on your BI machine)

With these latest crashes the HIK camera was indeed connected to the NVR.
I'm going to run with all cameras in record-only for a while and see how it goes.
If after a couple weeks its stable I will try re-enabling the motion detect events on the HIKVISION camera and see if it causes any problems.

 

[alastairstevenson]

From looking at what you've logged of the activity preceding the crashes there is a pretty good indication as to the cause, which for the firmware developer should allow the logical inconsistency to be identified and fixed.
Pretty good work, in my view.

By the way - for the benefit of the many readers (and please treat this as constructive, not critical), it's good to enclose things like log extracts in the code tags of the forum, to be found via the '+' in the edit menu. Long extracts are not a problem when using them, and they format better when the lines are long.

 

[Del Boy]

These are brand new hard drives, both used for less than a couple months.
The SMART diagnostics for the drives shows no issues at all. However who knows.
I wouldn't be surprised if the firmware on these NVRs is over sensitive to slight glitches in drives. I've NEVER had an issue in ~20 yrs using hard drives of a drive causing a crash like this though. So these NVRs would really be flaky snowflakes.

I've had issues with brand-new drives that pass SMART so that really means nothing unfortunately. And connecting HDDs to an NVR is different to a computer too. So the HDD is still a possibility.

I keep saying it but don't mix and match Hikvision with other brands. Dahua you can mix better but I still recommend not to mix big brands.

 

[alastairstevenson]

Bear in mind it's not just a single NVR that's experiencing these rebooting issues - there have been multiple reports, which suggests a single faulty HDD is not the common cause.
The log extracts suggest a firmware flaw.

 

[Del Boy]

Agree it's probably not the HDD but it can be an issue - so always worth trying it disconnected when you have problems.

 

[TechBill]

Andy from EmpireTec just posted that there a new firmware out for this NVR last night

Dahua NVR 5216-16P-4KS2

 

[Tygunn]

Agree it's probably not the HDD but it can be an issue - so always worth trying it disconnected when you have problems.

Got a feeling that would be the same as disconnecting the cams.

Sent from my Pixel XL using Tapatalk

 

[Del Boy]

Got a feeling that would be the same as disconnecting the cams.

Sent from my Pixel XL using Tapatalk

Sometimes you've got to try to do stuff in order to diagnose and fix things... you never know and it's not a huge effort to unplug some RJ45s from the back of an NVR.