Random outage on certain cameras.

[sdyfgasd]

I can't seem to track down the issue. Cameras are a mix of Reolink, Dahua, and Hik clones. The brand is irrelevant, as I have the same model camera with issues while another one is perfectly fine.
If I restart BlueIris and reboot the switch, all cameras will come back online eventually. But within a few minutes, I see dropped frames on certain cams. Leave it for a night, and a bunch of them will be offline. I can disable/enable the camera, and it will come back online, too.
The issue doesn't happen with just 10 cameras. But as I put all of them online, this starts happening. I've played with settings: UDP, no authentication, RTSP, re-encoded/direct to drive. Nothing seems to make a difference.
I've swapped the switch and added a dedicated network card. Still no joy. Wiring checks out for PoE reliability.

i7-9700, 16GB, Mobius storage array over USB3 in JBOD with 5x8TB 7200RPM drives.

 

[SouthernYankee]

1) Please provide a network diagram.
2) are any if the cameras connected WiFi ?
3) Does the camera network traffic pass through the router ?
4) it is NOT recommended to do record writes to a remote storage array, I would recommend writing to an internal drive (new) then move completed files to the storage array (stored)


Please provide a screen shots. full frame (use windows 10 snip & sketch tool)
1) windows task manager process tab sorted by memory (most at the top), Must contain, memory, disk, network, GPU, GPU engine columns
2) windows task manager performance, GPU
3) Blue Iris Setting about tab
4) Blue iris status (lighting bolt graph) clip storage tab

 

[sdyfgasd]

I'll be damned. The storage was the issue. I've set half the cameras to record to internal drive, the rest to USB array. Not a single failure overnight. Not sure why I didn't check this before. I'me going to put another Mobius box and split the workload.

 

[SouthernYankee]

My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders

Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.

 

[sdyfgasd]

Well, I spoke too soon. Still doing it.



I'm curious why BI stops attempting to reconnect at some point. The cam is obviously available, if all it takes is a toggle of any setting in the camera Properties to initiate a reconnect. And you can see half the cameras have a rare connection drop (1-4 in 50+ hours). When those happen, BI tries repeatedly and, eventually, reconnects. Does at some point BI give up?

 

[IAmATeaf]

You have got something seriously wrong, the number of disconnect/retries is way too high. I’d start by looking at each camera in turn trying to fix it before moving onto the next.

 

[SouthernYankee]

You have a major network problem.

1) Please provide a network diagram. All network equipment, make model, and local IP address.
2) are any if the cameras connected WiFi ? If so turn them off
3) Does the camera network traffic pass through the router ? if so re route the traffic to stay on the local network

 

[forzabiondo816]

My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders

Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.

Really appreciate the feedback I’m having similar issues and I’m doing different then what you advise with disk storage and such, what do you recommend as far as network config? I’ve got 3 4 port poe switches going into an eight port gigabit POE then the 8 port switch going into my router. Asus 1300g router, 2 tp link 4 port poes, 1 bv tech 4 port poe with 2 uplink.

 

[SouthernYankee]

Camera traffic NEVER goes through the Router on the local network.
There is no reason to chain POE switched into POE switches.

Connect the BI computer to a gigabit switch,
Connect the POE switches to a same gigabit switch,
Connect the Router to a same gigabit switch,

 

[shalem2014]

Well, I spoke too soon. Still doing it.

I'm curious why BI stops attempting to reconnect at some point. The cam is obviously available, if all it takes is a toggle of any setting in the camera Properties to initiate a reconnect. And you can see half the cameras have a rare connection drop (1-4 in 50+ hours). When those happen, BI tries repeatedly and, eventually, reconnects. Does at some point BI give up?

Sounds like a DNS or ARP issue. Make sure you've either assigned static IP address reservations to the cameras on your DHCP server, or have set static IPs on the cameras themselves. And then make sure you're addressing the cameras by IP address in Blue Iris and not by name (e.g. "http:// 10.0.0.229/" not "http:// cam-frontdoor/"). In practice, the latter should work fine, but I had the same problem you're having pop up randomly from time to time when I did it that way. ARP issues can crop up when there's a confusing daisychain of network devices and/or packet loss. Try to keep the network topology as simple as possible!

My Standard allocation post.

5) it is recommend to NOT store video on an SSD (the C: drive).

Good post, except I disagree with #5. This is how I set up most of my systems. Take the doomed-to-fail HDD out and replace it with a 500-2000 GB SSD (depending on the number of cameras). SSDs are a lot more hardy than most people think. I would recommend sticking with the Samsung EVO or PRO series devices though, as some of the cheaper brands aren't as robust. Just for an example, I will reference a six-camera system I installed using a cheap, 500 GB 860 EVO SSD ($70). To date, it has clocked 15,280 hours (1.7431 years) and 119 TB of writes. It is warrantied for 300 TB of writes, and studies indicate that it will be good for about 3000 TB before failing. So assuming the current rates persist, the warranty is will be good for about four years four months, and studies indicate that the SSD won't run out of write cycles for about 44 years! Even a quarter of that would outlive the average HDD, while being faster, quieter, and less sensitive to the elements (I often install these systems in attics or basements). And with the SSD for C:\, updates (especially Windows Updates) are completed quickly and easily.

 

[SouthernYankee]

@shalem2014

I have had two Samsung evo 120 GB fail on two different system . I did not use them for Video, just ever day usage on a home PC. They look like they were running but they spent most of there time doing error recovery , a system boot went for ~12 second to over 240 seconds. Look at the details of how a SSD write. The more you write to the same location the more likely they are to fail. Security video has a high write requirement and a low read requirement. The writeing to an NTFS logging files system increases the write load.

Try to collect on samsung warranty, it cost more to collect then the device currently costs.

One use for an SSD s to write video is if you have motion/event only recordings , an a low level of activity.

 

[shalem2014]

Were they the 840 EVO by any chance? That particular SSD has a severe firmware issue that can lead to very slow speeds on old data. The failure doesn't happen due to writes but do to the age of the written data and it needing to be "refreshed" to remain retrievable. I was actually one of the two programmers who wrote the investigative tools that unearthed that problem, causing many enthusiasts to put public pressure on Samsung to release a firmware update to address the problem. The updated firmware provided a better TLC read algorithm that could retrieve fading data better (now that they had studied how it faded), and a rewrite algorithm that would read data in the background and rewrite any cells that were fading to refresh them. I have the sneaky feeling that any TLC or QLC SSD would have this problem to some degree with similar mitigations, and could have serious issues with data retention if the SSD were shelved for several years without being powered up.

The "location" you write to, by folder or by sector makes absolutely no difference in wear as the SSD firmware constantly remaps frequently written sectors to less frequently written cells when new data is written as part of wear leveling. So the frequent updating of the NTFS log/journal won't wear the SSD any faster than a bunch of data being written, as the SSD will use fresh cells each time. Not only is this part of its wear leveling strategy, it is part of how it achieves fast write speeds: Erasing takes time, and so it writes new data to already TRIMmed cells, and ques the cell vacated by the old data to be TRIMmed in the background. Thus, the smaller the SSD, the greater the load on write cycles as it has fewer cells to spread the writes over.

The systems I set up are all set to record motion only, moderate level of activity—the amount of TB/year in my previous post should give a rough idea of the recording duty cycle (6 cameras, 2mp). The cost/reward ratio certainly shifts drastically in favor of HDDs when doing 24/7 recording due to cost/GB!