Disk cluster size and Blue Iris

[SouthernYankee]

I am running a 4TB disk for the new folder in blue iris with 13 camera recording, the majority of the files are around 2 GB for a 1 hour recording time. There are no other programs using the disk other than windows system files. The disk was formatted for a windows cluster size of 4K (the default). The disk had had 3.4 TB allocated to Blue iris new folder and 0.22 TB not allocated (free space).

Using the Passmark OSforensics tool to examine the file fragmentation. Each file over 2GB had on average more than 4100 fragments. I would expect a highly fragmented files as there are 13 files being written and growing simultaneously with overlapping write allocations to the disk.

As a side note, I shut BI down and tried to defragment the drive, the windows defragmenter ran for more than 70 pass in more than 24 hours and was unable to defragment the drive, this is also expected as the low amount of free space, prevent the defragmentation from running efficiently.

So I reformatted the 4TB drive with a cluster size of 64K with the same Blue Iris settings as before. After running till all the free space was used, Each file over 2GB had on average more than 1200 fragments.

This is not what I expected as the 64K cluster is 16 times bigger than the 4K cluster, I what have expected the fragmentation to be about 16 times less. This obviously is not the case.

This is not a problem, it was just something I was interested in. I strongly recommend to NOT defragment a video drive. The BI Video drive has a very high write rate compared to the read rate. It has very large files compared to a normal disk.

 

[Tinman]

Interesting, I just so happened to look at my drive chart and noticed The F: DRIVE was not using all of the allocated space. They usually are both the same as they are identical drives both set for 3550GB. So at first I thought maybe I have a issue with that drive and ran a check for errors. No errors were found, but it was showing drive fragmented at 99% ! The E: DRIVE was showing it was at 5% fragmented, UNTIL I analyzed it then it also showed 99% ?? Then I did a search here and found this thread. Has something changed in BI ? Running 5.0.8.2 I have tried to run repair/compact with no changes to chart. I first noticed this issue a few days ago because I set the repair/compact schedule to once a week instead of daily when Ken added that feature. So my big question is...can we rely on W10 to defrag the drives because both of the BI 4TB purple drives as well as my OS SSD were selected ...yet they weren't being done ??

 

[SouthernYankee]

Tinman

Nothing has change to my knowledge in BI for file allocation. It is just the way a windows drive will behave in a high write environment, with multiple writers. The disk files are growing slowly time, unlike a file copy, In BI a single file may take between 1 and 8 hour to write. For a 2mp camera at 15 FPS, a file can growing at the rate of 1/2 Megabyte per second. If you have a lot of cameras they are allocation space on the hard drive every second. This will cause the space to get chopped up. You delete a file and it leave a lot of little free space block all over the place. Which will be filled in by next set of group write.

Defragmenting the drive is a complete waste of time and resource of a video security drive. The video drive has a very high write rate and a very low read rate, this is completely different than other application behavior. The only reason to defragment a drive is to support a high read rate, with repetitive reads of the same file.

Based upon the assumption that the life of a disk drive is related to the number of read, number of writes and the number of head movement, I was looking at reducing this .

Defragmenting a BI video drive with less than 10% free space more than likely will not every complete with zero fragmentation.

 

[Tinman]

Yes...defrag is impossible. I have never seen this issue before as why I was asking. First I did a repair/regen the database to see if that was giving a wrong reading on chart. Nope, same result, so I just deleted the DB and formatted both drives again using the default 4096 cluster. Also note I did turn OFF drive indexing and unchecked the option to auto defrag each drive as well. All seems good for now and in a couple of weeks I will see what the chart shows and reoprt back. BTW, I wasn't experiencing any issues with BI, just noticed the chart looked goofy. I record my clips continuous @ 30 min. and my cameras are split about 50/50 between drives.

 

[SouthernYankee]

Splitting load between multiple drives is my recommend way to go. Moving files around is a waste of resources. Also always leave a chunk of unallocated space for BI, i leave about 10% unallocated.

The space unbalance between your two drives may be cayse by the sum of the New+stored+alert space and the space of aux1. Look at the space allocation in BI for all the folders. If not using stored set the size to zero

 

[Tinman]

Splitting load between multiple drives is my recommend way to go. Moving files around is a waste of resources. Also always leave a chunk of unallocated space for BI, i leave about 10% unallocated.

The space unbalance between your two drives may be cayse by the sum of the New+stored+alert space and the space of aux1. Look at the space allocation in BI for all the folders. If not using stored set the size to zero

Yes, I don't use alert folder and both my drives are configured the same. There will always be a small difference between drives due to cameras have different FPS settings, but BOTH should reach the full allocated 3550GB specified....and F: wasn't going there. It use to, but something got hosed somewhere. I hope a fresh start will cure what it was, just wondering if it had become so fragmented to cause this issue. Time will tell Thanks for your input !

 

[SouthernYankee]

Just an update on additional testing of cluster size. I set the cluster size to 1 Megabyte (1024Kilobytes or 1,048,576bytes) . The video files around 2 GBytes in size have 140 fragments, so on average a fragment is about 15 MB per fragment. This is much better that the default 4K clusters.

I would recommend that for a disk used for video storage, continuous recording that it be formatted with a cluster size of 1024 kilobytes.

 

[samplenhold]

So here is a couple of questions that is related to this thread. I have two WD Purple 10TB drives that I split my cameras between in BI. This thread has caused me to investigate those drives.
1. In WIN10 drive properties, General tab, one of these drives has the 'Allow files on this drive to have contents indexed in addition to file properties' and the other drive does not have this checked. Pros and cons on this option?
2. Under Hardware Properties/Policies there is a 'Write-caching policy'. Should that be enabled? If so, should one select 'Turn off Windows write-cache buffer flushing...'?
3. I have space for a third HD and was thinking of putting a third WD Purple 10TB drive to enable keeping clips for a few more days. How can I tell if the MB/processor etc can handle all that writing without bogging down?



Right now I have 18 cameras all but one are 2mp and one is 4mp Dahuas. Will be adding at least two more in the next couple of months. They are constantly recording. Currently showing Totals in BI as 7858.30 kB/s 583.3 MP/s.

 

[SouthernYankee]

I use write caching, as this will mostly preventing rewriting the same block over and over, as data is added to the cluster.

I have indexing turned off, what is there to index.

Looking at your data, in my opinion there is not enough free space. I would recommend at least 50GB free. Not allocated in BI. I normally recommend at least 10% free space. Windows OS needs space on the drive to manage NTFS. Also BI some times writes more data than it has allocated. This happens when files are being moved or deleted

The use of an additional internal drives will not have much if any effect on the CPU. May use additional memory for caching.