URGENT: HDD pinging at 100% constantly. Help please!

[IAmATeaf]

So your Z drive is clearly a bottleneck, especially when writing it to it. Download the manufacturers utility to test it and perform a long test on the drive to see if it passes.

Maybe also have a look at the SMART stats of the drive to see if there’s anything obvious.

Me personally, I write nothing to the SSD at all except the BI database, everything else gets written to the spinners and I also don’t move things around between New and Stored as there’s really no point doing this on the same drives within the PC.

 

[delphinus]

Sorry for the conflicting information, I'm new here and probably those guys know more than me but I'm an IT for many years and all the information that need to be processed is stored in the fastest drives if you have a SQL or Exchange server the immediate data produces log files that are placed in fast disks and at the end of day after the backup the data in the logs is placed in the databases, in my case I always try to have has much ram as the databases size.

With CCTV my opinion is that alerts and new (I call them live) should be stored the fastest drive, the same goes for the database of BI + deepstack and the storage should go to large mechanical disks.
I manage the servers of a quite larger system of about 400 cameras and what we do is we store the images in SSD disks and every 8 hours we transfer the images to mechanical disks, this way we don't have loss of frames because 2 or more persons had to revise some footage.

You are right the disk wear out faster and my experience is that they last much less than brands say but if you want security you have to invest in it, probably the hours that you spend trying to fix your problem cost more than 2 or 3 ssd's. One Samsung 980 Pro 1Tb that has 5 years warranty cost in amazon 165€ that is 2.75€ a month during those 5 years 4.12 AU $, other option is to go for the EVO or QVO that have 3 years and cost 50% less but after the warranty you can buy a bigger sdd for that price difference.

 

[delphinus]

“Why would the network drive have anything to do with it? I've completely bypassed the network drive and only relying on the Z: drive for now”

It shows the quantity of data that you transfer and when is written to your network drive the data to the other disks has to wait and vice versa.

Microsoft by default create very large swap files you have 16Gb don’t go over 1.5x use a max of 24Gb and make sure you turn of the swap file in your mechanical disk, use it only in your SSD. I run my BI in a virtual machine with 6Gb and only use 58% of ram with 6 cameras.

There is no doubt that you have a problem, test the disk and also make sure you have the latest drivers and firmware, don’t use Microsoft drivers get them in the brand website.

“At first it immediately shoots up to over 1GB/s and then plummets to around 50-100MB/s.

Does this make sense?”

Yes makes sense, when you start a copy it goes to ram so it’s very fast when ram is full you get the real speed. This is why you never remove a pen without first eject it because when it display that a copy ended in reality is still copying from ram to the device.

I assumed that your C: is the SSD and your Z: is the mechanical but your prints indicate the opposite 27.5 in one case and 64.2, this last value is slow but OK but the 27.5 was low 15 years ago, also in CCTV you don’t write 2Gb the speed with small files will be much worst.

 

[delphinus]

Me personally, I write nothing to the SSD at all except the BI database, everything else gets written to the spinners and I also don’t move things around between New and Stored as there’s really no point doing this on the same drives within the PC.

This is other option and for home use probably it makes more sense than mine has long it's used a CCTV HD.

For this scenario what I would do is separate the recordings of each camera, camX goes to auxX, I already have 18000 files, open the stored folder with 30 days recordings can produce lag and also the preview should be turn off.

 

[IAmATeaf]

As far as I know Windows 10 won’t cache disk writes to RAM? If the disk write cache option is enabled then the drive itself will cache writes using its onboard cache.

I seem to recall there being a setting for write-caching but can’t be sure.

Bit off topic but on my server which has SAS RAID controllers with write caching enabled the cache has a battery backup so in the event of a power failure the cache data isn’t lost.

 

[wittaj]

Corporate systems versus home systems are quite different.

This has been tested here by many people.

For a home setting, Windows, BI, the BI database, Deepstack go on the SSD. The video goes on a surveillance rated HDD.

You are wasting CPU% and wearing out the drives faster moving video files around from New to Stored, especially if they are on the same drive. Only if you have a NAS.

There is no lag leaving the video all in New.

If you have more than one HDD in the system, then you split the cameras across the HDDs. This is much better than having one drive for New and the other for Stored. This way if one HDD poops out, you still have half the camera recordings.

 

[Corvus85]

I assumed that your C: is the SSD and your Z: is the mechanical

It is.

Does this change your assessment?

Maybe also have a look at the SMART stats of the drive to see if there’s anything obvious.

Where do I find this?

 

[Corvus85]

I asked this on the last page, but can anyone tell me if there's anything I need to do in BI before I format the HDD?

 

[wittaj]

I would disable the service and shut BI down and delete the DB folder and then format the HDD and once complete then start BI back up.

 

[delphinus]

As far as I know Windows 10 won’t cache disk writes to RAM?

To explain for everyone I have to simplify and you are right it doesn't by default but can be done.

NEVER ENABLE WRITE CACHE EVEN IF YOU HAVE BATTERY.

How to manage disk write caching for external storage on Windows 10

You can manage how disk write caching operates on Windows 10, and in this guide, we show you the steps to complete this task.

www.windowscentral.com

Also by default windows use read cache and that goes to memory to write it has to read first.

 

[delphinus]

It is.
Does this change your assessment?
Where do I find this?

Yes because drives read faster and write slower, you should had a faster copy to the SSD and a slower to the other, don't lose more time, life is short.

Update windows drivers, update disk firmware, backup you disk and format it, it's the easy thing to do, I hope to be wrong but seems it won't fix it but you have to start from somewhere.

 

[Corvus85]

update disk firmware

How do I do this? Through device manager?

 

[delphinus]

How do I do this? Through device manager?

No, you have to go your disk brand, support and search what they have for your model, some brands don't provide them to the regular user but if they have one install it and don't turn power off during the process.

 

[sebastiantombs]

One more comment to help confuse things.

When formatting a drive for video storage set the block size to 1024 rather than the default of 4 or 16. This will reduce block fragmentation on the drive and help keep things more organized and snappy.

 

[IAmATeaf]

To explain for everyone I have to simplify and you are right it doesn't by default but can be done.

NEVER ENABLE WRITE CACHE EVEN IF YOU HAVE BATTERY.

How to manage disk write caching for external storage on Windows 10

You can manage how disk write caching operates on Windows 10, and in this guide, we show you the steps to complete this task.

www.windowscentral.com

Also by default windows use read cache and that goes to memory to write it has to read first.

That article is a bit confusing, talks about external drives and safely remove options? It could maybe apply to some SATA controllers that support hot plug?

Assuming you aren’t using W10 Home and you enable this policy option and it works on your internal drives and then I personally wouldn’t do so unless your system is properly battery backed up.

 

[IAmATeaf]

As I advised a while back, go to the disk manufacturers website, find their Support page, download their disk utility and perform a long test in the drive to eliminate the drive.

You are going to have to do this in stages, checking the drive is good should be nice and easy to do.

 

[Flintstone61]

You should probably refresh everybodies memory ( perhaps a photo) of this computer. Is is a micro form factor from a HP or Dell? Is it a home built micro? is it a full tower? is it using So-dimm for memory? Is it a clean install of Windows?
How is it that my lowly i5-4590 in an old Dell 9020 tower, can run 9 cameras, and surf the web, and watch you tube movies, and run Full image backups with out a hiccup?
How is it that my old i7-3770 Dell 7020 ran 15 cameras at 9% cpu and did not hit 100% disk usage? Of course none of these were storing video on a 2.5" laptop drive.
What brand and make of 2.5" hard disk are you using? Have you verified whether or not you're using an SMR Drive?

 

[Corvus85]

Is is a micro form factor from a HP or Dell?

It's a home built micro form factor in an In Win Chopin chassis.

Is it a clean install of Windows?

Yep. Clean install of Windows 11.

How is it that my lowly i5-4590 in an old Dell 9020 tower, can run 9 cameras, and surf the web, and watch you tube movies, and run Full image backups with out a hiccup?

That's the million-dollar question.

What brand and make of 2.5" hard disk are you using?

It's a WD Blue 2TB - WD20SPZX

Have you verified whether or not you're using an SMR Drive?

According to the WD website, it's an SMR drive.
WD Blue SATA 2.5" Laptop Hard Drive HDD | Western Digital

Currently downloading the 'Western Digital Dashboard' to perform tests and see SMART info.

 

[Flintstone61]

Have you tried Win 10 on this hardware?
some people were having issues with w11 and Bi

 

[Corvus85]

Have you tried Win 10 on this hardware?

Not yet, but will do so after ruling out hardware faults/inadequacies.

FYI, here's the CrystalDiskInfo screenshots for both my OS SSD and 2.5" HDD.




Anything out of the ordinary here?