Drive free space is inaccurate

[Jeremy Boelter]

Hello!

Hoping someone here can help me through this -- i'm not sure when this started (within the last year, probably), but i'm getting frequent disk over-allotment errors and inaccurate free space reporting in BI.

I'll see the following warning when looking at the disk stats in the WebUI:



Prior to repair, the storage tab looks like:

(overflow on the G drive visualization? also almost the entire drive is allocated for BI, so i'm not sure where the green portion is coming from)

And after a repair:


The only solution i've found to date is to completely delete the \db folder and let it rebuild. When it regenerates the db, it typically will only find a few .bvr files (maybe just \new?) and i lose most of my history. I end up nuking everything (all storage and \db) every month or so and starting over.

Here are my storage configs:









And disk allocation in windows:



Any help here would be greatly appreciated -- I can attach logs / provide other screens or context if needed.

Thanks!

Jeremy

 

[samplenhold]

Why are you moving the video files around? That just wastes CPU cycles and bogs down the IO functions. Use that 16TB drive to store NEW and let it delete when it gets full. Same goes with drive I.

I am guessing that all of that constant movement of files is getting things over allocated during that move process.

 

[Jeremy Boelter]

Why are you moving the video files around? That just wastes CPU cycles and bogs down the IO functions. Use that 16TB drive to store NEW and let it delete when it gets full. Same goes with drive I.

I am guessing that all of that constant movement of files is getting things over allocated during that move process.

Thanks for the quick response -- using multiple drives as such:

H: = NVME for High IO, with the intent of near-instant retrieval of recent alerts and previews on the webui (correct me if i'm wrong there)
G: = External USB 3.0 for long-term storage
I: = 2nd External USB 3.0 for longer-term storage

It's enterprise hw so i'm really not concerned about burning cycles... tons of headroom--specifically two 24 core Xeon 2690v3s running @ around 18% load.

While I appreciate the notion of being frugal with resources, it's really not a limiting factor in this case. It appears this is a common scenario that 'Storage' is an external / network destination--both are discussed frequently here and also called out in the BI documentation.

Is it possible this setting may have an impact?


Perhaps lowering that size so we have more frequent but less sustained writing?

 

[wittaj]

Make sure you don't have system restore turned on for those drives and set the recycle bin to zero.

 

[Jeremy Boelter]

Make sure you don't have system restore turned on for those drives and set the recycle bin to zero.

Restore was off, but Recycle bin was on -- now disabled for those drives. Scrapping \db and rebuilding fixed it as usual (repair didn't change anything); i'll keep an eye on it.

 

[bp2008]

I've had similar problems to what you showed there, thankfully not every month, but still too often. October 2nd I discovered most of my cameras hadn't recorded anything for the past 4.5 days because Blue Iris forgot about most of the clips and let the disk get full. I went through much the same process trying to fix it. Repair/Regenerate took SEVEN HOURS and didn't fix anything. I had to delete the database and regenerate it from scratch. Took another seven hours. Meanwhile no recordings could happen because the rebuild procedure disables recording while it is active.

This is one of the reasons I run a second Blue Iris system on a different machine and do not update it at the same time as the main system.

I like Blue Iris, but I can't make excuses for the clip database. It is very unpleasant (that is the diplomatic way of saying it).

 

[samplenhold]

It's enterprise hw so i'm really not concerned about burning cycles... tons of headroom--specifically two 24 core Xeon 2690v3s running @ around 18% load.

How many cams and what are your kB/s and MP/s rates? For all of that hardware, and at 18% load?

I have an i7-8700 with 22 cams running at 20100 kB/s and 560 MP/s at only 16%.

 

[JNDATHP]

Make sure you don't have system restore turned on for those drives and set the recycle bin to zero.

Not sure how to do this; perhaps you could provide the answers?

 

[Jeremy Boelter]

How many cams and what are your kB/s and MP/s rates? For all of that hardware, and at 18% load?

I have an i7-8700 with 22 cams running at 20100 kB/s and 560 MP/s at only 16%.

Looks like I'm closer to 15% when the BI console isn't open -- that's total, btw. BI is only using about 8%.
22 cams, 15600 kB/s and 520 MP/s

 

[fenderman]

I've had similar problems to what you showed there, thankfully not every month, but still too often. October 2nd I discovered most of my cameras hadn't recorded anything for the past 4.5 days because Blue Iris forgot about most of the clips and let the disk get full. I went through much the same process trying to fix it. Repair/Regenerate took SEVEN HOURS and didn't fix anything. I had to delete the database and regenerate it from scratch. Took another seven hours. Meanwhile no recordings could happen because the rebuild procedure disables recording while it is active.

This is one of the reasons I run a second Blue Iris system on a different machine and do not update it at the same time as the main system.

I like Blue Iris, but I can't make excuses for the clip database. It is very unpleasant (that is the diplomatic way of saying it).

Curious how many clips/images you have stored that it takes that long to rebuilt. I can perform a rebuild in a few minuets.
He does need to completely replace the DB though...

 

[CCTVCam]

I have Database Rebuild scheduled for every night. A rebuild take literally seconds (4TB) with this regime and so poses little threat to footage capture and I have no issues. Maybe a more regular routine revuild is needed?

 

[Jeremy Boelter]

Not sure how to do this; perhaps you could provide the answers?

If you right click on the recycle bin > properties, you'll get options that allow you to adjust sizing or just disable all together:


System restore can be turned off by searching the start menu for "system protection" & click on "create a restore point". Click "configure" and you can configure / disable from that dialog:

 

[Jeremy Boelter]

I have Database Rebuild scheduled for every night. A rebuild take literally seconds (4TB) with this regime and so poses little threat to footage capture and I have no issues. Maybe a more regular routine revuild is needed?

good call -- I forgot to mention that in my original post. I rebuild nightly as well.

 

[CCTVCam]

Maybe a build issue? Might be worth posting a build number. Mike Lud is usually around somewhere so might know something.

 

[Jeremy Boelter]

Maybe a build issue? Might be worth posting a build number. Mike Lud is usually around somewhere so might know something.

5.8.0.1 currently, i see .0.4 is available, might try that too.

 

[bp2008]

Curious how many clips/images you have stored that it takes that long to rebuilt. I can perform a rebuild in a few minuets.
He does need to completely replace the DB though...

Actually the slowness was probably due to an HDD malfunction. I remember this normally taking only 20 minutes or so.

 

[CCTVCam]

5.8.0.1 currently, i see .0.4 is available, might try that too.

I will say beware changing builds if running CPAI. I've encountered all kinds of problems doing it and have had to wipe the whole system and reinstall windows every time. I'd only change if others are running the build you intend to move to sucecssfully, or use a system back up utility to backup your entire system including windows 1st so you can go back without data loss.

 

[looney2ns]

5.8.0.1 currently, i see .0.4 is available, might try that too.

If you upgrade to the bleeding edge, you are in essence a Beta tester.
It's best if you stay with updates deemed stable.

 

[samplenhold]

View attachment 174601

Actually the slowness was probably due to an HDD malfunction. I remember this normally taking only 20 minutes or so.

View attachment 174602

So I am curious. You have more records and files than me, but your index time is way less than mine. Anyone with any ideas?

 

[bp2008]

So I am curious. You have more records and files than me, but your index time is way less than mine. Anyone with any ideas?

View attachment 174621

My database is on a SATA SSD. What is yours on? But I also remember index being slower, more like yours, in the past. I'm on 5.8.0.4. Maybe recent updates sped it up.