Upgraded 5.5.8.2 to 5.7.0.4 - Issues with Compact/Repair DB and Clip List

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
As the title says, I upgraded from 5.5.8.2 to 5.7.0.4. I noticed issues with compact/repair DB and repair/regenerate DB failing. I also noticed a very high amount of time where it would take to view the list of recent clips via both UI3 and the mobile app. I downgraded back to 5.5.8.2 for now and everything is working perfectly again. Of course, I don't want to stay at this version as I did just pay the $35 maintenance fee as I was hoping to try out ALPR and Direct to Wire.

Is there an upgrade path that would be better to take in getting to version 5.7.0.4?
 

biggen

Known around here
Joined
May 6, 2018
Messages
2,539
Reaction score
2,765
Is 5.7.0.4 listed as stable? If not, don’t use it and wait til 5.7.x stable.
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
I tested with the latest stable, 5.6.9.8, as well. Same issues.
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
Here are my findings thus far:

5.5.8.2 - original working
5.5.9.6 - working
5.6.0.11 - broken - DB corruption detected
5.6.1.3 - broken - DB corruption detected
5.6.2.10 - broken - DB corruption detected
5.6.3.7 - broken - DB corruption detected
5.6.4.2 - broken - DB corruption detected
5.6.5.9 - broken - DB corruption detected
5.6.6.1 - broken - DB corruption detected

Error starts popping up in the logs, "DB corruption detected" and my clips are partially missing from the clip list.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,448
Reaction score
47,574
Location
USA
You lose the alert thumbnails, but not the actual files themselves.

Do a search here, once the move was made to 5.6.x.x. it pretty much has required a delete of the DB and let it rebuild. Losing the alert thumbnails is a sacrifice of updating to 5.6 and beyond.

You can bounce all around updates and anything above 5.6 will result in what you are experiencing, either go below 5.6 or recognize to run beyond 5.6 you need to delete the DB and let it rebuild.

It is a one time pain and others that have done that have then not had problems thru the latest updates.
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
You lose the alert thumbnails, but not the actual files themselves.

Do a search here, once the move was made to 5.6.x.x. it pretty much has required a delete of the DB and let it rebuild. Losing the alert thumbnails is a sacrifice of updating to 5.6 and beyond.

You can bounce all around updates and anything above 5.6 will result in what you are experiencing, either go below 5.6 or recognize to run beyond 5.6 you need to delete the DB and let it rebuild.

It is a one time pain and others that have done that have then not had problems thru the latest updates.
That surely explains a lot! Thank you for that info. I can effectively do a 'Database >> Delete & Regenerate' ?
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,448
Reaction score
47,574
Location
USA
Yeah, you can do that, but the best solution is to do what is in post 4 - shut down the service, delete the DB folder and then open.

If you do it under the option while BI is running, it can cause the problem to carry over.
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
Yeah, you can do that, but the best solution is to do what is in post 4 - shut down the service, delete the DB folder and then open.

If you do it under the option while BI is running, it can cause the problem to carry over.
Thank you. I'll go that route then.
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
I got everything up and running again. The rebuild took ~48 hours due to the amount saved. I have my cameras set, "Add to alerts list: Hi-res JPEG files", so those images were able to be used when rebuilding the database so nothing was lost. I really wish there was a way to rebuild while still recording, so all that downtime doesn't exist.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
I got everything up and running again. The rebuild took ~48 hours due to the amount saved. I have my cameras set, "Add to alerts list: Hi-res JPEG files", so those images were able to be used when rebuilding the database so nothing was lost. I really wish there was a way to rebuild while still recording, so all that downtime doesn't exist.
The rebuild should never take 48 hours... You would have to have a building full of hard drives for that to happen .. something is wrong with your storage... Is your database on your SSD?
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
24,448
Reaction score
47,574
Location
USA
Are you saying you have alert image thumbnails in the BI console on the left from before you deleted the database?

Are your drives on USB connections? 48 hours is crazy long.

1677296370581.png
 

DLONG2

Known around here
Joined
May 17, 2017
Messages
763
Reaction score
454
Seems like the 4th time I've had to delete the db folder in the past few months . . .
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
The database is on an SSD. I use StableBit CloudDrive to extend my storage so I can keep a big archive of clips (~19k BVR files) which is what slowed down the rebuilding process. Rebuilding the clips portion of the database took ~24 hours by itself. The other big chunk of time was BlueIris looking through the alerts folder.

Ideally the database rebuilding process should be revamped. I only needed a small portion rebuilt for immediate use instead of the whole thing. Ideally it should rebuild the database in the background so recording never stops. After it's done it should merge the databases together (currently recording and rebuilt) which would minimize downtime.

I had an issue a couple of days ago during the daily compact/repair where the process wasn't working. I woke up to find that my system did not record for HOURS. I ended up just disabling the compact/repair process.

I have been using Veeam for daily backups, which has saved me quite a few times, where I could just restore the database files in 15 minutes.
 

CrazyAsYou

Getting comfortable
Joined
Mar 28, 2018
Messages
246
Reaction score
262
Location
England, Near Sheffield
The database is on an SSD. I use StableBit CloudDrive to extend my storage
OK so here lies the problem, the cloud drive will be orders of magnitude slower than even an old IDE hard disk, so is not going to be ideal at all. I'd recommend using only your local storage for BI and use your cloud storage only as a long term archive. Personally, I'd have a simple robocopy script set to run a few times a day to move clips and alerts images from the BI live local folders to the clound storage mount point that are older than a set number of days, maybe 14-30days held on local SATA drives and everything else moved to cloud, you would see fewer issues and DB rebuilds taking minutes not hours. I've got 30days of alerts and 24/7 clips from 8 cameras (about 6.5TB on SATA 5400rpm) and my DB rebuilds are about 40 seconds.
 

djmadfx

Getting the hang of it
Joined
Sep 29, 2014
Messages
106
Reaction score
19
OK so here lies the problem, the cloud drive will be orders of magnitude slower than even an old IDE hard disk, so is not going to be ideal at all. I'd recommend using only your local storage for BI and use your cloud storage only as a long term archive. Personally, I'd have a simple robocopy script set to run a few times a day to move clips and alerts images from the BI live local folders to the clound storage mount point that are older than a set number of days, maybe 14-30days held on local SATA drives and everything else moved to cloud, you would see fewer issues and DB rebuilds taking minutes not hours. I've got 30days of alerts and 24/7 clips from 8 cameras (about 6.5TB on SATA 5400rpm) and my DB rebuilds are about 40 seconds.
Yeah, I know that was the issue. It's nice to keep everything within the BlueIris interface. I never imagined having to do a db rebuild. It was pretty much a one-time thing for me. Yeah, I can move the footage, but I won't have the indexing of the clips available. It's not a huge problem.
 
Top