BlueIris decides to delete all of my videos (5.8.0.7, PSA)?

[EyeOfSauron]

I have been running Blue Iris for a few years now. No config changes to storage policies for a while.

On 10/21, BI decided to delete all of the stored recordings, except for the protected ones. It's been deleting all videos repeatedly since then, until I restarted/updated it to 5.8.0.9.

You can see in the log below, where it went from Delete: nothing to do [1.56T/1.56T, 206.8G free] 1 locked at 8:56PM to Delete: 673 items 1.03T [2.59T/1.56T, 205.9G free] 1 locked at 9:02PM.

I was running version 5.8.0.7. So a PSA, if you're on that version, you might want to update or revert.

(I am aware of the "highly stable option", I was running a newer version to try to see if it helped with some issues I was seeing with audio glitches when streaming to the iOS app).

Anyone ever seen anything like that? My first thought was some sort of a drive failure/corruption, but it all checks out fine and no SMART errors or anything.

0     10/21/2023 8:56:09.943 PM    New                     Delete: nothing to do [1.56T/1.56T, 206.8G free] 1 locked
0     10/21/2023 8:56:09.943 PM    Stored                  Delete: nothing to do [0B/0B, 206.8G free]
0     10/21/2023 8:56:09.944 PM    Alerts                  Delete: nothing to do [0.99G/1.00G, 206.8G free]
0     10/21/2023 8:56:09.944 PM    Aux 1                   Delete: nothing to do [0B/1.00G, 206.8G free]
1     10/21/2023 8:56:46.235 PM    PrusaCam                Signal: network retry
4     10/21/2023 8:56:46.235 PM    PrusaCam                Signal: restored
1     10/21/2023 8:58:38.346 PM    PrusaCam                Signal: network retry
4     10/21/2023 8:58:38.347 PM    PrusaCam                Signal: restored
0     10/21/2023 9:02:29.684 PM    New                     Delete: 673 items 1.03T [2.59T/1.56T, 205.9G free] 1 locked
0     10/21/2023 9:02:29.684 PM    Stored                  Delete: nothing to do [0B/0B, 1.23T free]
0     10/21/2023 9:02:29.684 PM    Alerts                  Delete: nothing to do [0.99G/1.00G, 1.23T free]
0     10/21/2023 9:02:29.685 PM    Aux 1                   Delete: nothing to do [0B/1.00G, 1.23T free]
1     10/21/2023 9:03:04.829 PM    PrusaCam                Signal: network retry
4     10/21/2023 9:03:04.829 PM    PrusaCam                Signal: restored
1     10/21/2023 9:04:21.813 PM    PrusaCam                Signal: network retry
1     10/21/2023 9:04:23.171 PM    Drive-6mm-Left          Signal: network retry
1     10/21/2023 9:04:23.181 PM    DriveLeft               Signal: network retry
1     10/21/2023 9:04:23.257 PM    Garage                  Signal: network retry
1     10/21/2023 9:04:23.310 PM    FrontDoor               Signal: network retry
4     10/21/2023 9:04:25.070 PM    PrusaCam                Signal: restored
4     10/21/2023 9:04:28.668 PM    Garage                  Signal: restored
4     10/21/2023 9:04:28.817 PM    Drive-6mm-Left          Signal: restored
4     10/21/2023 9:04:28.937 PM    DriveLeft               Signal: restored
4     10/21/2023 9:04:29.599 PM    FrontDoor               Signal: restored
1     10/21/2023 9:05:02.333 PM    PrusaCam                Signal: network retry
4     10/21/2023 9:05:02.333 PM    PrusaCam                Signal: restored
1     10/21/2023 9:05:32.352 PM    PrusaCam                Signal: network retry
4     10/21/2023 9:05:32.353 PM    PrusaCam                Signal: restored
1     10/21/2023 9:08:05.066 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.075 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.177 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.178 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\Drive-6mm-Left.20231021_201505.bvr
1     10/21/2023 9:08:05.235 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.236 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.236 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.237 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.237 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.237 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.238 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr
1     10/21/2023 9:08:05.238 PM    Clips                   DeleteFile Error 32: V:\BlueIris\New\DriveLeft.20231021_191319.bvr

I will email support later today.

 

[bp2008]

I will email support later today.

That is the right call.

It appears the periodic disk maintenance routine suddenly decided you had 1.03 TB of extra video exceeding your allocation, and consequentially it decided to delete the oldest 1.03 TB of clips. Definitely sounds like a bug to me, and a nasty one at that.

With Blue Iris not being open source, all I could do is make a wild guess about the cause. My suspicion is that Blue Iris got stuck in a loop adding the same clip(s) to the database over and over again, and in a matter of minutes it had convinced itself that you had an extra terabyte of clips. Since Blue Iris uses a primitive home-grown database format, there don't exist any good tools for inspecting the content to see what is in there.

I've seen Blue Iris's database malfunction in crazy ways often enough that I have very little confidence in it doing the right thing anymore, and that is one of the reasons I run two independent BI systems at my own home.

 

[EyeOfSauron]

Since Blue Iris uses a primitive home-grown database format, there don't exist any good tools for inspecting the content to see what is in there.

I only play a programmer on TV, but I would think rolling your own DB engine would be as good of an idea as a DIY encryption library, or even worse, a library for handling dates .

disk maintenance routine suddenly decided you had 1.03 TB of extra video exceeding your allocation, and consequentially it decided to delete the oldest 1.03 TB of clips

And it kept doing it over and over again, this is the craziest part. I noticed because logging into UI3, all of my clips were 1 second long, and there was only 1 clip per camera.
It was quite a surprise, to say the least!

 

[bp2008]

I only play a programmer on TV, but I would think rolling your own DB engine would be as good of an idea as a DIY encryption library, or even worse, a library for handling dates .

Yup. I'm sure that at the time, 3rd-party DB engines would have seemed unnecessarily complex. As far as I've ever been told, Blue Iris's "database" files are each just a flat list of objects dumped out of RAM. No checksums or journaling to ensure data integrity. No mechanisms for changing field sizes or inserting objects out of order (that is why repair and regenerate operations prevent concurrent recording). And no baked-in thread safety mechanisms. It is a recipe for disaster as a project grows.

And it kept doing it over and over again, this is the craziest part. I noticed because logging into UI3, all of my clips were 1 second long, and there was only 1 clip per camera.
It was quite a surprise, to say the least!

This seems consistent with my guess that some kind of loop is adding the same new clips over and over again. I don't know what else would cause it to think it had terabytes of new clips, and then proceed to spew errors about deleting the same file over and over.

I'd suggest that you close Blue Iris (stop the service), delete the db folder, and open Blue Iris. BI should immediately begin generating a new database from whatever clips you still have on disk, and hopefully the glitch and any lasting effects will be gone. You can also access DB maintenance actions by right clicking inside the clip list in the local console, but in this case I would advise letting Blue Iris start up fresh without a corrupt database in order to avoid any additional weirdness.

 

[EyeOfSauron]

I'd suggest that you close Blue Iris (stop the service), delete the db folder, and open Blue Iris. BI should immediately begin generating a new database from whatever clips you still have on disk, and hopefully the glitch and any lasting effects will be gone. You can also access DB maintenance actions by right clicking inside the clip list in the local console, but in this case I would advise letting Blue Iris start up fresh without a corrupt database in order to avoid any additional weirdness.

Thanks, I did what you've suggested. It was actually "fixed" after updating/restarting BI, but I agree it's probably better to start from zero.

 

[jaydeel]

This happened to me too. More details here.

Lost recordings; BI issue or disk issue?

Have been running BI for a few years now, and it's been very stable, thanks to the help I've gotten from reading the forum... until now. Running on i5-8500@3GHz, 32GB RAM, 256G SSD for OS, 7TB WD for recording. A few days ago, I get a warning that my drives are over-allocated, and the...

ipcamtalk.com

 

[jaydeel]

No mechanisms for changing field sizes or inserting objects out of order

I recall that update 5.6.0 (8/16/2022) had this entry in the Help PDF...


I remember it because the 'ad-hoc alert after the fact' was something I'd requested way back in 2018.

 

[EyeOfSauron]

I don’t have any more technical details on this potential bug, but this is what I got from support:

Apologies for recent issues. To avoid issues consider using the stable update track from the settings page.

While it doesn’t explicitly confirm this is a known issue, it doesn’t deny it either . And the suggestion to use the “stable” channel is of course valid (I only used latest to see if it helped with another issue, iOS 17 having high pitched noise when audio stream starts).

 

[IAmATeaf]

I don’t have any more technical details on this potential bug, but this is what I got from support:



While it doesn’t explicitly confirm this is a known issue, it doesn’t deny it either . And the suggestion to use the “stable” channel is of course valid (I only used latest to see if it helped with another issue, iOS 17 having high pitched noise when audio stream starts).

If everybody did that then the bugs wouldn’t get discovered until the next so called stable release which then wouldn’t really be that stable?

 

[bp2008]

I recall that update 5.6.0 (8/16/2022) had this entry in the Help PDF...

View attachment 175945
I remember it because the 'ad-hoc alert after the fact' was something I'd requested way back in 2018.

Ah, well as long as recording is disabled during any DB maintenance operation, I consider it inadequate at best.

 

[jpc-s4]

This is the same thing I reported in troubleshooting, but alas, I had no logs...

 

[smiticans]

With Blue Iris not being open source, all I could do is make a wild guess about the cause. My suspicion is that Blue Iris got stuck in a loop adding the same clip(s) to the database over and over again, and in a matter of minutes it had convinced itself that you had an extra terabyte of clips. Since Blue Iris uses a primitive home-grown database format, there don't exist any good tools for inspecting the content to see what is in there.

I've seen Blue Iris's database malfunction in crazy ways often enough that I have very little confidence in it doing the right thing anymore, and that is one of the reasons I run two independent BI systems at my own home.

BI's fragile database is my only real complaint about the software. I can only hope BI 6 (assuming that's what it'll be called) would incorporate a new modern database that doesn't need to pause recordings to do maintenance. And a client application would be nice too.

I've considered switching from BI to Digital Watchdog just due to how fragile the database is. I really don't trust it. The only thing stopping me from switching is figuring out how to get pushover notifications to work with it, and to make sure DW will be able to accept AI triggers from all of my cameras. So far it appears the 5442 series cameras are working, I just need to email them to request support for the color 4k series. The cost of DW will also be a lot considering I have 12 cameras, but at least i'll own the software forever.

 

[samplenhold]

The cost of DW will also be a lot considering I have 12 cameras, but at least i'll own the software forever.

Not sure your point on the own the software forever comment. Your BI license is forever.

 

[wittaj]

And has the BI database ever failed you in an after the fact incident and the video was missing?

 

[smiticans]

Not sure your point on the own the software forever comment. Your BI license is forever.

Sorry, I should've explained that better. I'm aware its forever. I've been using BI since 2015.

What I mean by the comment is that VMS solutions that are in the same or similar class as Digital Watchdog require yearly renewal and/or upgrade fees. When you purchase Digital Watchdog, you own it forever, including version upgrades. So, if you purchase it while the current version is version 5, when version 6 is released you can upgrade free of charge. That comment wasn't meant to compare Blue Iris to Digital Watchdog.

 

[smiticans]

And has the BI database ever failed you in an after the fact incident and the video was missing?

@wittaj I respect you and all I appreciate all of the content you've created and posted as I've used it to fine tune my cameras. With that being said, I feel that comment is pretty dumb and you're blindly supporting BI.

If your smoke detectors were known to malfunction, is that okay as long as it didn't happen during a house fire?
If your seatbelts in your car occasionally malfunction, is it okay as long as it didn't happen during a motor vehicle crash?
If burglar alarm system sometimes didn't detect a door opening, is it okay as long as someone wasn't breaking in?

Obviously the answer to the above questions are no.

No, BI's database has never failed when I needed to review footage of an incident. I'm pretty good about staying on top of the database maintenance. However, I have had multiple occasions where the database would fail overnight and I've woken up to emails indicating so. Thankfully nothing happened. I've also had to remote VPN into remote locations to fix database errors. Nothing happened during that time period where I needed to review footage but that doesn't change the fact that the database is fragile and needs improvement.

I love BI and I think its a great software especially at its price point. Just because I support something doesn't mean I can't provide constructive criticism. The database is BI's weakest link and really needs improvement. Clearly I'm not the only one on this forum that thinks that.

 

[wittaj]

@wittaj I respect you and all I appreciate all of the content you've created and posted as I've used it to fine tune my cameras. With that being said, I feel that comment is pretty dumb and you're blindly supporting BI.

If your smoke detectors were known to malfunction, is that okay as long as it didn't happen during a house fire?
If your seatbelts in your car occasionally malfunction, is it okay as long as it didn't happen during a motor vehicle crash?
If burglar alarm system sometimes didn't detect a door opening, is it okay as long as someone wasn't breaking in?

Obviously the answer to the above questions are no.

No, BI's database has never failed when I needed to review footage of an incident. I'm pretty good about staying on top of the database maintenance. However, I have had multiple occasions where the database would fail overnight and I've woken up to emails indicating so. Thankfully nothing happened. I've also had to remote VPN into remote locations to fix database errors. Nothing happened during that time period where I needed to review footage but that doesn't change the fact that the database is fragile and needs improvement.

I love BI and I think its a great software especially at its price point. Just because I support something doesn't mean I can't provide constructive criticism. The database is BI's weakest link and really needs improvement. Clearly I'm not the only one on this forum that thinks that.

Calm down there sparky LOL.

I am not blindly defending BI and have expressed my concerns about the database before and other BI issues on the forum.

I was simply asking did a corrupt database cause you to not have video files. I get that yeah you probably lose the thumbnail alerts and ability to quickly find an alert, but are the video files still there?

In other words, did BI quit recording when the database got corrupt and you had no video that you could playback, even if it was simply a long video file?

I have never had missing video files when my DB corrupted. And yes it is a pain the @$$ to lose the alert thumbnails when deleting and regenerating the database, but I still have the video.

 

[smiticans]

Calm down there sparky LOL.

I am not blindly defending BI and have expressed my concerns about the database before and other BI issues on the forum.

I was simply asking did a corrupt database cause you to not have video files. I get that yeah you probably lose the thumbnail alerts and ability to quickly find an alert, but are the video files still there?

In other words, did BI quit recording and you had no video that you could playback?

I have never had missing video files.

Sorry, I misunderstood your post then lol.

There where a couple of times where the database errors did not stop the recordings and were just annoying. But I'd say the majority of the times I've had database errors all the recordings were halted until I fixed the issue and I had no footage between then.

 

[samplenhold]

Personally, I have been lucky with Bi and any errors. I have only had it die on me maybe 3-4 times since 2018. But after setting it to only go to stable versions and only manual updates, I have not had the program die.

Now for the database, recently have had it stop recording overnight. I had a database error and drives showing overallocated in the status, but they were not overallocated. I reverted back to the previous stable version and not had a problem. That was the reason for me to go to manual updates. I was out on the road at the time this happened. I always check the cams in the morning and at night to make sure things are fine. One morning they were not. So I logged in remotely and reset the BI app and set it to manual updates since I noticed that it happened after an update to BI that night.

That was the only time I had an issue with the database. So I am not worried that it will fail me when needed.

 

[DigitalPackrat]

The clip deletion has hit a few of us. In my case it started on 5.8.0.4 along side the memory leak, kept having issues through 5.8.0.8 and I didn't catch it till I was looking for specific footage. Reverting to last stable 5.7.9.12 and 5.8.0.2 fixed the issue till Ken replied to my email and released 5.8.0.9 which is behaving so far.

The issue was just like you are describing, system thought more space was being used than it really was, drive went over quota and a month of footage was gone. Rebuilding or repairing the DB would fix it for a short time then it would go nuts again.