Assuring that your 'Protected' Alerts survive a Database/Regenerate action

[jaydeel]

I've created a PowerShell script that enables inspection of all the 'protected' Alerts in your Blue Iris database.
If the script detects issues with a protected Alert, it will also help you fix them so that the Alert will survive a Database Repair/Regenerate or Delete/Regenerate action.

This utility may be for you if you've ever experienced the following:

• The loss of Alerts that you are certain you previously 'protected', and/or
• The loss of Alert memos that were automatically created by AI, or directly edited by you.

Reasons for a protected Alert not surviving Database/Regenerate action are:

1. The 'flag' protecting an Alert exists solely in the Blue Iris database and can be lost if the database is damaged or corrupted, and therefore must be regenerated.
2. The Alert has has no JPG file (or DAT file).
3. The Alert JPG file does not contain up-to-date Exif metadata .

Per Blue Iris support, the following requirements are necessary for 'full recovery' of a protected Alert during a Database/Regenerate action:

1. The Alert must have either a JPG file or a DAT file;
2. At least one of these files (JPG or DAT) must have its 'read-only' file attribute set;
3. The JPG file's Exif metadata must have up-to-date values, including a 'Title' property that matches the original database memo text.

Note that 'full recovery' (setting the 'protected' flag AND restoring the alert memo) ALWAYS requires a JPG file.

In addition to the core functions of finding and fixing protected Alerts that have issues, this script also has these features:

• It saves timestamped results from each execution (to CSV files in the script's '\_output' subfolder), and supports auto-pruning of the old files from this folder.
• It logs all protected Alert fixes (to a CSV file name 'fixed_alerts_log.csv' in the script's root folder).
• It always performs a Blue Iris database Repair/Regenerate action if a missing JPG or DAT is fixed, or a JPG's Exif metadata is updated.
• It always backs up the Blue Iris database folder before performing a database Repair/Regenerate action (to the script's '\_backup_db_folders' subfolder)
• It always backs up original JPG file before changing the Exif metadata (to the script's '\_backup_jpgs' subfolder)
• It can log results to the Blue Iris logfile (optional)
• It can 'Flag' all protected Alerts in the Blue Iris database (Alert's with a 'Flag' state lose this designation with a database Delete/Regenerate action)

HOW TO GUIDE...

STEP #1:

Install the script and its companion 'user settings file'

1. Extract all the files from the attached zip file to your desktop, or scripts folder.
2. Please note that the folder used must be in your PC's environment path.
3. Create a working folder for this script - it will be used to process the attachments files; to hold the script's log file, etc.
4. To use the folder, you must edit a user setting $app_path in the user-settings script file (see STEP #2)

NOTE: If you've never used PowerShell on your Blue Iris PC, you may first to need to change its default security policy.
Open PowerShell in Administrator mode, and type the following, then hit the Enter key: Set-ExecutionPolicy RemoteSigned
(click here for a nice beginner's tutorial to using PowerShell).

STEP #2:

Edit the user settings file

1. The user settings file is named 'BI_alert_protect_user_settings.ps1'
2. Open the file in a text editor (a code text editor is recommended; I use Notepad++)
3. Edit the REQUIRED SETTINGS section (see screenshot)
4. Ignore the remaining settings for now. They provide customizations that you may want to explore later.

STEP #3:

Use the script from the PowerShell console

Open an Admin PowerShell window, or Powershell-ISE window

3.1 - Select and copy the following command and paste it into the console.
BI_alert_protect.ps1 -NoInspect
Hit the Enter key.
(Be patient, it may take awhile for the script to respond if you have a lot of Alerts.)
The response should look like following two screenshots

This screenshot illustrates the first screen of output to the PowerShell Console.
Key features are highlighted.
The 'Flags' column shows if the Alert's database 'Flagged' and 'Protected' flags are set.
The 'RO' column shows if the read-only file attributes are set for the JPG and DAT files (if they exist)

This screenshot shows the result after hitting the keyboard spacebar until all results are displayed.
Additionally, it shows the timestamped output CSV file (displayed over the Console output).
Note that this file contains the details for those issues that have a 'Yes' column entry in the Console output.

When executed, the script creates the following folders.
It also creates the 'fixed_alerts_log.csv' if any Alerts have been 'fixed'.

The timestamped output CSV file is located here:

3.2 - Select and copy the following command and paste it into the console.
BI_alert_protect.ps1
Hit the Enter key.

The response should look like following screenshot.
Note that the 'JpgExifMemo' column is now populated. Furthermore, the 'JpgExifMemo' column entry is 'match!' when the JPG Exif matches the 'AlertMemo' column entry. Otherwise the entry is actual JPG Exif value ('Title' tag).

For commands to 'fix' your protected Alerts that have issue, see post #2.

Finally, here's a screenshot of the script's help screen.

Script Zip File

EDIT (2024-03-23 3:40p): updated to v3
PLEASE NOTE: if you are updating, you must edit & use the new user-settings-file - tip: to facilitate editing, rename the existing file before unzipping the update.

 

[jaydeel]

BI Alert Protect Utility - Some command examples by scenario:

GETTING HELP

1. I want to see only the help screen
bi_alert_protect.ps1 -Help


INSPECTING PROTECTED ALERTS

2. I want to see FAST results without waiting for the script to inspect the Exif metadata in the all the JPG files
bi_alert_protect.ps1 -NoInspect

3. I want to see results for all my protected Alerts
bi_alert_protect.ps1

4. I want to see results for only those protected Alerts THAT HAVE ISSUES
bi_alert_protect.ps1 -FilterIssues

5. I want to see results for only those protected Alerts that have a specific issue
bi_alert_protect.ps1 -FilterNoJPG ... that lack a JPG file
bi_alert_protect.ps1 -FilterNoDAT ... that lack a DAT file
bi_alert_protect.ps1 -FilterNoBOTH ... hat lack both a JPG & DAT file
bi_alert_protect.ps1 -FilterNoRO ... w/ JPG or DAT files do not have their read-only file attribute set
bi_alert_protect.ps1 -FilterNoMATCH ... w/ database memos that do not match the JPG Exif metadata

6. I want to see results for only a single camera, or specific camera
bi_alert_protect.ps1 -Cams "DW1"
bi_alert_protect.ps1 -Cams "DW1,DW2"

7. I want to see results for only a single Alert, where the column 'AlertID' entry is like 'GT.20231113_010000.1442772.5-0'
bi_alert_protect.ps1 -Pattern "1442772"

8. I want to see results for only Alerts that have a specific substring
bi_alert_protect.ps1 -MemoTag "!"


FIXING PROTECTED ALERTS WITH ISSUES

9. I want to fix all of my protected Alerts THAT HAVE ISSUES
Note: the following commands yield identical results, The only difference is in the the contents of the output*.csv file.
bi_alert_protect.ps1 -FixAll ... output*.csv file lists ALL the protected alerts inspected
bi_alert_protect.ps1 -FilterIssues -FixAll ... output*.csv file lists only the protected alerts that have issues

10. I want to fix specific issues for a subset of my protected Alerts
bi_alert_protect.ps1 -FilterNoJPG -FixFiles
bi_alert_protect.ps1 -FilterNoDAT -FixFiles
bi_alert_protect.ps1 -FilterNoBOTH -FixFiles
bi_alert_protect.ps1 -FilterNoRO -FixReadOnly
bi_alert_protect.ps1 -FilterNoMATCH -FixJpgExif

11. I want to fix all issues for only a single camera, or multiple cameras
bi_alert_protect.ps1 -Cams "DW1" -FixAll
bi_alert_protect.ps1 -Cams "DW1,DW2" -FixAll

12. I want to fix all issues for only a single Alert, where the AlertID is like 'GT.20231113_010000.1442772.5-0'
bi_alert_protect.ps1 -Pattern "1442772" -FixAll

13. I want to fix all issues for only Alerts that have a specific substring
bi_alert_protect.ps1 -MemoTag "!" -FixAll

14. I want to 'Flag' all of my protected Alerts in the Blue Iris database
Handy if you manually 'Flag' your protected Alerts (this designation is lost with a database Delete/Regenerate action)
bi_alert_protect.ps1 -SetFlags -NoInspect

 

[jaydeel]

BI Alert Protect Utility - More Details

If the script repairs a protected Alert that is lacking a JPG file...

The following actions are performed by the script:

• A dummy JPG file is created in the Blue Iris 'Alerts' folder.
• The file is given the expected filename.
• The file is give the expected timestamp.
• The file is given the expected Exif metadata.
• A Blue Iris database Repair/Regenerate action is executed (to import the JPG's info into the database).

Please note that after creating the missing JPG file, the Blue Iris clips list will continue to display the small alert image stored in the database.
It is only AFTER a database Delete/Regenerate action has occurred that dummy JPG will appear in the clips list.

Example:

Here's what we expect to see when inspecting the Example Alert's JPG file.
Note that the file has the appropriate timestamp and Exif metadata.

BTW, Ken provided the following when I asked for an explanation of the Exif 'Comments' field...
PropertyTagExifUserComment
1.1366323.6149-65.(plate.)23195.28236.jpg
1 ... the version of this code ... should be 1
1366323 ... the time offset into the clip
6149 ... the "high order" of flags (bits 16-31 shifted left 16)
65 ... the zones
(plate) ... (LPR plate would go here optionally)
23195 ... the trigger duration (msec)
28236 ... this time includes the pre-trigger time for playback

FINAL NOTE:
If you would really prefer to have an Alert with a normal JPG image, you might consider trying the Blue Iris Search in Clip feature.
It can be used while playing back a clip to add a new Alert directly to the database. I have use this several times to add after-the-fact missing alerts

 

[jaydeel]

Script updated to V2. see post #1 to download the file

This new version may helpful if you want to automate the script via Task Scheduler (I'm running it daily).
It provides:

• more informational messaging via the Blue Iris logfile (#6).
• a user setting to confirm background execution (#5).
• a user setting to temporarily disable database maintenance actions (#4).

Changelog:

1. BUG: fixed script incorrectly detecting if running in background (via Task Scheduler, or Blue Iris)
2. BUG: fixed duplicate entries in the 'Issue(s)' column in the script's log file ('fixed_alerts_log.csv')
3. NEW: added progress bar for alertlist parsing (provides quicker console feedback when the script starts)
4. NEW: added user setting $disable_clip_maintenance to avoid possible simultaneous Database/Repair actions
5. NEW: added user setting $enable_bg_messages for temporary use to verify background execution
6. IMPROVEMENT: made posted messages to the Blue Iris logfile more informative; now includes script's runtime, and counts of issues found & fixed
7. IMPROVEMENT: argument -DevMode now outputs more timing information; demos the dummy JPG; saves each camera's alertlist

The following screenshot shows examples of improved entries in the Blue Iris logfile:

The following screenshot shows the new USER SETTINGS.

 

[EvanVanVan]

Hi, this tool has been great.

As an update to this script, rather than creating dummy yellow jpgs, couldn't you extract the AI confirmed jpg like you do in your other script? The resolution wouldn't be good but probably still pretty clear as a small thumbnail.

 

[jaydeel]

Hi, this tool has been great.

As an update to this script, rather than creating dummy yellow jpgs, couldn't you extract the AI confirmed jpg like you do in your other script? The resolution wouldn't be good but probably still pretty clear as a small thumbnail.

Yes, this is possible-- if the Alert has a DAT file. Otherwise I could grab a frame from the clip.

For the DAT files, I'm not sure I can identify the actual confirmed image (asterisked in the viewer). However, for this purpose, any of the JPGs is probably acceptable.

 

[jaydeel]

Script updated to V3. see post #1 to download the file

If you tried an earlier version and did not like the ugly yellow replacement that JPGs it created (when a DAT file could not be found), then you will like this new version.

Key features of this new version are:

1. In recognition that the usage options are formidable, the script now performs argument validations, and feedback if the commandline includes unsupported arguments.
2. Improved dummy/replacement JPG creation - 3 methods are now used, in this order: 1) from the DAT file (if it exists & is valid), then 2) from the Alert's BVR clip file (if it exists), then 3) the Alert's database thumbnail image. If all 3 methods fail, the script then creates a solid yellow dummy/replacement JPG. For DAT files, the AI-confirmed image is used (if it exists), otherwise the middle image is used. For BVR files, the image used is grabbed at the Alert's 'offset'; furthermore, these images may be customized via user settings.
3. When grabbing JPGs from BVR files, the script now supports multiple Blue Iris 'New' folders.
4. Improved the method for setting that CreationTime & LastAccessTime attributes for dummy/replacement JPGs; previously, the script set only the LastWriteTime, which could result in incorrect database timestamps when the protected Alerts were recovered during a Blue Iris Database/Regenerate action. (Note: The 'Date taken' Exif metadata value for dummy/replacement JPGs is automatically set by Blue Iris).
5. Script now identifies the DAT file's confirmed image and extracts its associated AI analysis memo -- thanks @EvanVanVan .
6. The overlay message on replacement JPGs is now optional and user customizable.
7. Dummy/replacement JPGs are backed up before adding the overlay (thus can be recovered, if desired).
8. The script no longer creates dummy DAT files (simple text file).
9. Added new argument -AllAlerts to process Alerts regardless of 'Protected' flag.
10. Added new argument -Flags to analyze Alerts by DB flag, like -Flags 'F','P','!C'.
11. Output column 'Flags' now shows if alerts are 'Confirmed' in the database.
12. Detects & repairs Alerts w/ empty (or 'NA') memos by using the memo string from the confirmed image in the DAT file.
13. To make repairs more obvious in the Blue Iris clips list, the script now prepends a caret (^) character (optional & customizable).
14. Added new argument -ClearMarks to facilitate the batch removal the memo caret (^) marks.

Rare issue - Zero Date Alerts

1. Detects & repairs Alerts w/ zero dates - from either a existing DAT or JPG file's date
2. Added new output column 'HasZD' that shows if alert's DB record has a zero date
3. Unfixable zero-date Alerts may be optionally deleted from the DB

To view the entire changelog, open the script file in a script editor.

Current usage guide
bi_alert_protect.ps1 -Help