BI DAT Tool - Filter/inspect/delete Blue Iris AI-analysis 'DAT' files; Extract JPG images and JSON results

[jaydeel]

While developing my BI_gif_tool.ps1 utility (for sending GIF alerts via Pushover), I learned how to extract the JPG images embedded in Blue Iris Artificial Intelligence DAT files. This project resulted in 3 ideas:

1. A utility that could remove selected DAT files that are no longer useful from Blue Iris storage (Alerts folder).
2. A utility that could be used to inspect DAT files for information like image counts, AI labels found, etc.
3. A utility that could be used to 'mine' DAT files for JPG images that could be useful in training AI models.
   Using images extracted from DAT files is an efficient way to generate a lot of candidate AI model images for your specific fields of view and conditions (day/night, sunny/cloudy, etc.)

This post introduces a Powershell script that does the following things with Blue Iris generated AI-analysis 'DAT' Files:

1. Inspect DAT files showing image counts, ages, alert memos, AL models used, and AI labels found (by image)
   The results are displayed in the console, and also saved to a CSV file.
2. Filter / down select DAT files based on DAT filename(s), naming pattern(s), image count, age, and/or alert memo(s)
3. Perform the following actions on a any set of filtered DAT files:
   • Remove the files from Blue Iris storage to the recycle bin (by default), or permanently delete them
   • Extract all JPG images
   • Extract all JSON strings
4. Down select which JPG images are extracted based on AI analysis RESULTS:
   • the AI models used, and/or
   • the AI labels found
     - OR -
   • if 'Nothing found' is the AI analysis result for any (or all) models used, or
   • if 'Something found' is the AI analysis result for any (or all) models used.

These screenshots show typical output - via the Console, or the automatically created CSV file

Saved JPG and JSON files are stored in a user-selected folder (specified in the user-settings file)


This screenshot shows the available set of commandline parameters. (EDIT: updated for v 1.2)


I will provide easier-to-consume scenario-based command examples in the next post.

The remainder of this post is the installation guide...

STEP #1:

Install the script and its companion 'user settings file'

1. Extract all the files from the attached zip file to your desktop, or scripts folder.
2. Please note that the folder used must be in your PC's environment path.
3. Create a root folder for this script - it will be used to process the GIF files; to archive GIF and DAT files, etc.
4. To use the folder, you must edit a user setting in the appropriate Powershell script file (see Step #3)

NOTE: If you've never used PowerShell on your Blue Iris PC, you may first to need to change its default security policy.
Open PowerShell in Administrator mode, and type the following, then hit the Enter key: Set-ExecutionPolicy RemoteSigned
(click here for a nice beginner's tutorial to using PowerShell).


STEP #2:

Edit the user settings file

1. Open the file in a code text editor (I use Notepad++)
2. Edit the REQUIRED variables in the USER SETTINGS section.
3. Ignore the remaining settings for now. They provide customizations that you may want to explore later.

Script Zip File
EDIT (2024-03-22): Script updated to V2
PLEASE NOTE: if you are updating to v2, you must replace the old user-settings file with the new one to support new features.
(This means you will need to edit the user-settings file again. Tip: before updating, rename the old file.)

 

[jaydeel]

BI DAT Tool - Some command examples by scenario:

1. I want to see only the help screen
bi_dat_tool.ps1 -Help

2. I want to see FAST results without waiting for the script to inspect the DAT files
bi_dat_tool.ps1 -NoInspect


3. I want to see the total number DAT files that are older than 2 weeks
bi_dat_tool.ps1 -AgeHours 336


4. Before I delete DAT files that are older than 158 hours for 2 specific cameras, I want to double-check the JSON first
bi_dat_tool.ps1 -Patterns "DW1.","DW2." -AgeHours 158 -SaveJSON
Use Windows explorer to peak inside the *.json files for each image


5. I want to remove Alerts folder DAT files that are older than 158 hours for 2 specific cameras
bi_dat_tool.ps1 -Patterns "DW1.","DW2." -AgeHours 158 -Delete


6. I want to permanently delete Alerts folder DAT files that are older than 158 hours for 2 specific cameras
bi_dat_tool.ps1 -Patterns "DW1.","DW2." -AgeHours 158 -Delete -NoRecycle

7. Before I delete DAT files with 'vehicle' labels, I want to double-check the JSON first
bi_dat_tool.ps1 -IfAiLabels vehicle -SaveJSON
Use Windows explorer to peak inside the *.son files for each image <screenshots>

8. I want to delete DAT files that are older than 1 week, leaving undeleted DAT files that have ANY 'Nothing found' AI results
bi_dat_tool.ps1 -AgeHours 168 -IfAiSomethingAny -Delete

9. I want to delete DAT files that are older than 1 week, leaving undeleted DAT files that have ONLY 'Nothing found' AI results
bi_dat_tool.ps1 -AgeHours 168 -IfAiSomethingAll -Delete

10. I want to see the total number DAT files that contain at least 5 JPGs
bi_dat_tool.ps1 -MinImages 5

11. I want extract all JPGs and JSON results for a single DAT file
bi_dat_tool.ps1 -Files "DW1.20230715_150000.3251912.17-0.dat" -SaveJPGs -SaveJSON

12. I want extract all JPGs and JSON results for several DAT files
bi_dat_tool.ps1 -Files "DW1.20230715_150000.386323.17-0.dat","DW1.20230715_150000.386323.17-0.dat" -SaveJPGs -SaveJSON

13. I want to extract JPGs to a SINGLE folder for all DAT files that contain 5 to 8 JPGs
bi_dat_tool.ps1 -MinImages 5 -MaxImages 8 -SingleFolder -SaveJPGs

14. I want to extract all JPEG files for all DAT files created by a single camera (DW1) on July 1
bi_dat_tool.ps1 -Patterns "DW1.20230701" -SaveJPGs

15. I want to extract all JPEG files for all DAT files for a single cameras (DW1) on July 1 between noon and 3 pm
bi_dat_tool.ps1 -Patterns "DW1.20230701_1200","DW1.20230701_1300","DW1.20230701_1400" -SaveJPGs

16. I want to extract all JPGs for all DAT files for multiple cameras (DW1,DW2) on July 1 between noon and 1 pm
bi_dat_tool.ps1 -Patterns "DW1.20230701_1200","DW2.20230701_1200" -SaveJPGs

17. I want to filter & save JPGs and with JSON results with 'dog' labels via AI model 'combined'
bi_dat_tool.ps1 -IfAIModel combined -IfAiLabels dog -SaveJPGs -SaveJSON

18. I want to extract JPGs for a single camera's DAT files that have ANY 'Nothing found' AI results
bi_dat_tool.ps1 -Patterns "DW1." -IfAiNothingAny -SaveJPGS
Tip: You can also use argument -SaveJSON to verify the results (save every image's AI results)

19. I want to extract JPGs and JSON results for a single camera's DAT files that have ONLY 'Nothing found' AI results
bi_dat_tool.ps1 -Patterns "DW1." -IfAiNothingAll -SaveJPGS
Tip: You can also use argument -SaveJSON to verify the results (save every image's AI results)

20. I want filter & extract JPGs for DAT files that found a 'dog' or 'cat' using a model named like 'animal'
bi_dat_tool.ps1 -IfAIModel animal -IfAiLabels dog,cat -SaveJPGs
Tip: You can also use argument -SaveJSON to verify the results (save every image's AI results)

 

[jaydeel]

Demo example - using the Powershell script to extract JPGs for use in AI Models

1. On a daily basis, I use the following strategy to flag alerts whose DAT files I may want to 'mine' for images for incorporation to an AI model.

Strategy: I flag & protect the alert, then manually edit the memo and assign it a label followed by an exclamation point (like 'skunk!')​

​

​

​

WHY? The unique exclamation point character makes it easy to locate the DAT files I plan to 'mine' for JPG images.​

​

This results in alert entries like these...​

​

​

2. To mine the DAT JPG files from these ‘memo-flagged’ Alerts, I use the Powershell script as follows:

​

All candidate DAT files​

example command:​

bi_dat_tool.ps1 -Patterns "FE.","FD.","GT" -AlertMemos "!" -SaveJPGs -SingleFolder -Quiet​

​

Note the use of the argument -AlertMemos​

This is how I use the exclamation point character to filter the DAT files I want​

​

Example Output:​

​

​

​

​

3. To refine the JPG image extraction for false positives and false negatives, I use the script as follows

3A. False positive DAT files​

example command:​

bi_dat_tool.ps1 -Patterns "FE.","FD.","GT" -AlertMemos "!" -IfAiSomethingAny -SaveJPGs -SingleFolder -Quiet​

​

Note the addition of the argument -IfAiSomethingAny​

This is how I isolate the false positive images​

​

Example Output:​

​

​

​

3B. False negative DAT files​

example command:​

bi_dat_tool.ps1 -Patterns "FE.","FD.","GT" -AlertMemos "!" -IfAiNothingAll -SaveJPGs -SingleFolder -Quiet​

​

Note the addition of the argument -IfAiNothingAll​

This is how I isolate the false negative images​

​

Example Output:​

​

​

OBSERVATIONS / TAKE-AWAYS:

1. In the Windows Explorer screenshots above (3rd thumbnail), note that the names of JPG files include:
   the DAT filename, the image number, and useful AI label info.
2. Note that the total count of DAT files filtered in Steps #3A and #3B (25 & 7), match the total number of DAT files in #2 (32).
3. Note in Step #2 that the number of JPG images saved is 322 from 32 DAT files.
4. Using images extracted from DAT files is an efficient way to generate a lot of candidate AI model images for your specific fields of view and conditions (day/night, sunny/cloudy, etc.)

 

[jaydeel]

Updated post #3 (previously a placeholder).

 

[EvanVanVan]

Wow, there's a lot of information here. I guess I just have to play around with the tool until I get the hang of it lol.

Question, what is the webserver access used for? I see the "NOTE: ACCOUNT MUST BE ADMIN TO POST TO CUSTOM MESSAGES TO THE BLUE IRIS LOGFILE," but not sure what it really does.

Also, fyi, not changing the default webserver settings caused thousands of the following errors until the script was interrupted. I assume they could probably be handled nicer like the error messages for other settings.

Spoiler: Error Messages

Invoke-RestMethod : Cannot bind parameter 'Uri'. Cannot convert value "http://YOUR_BI_SERVER_ADDRESS:PORT/json" to type "System.Uri". Error: "Invalid URI: Invalid port specified."
At C:\Users\evanv\OneDrive\Desktop\BI_dat_tool.V1.0\BI_dat_tool.ps1:526 char:41
+             $jsonResponseObj = Invoke-RestMethod @params
+                                                  ~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Invoke-RestMethod], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : Cannot bind parameter 'Uri'. Cannot convert value "http://YOUR_BI_SERVER_ADDRESS:PORT/json" to type "System.Uri". Error: "Invalid URI: Invalid port specified."
At C:\Users\evanv\OneDrive\Desktop\BI_dat_tool.V1.0\BI_dat_tool.ps1:526 char:41
+             $jsonResponseObj = Invoke-RestMethod @params
+                                                  ~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Invoke-RestMethod], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : Cannot bind parameter 'Uri'. Cannot convert value "http://YOUR_BI_SERVER_ADDRESS:PORT/json" to type "System.Uri". Error: "Invalid URI: Invalid port specified."
At C:\Users\evanv\OneDrive\Desktop\BI_dat_tool.V1.0\BI_dat_tool.ps1:526 char:41
+             $jsonResponseObj = Invoke-RestMethod @params
+                                                  ~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Invoke-RestMethod], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
...
...
...
...
...

Thanks for this tool, it is going to be very helpful for the candidate AI model images!

Edit: I like the strategy of adding a ! to the memo of alerts to be processed. Jw, could the script automatically remove the ! once it's finished?

My thinking is it would be nice to have the task scheduler run once a day as suggested to process everything. But I don't necessarily want to delete the alerts, so going back and cleaning up memos would ensure they wouldn't be reprocessed the next day.

 

[jaydeel]

what is the webserver access used for?

Currently, the script imports the Blue Iris alerts list via the JSON interface cmd alertlist.
This provides access to the memo strings displayed in the Blue Iris Clips list.
The alertlist cmd does not require an admin account

I see the "NOTE: ACCOUNT MUST BE ADMIN TO POST TO CUSTOM MESSAGES TO THE BLUE IRIS LOGFILE," but not sure what it really does.

Logging to the Blue Iris logfile is optional. And the HTTP interface command it uses /admin?log requires an admin account.
The logging functionality is provided mainly as a means to document automated actions, e.g., when running the script via the Task Scheduler.
BTW, even if you've enabled logging by default (via user setting $post_to_BI_logfile_flg = $true), logging can can be disabled on-demand via the script's -NoBiLog argument.

fyi, not changing the default webserver settings caused thousands of the following errors until the script was interrupted. I assume they could probably be handled nicer like the error messages for other settings.

Good suggestion. An oversight on my part. I'll trap this condition in an update.

I like the strategy of adding a ! to the memo of alerts to be processed. Jw, could the script automatically remove the ! once it's finished?

Good idea. I'll add an argument that does this.

 

[jaydeel]

HEADS-UP...

Until further notice - Please do not use the script's -Delete argument

I've just observed a issue requiring a database Repair/Regenerate after using the -Delete argument to remove DAT files.

I'll contact Ken to make sure I'm using an appropriate method of deleting DAT files.

EDIT (10/8/2023) - Issue resolved as of v1.2. Argument -Delete now forces a Database/Regenerate, as recommended by Blue Iris support.

 

[EvanVanVan]

Any idea why editing the memo in BI to add a "!" isn't working? It shows up in BI properly, but when I scan the DAT files with your tool it doesn't show up. Also, Windows also doesn't the change the "Date Modified" of the DAT file, so I'm guessing the change isn't saving properly? I tried unflagging/unprotecting the alert first without success. Do I need to run maintenance on the Database or something?

There seems to be something (Edit an existing memo?) about once the DAT file is created you can't actually change the memo?

Ty

 

[jaydeel]

I’m sorry you are having difficulty with the script.

Any idea why editing the memo in BI to add a "!" isn't working? It shows up in BI properly, but when I scan the DAT files with your tool it doesn't show up.

The script is directly importing the alert memos from the Blue Iris database via the JSON interface alertlist command.
I would first try to do a database Compact/Repair, and if that doesn’t work, so a Repair/Regenerate.

It may be possible that the script is failing to correctly build a lookup array that associates alert memos to DAT filenames. But I’m testing the script a lot the past week, and haven’t observed such a failure.

Also, Windows also doesn't the change the "Date Modified" of the DAT file, so I'm guessing the change isn't saving properly?

I would not assume that editing the alert memo would result in a timestamp change on the DAT file. The memo is an entry in the database.

There seems to be something (Edit an existing memo?) about once the DAT file is created you can't actually change the memo?

In my experience this is not the case with the current version of Blue Iris. The past few days I’ve been editing the memos of existing alerts via the JSON interface update command (via a previously undocumented memo parameter, which Ken told me about this week). This is working whether or not the alert has a DAT file.

BTW, I’m very close to releasing an update that will add memo editing to the script.

 

[EvanVanVan]

The script is directly importing the alert memos from the Blue Iris database via the JSON interface ‘alertlist’ command.
I would first try to do a database Compact/Repair, and if that doesn’t work, so a Repair/Regenerate

Ah, ok, I was trying to use it on local copies of the DAT files on my computer, while giving it webserver access via VPN. I was probably screwing something up in the process.

In my experience this is not the case with the current version of Blue Iris. The past few days I’ve been editing the memos of existing alerts using the JSON interface ‘update’ command (via a previously undocumented ‘memo’ parameter, which Ken told me about this week). This works whether or not the alert has a DAT file.

BTW, I’m very close to releasing an update that will add memo editing to the script.

Very cool, can't wait! Thank you!

 

[jaydeel]

Ah, ok, I was trying to use it on local copies of the DAT files on my computer, while giving it webserver access via VPN. I was probably screwing something up in the process.

That would do it.

I’ll have the script issue an error message when attempting to use the alert memo features when the DAT files are not coming from the Blue Iris Alerts folder.

 

[EvanVanVan]

That would do it.

I’ll have the script disable the alert memo features when the DAT files are not coming from the Blue Iris Alerts folder.

It may have been my fault entirely...I unchecked everything for the BI User not understanding what access it needed. Now I'm guessing it needs at least View' Clips It might have worked otherwise.

 

[jaydeel]

An update on post #7...

I've yet to hear back from Ken regarding the issue of possibly damaging the database by manually deleting DAT files, but while searching the Help PDF today I found the following excerpt in the Update Notes section for version 5.5.4 (12/8/2021)...

This makes it pretty clear that we probably should not move/delete DAT files outside of Blue Iris.

 

[jaydeel]

Script updated to V1.1. see post #1.

Changelog:

1. NEW: added new argument -ChgAlMemoTags to change/clear Alert memo tag characters (like "!")
2. NEW: added new 'Flags' column to output to show if the associated Alert is 'flagged' or 'protected' in the BI database
3. NEW: added new user-settings to automatically open the output CSV file in a Powershell utility UI (Out-GridView cmdlet)
4. NEW: added new argument -Archive to copy filtered DAT files to \_archive* subfolder in script's destination folder
5. CHANGE: replaced argument -AlertMemos with -AlMemoTags (-AlertMemos remains supported)
6. CHANGE: in the output, non-existent DAT files now assigned 'AgeHrs' and 'ImgsCnt' column values of -1 (previously 0)
7. CHANGE: in the output, -NoInspect DAT files now assigne 'ImgsCnt' column values of -1 (previously 0)
8. IMPROVEMENT: script now recognizes unedited BI webserver address & credentials in the user-settings file
9. IMPROVEMENT: more efficient JSON 'alertlist' GET when using arg -AlMemoTags (uses 'search' parameter)
10. DISABLED: argument -Delete has been disabled until Blue Iris provides a safe method to delete DAT files

Item #3 shows the console's output in a nice built-in Powershell CSV viewer (Out-GridView cmdlet), which also supports column filtering.

 

[EvanVanVan]

Nice, can't wait to get this automated. It's really going to help improve the accuracy of my model and super efficiently at that!

Thank you

 

[EvanVanVan]

Hi, so I'm playing around with this trying to find the right command before automating. It's scanning everything properly and I'm getting the hang of it lol.

One issue, Alert Memos don't seem to work with cancelled alerts? Whether the BI alert memo is "nothing found," or "occupied," or even a postive "To Cancel" label (i.e. "Dog:86%"), the "Alert Memo" field is blank after the tool scans the DAT file. (Labels Found scans properly though.)

Is that correct/intended? Jw, in case the AI misses something and I want to process the cancelled alert. I see a couple of -IfAiNothing arguments, but can't seem to get them to filter/search correctly. (I think it's the -AlMemoTag "!" that's filtering everything out"

Worth mentioning, -ChgAlMemoTags "" (thank you!) also requires Admin...took me a few attempts before I figured that one out lol. Makes sense, but without it the process just hung until interrupted so it was hard to diagnose.

Thank you for the tool!

Side note: Damn, after roboflowing so many hi-res camera images, it's hard for my OCD to handle working with the sub-stream-AI processed images. (Even though I know the first thing Roboflow does is "stretch" them down to 640x640 upon processing.) I might have to waste the processing power and switch the AI to "use main stream when available" for a bit. (Then my OCD will really stress about training after actually streching from 480px to 640px lol)

 

[MikeLud1]

Then my OCD will really stress about training after actually streching from 480px to 640px lol)

Use ether D1 or VGA for your Sub Stream. When training it does not stretch 480px it fills the rest of the image with the color gray

 

[MikeLud1]

Then my OCD will really stress about training after actually streching from 480px to 640px lol

Also RoboFlow has the option to fill black or white

 

[jaydeel]

One issue, Alert Memos don't seem to work with cancelled alerts? Whether the BI alert memo is "nothing found," or "occupied," or even a postive "To Cancel" label (i.e. "Dog:86%"), the "Alert Memo" field is blank after the tool scans the DAT file. (Labels Found scans properly though.)

Is that correct/intended? Jw, in case the AI misses something and I want to process the cancelled alert. I see a couple of -IfAiNothing arguments, but can't seem to get them to filter/search correctly. (I think it's the -AlMemoTag "!" that's filtering everything out"

A couple of points:

• Arguments -AlertMemos and -AlMemoTags are identical and can be used interchangeably.
• Arguments -AlertMemos and -AlMemoTags are 'DAT filters'. That is, if used, the script will only output results for those DAT files where the associated Blue Iris alert's database memo matches the specified substring (e.g., '!') anywhere in the alert's memo.
• 'IfAi' arguments like -IfAiNothingAny and -IfAiNothingAny are 'JPG filters'. When used they restrict which JPGs are extracted. They do this by looking at the AI labels the JPGs do, or do not, contain... I understand that this can initially seem very complicated (even more so to code!). Please re-read the 'false positives' and 'false negatives' script command examples in post #3; note how they are used to control which JPGs are extracted. Hope this eventually makes sense, lol.

If you do not use the the -AlertMemos and -AlMemoTags arguments, the output should show results for 'nothing found', etc.
For example...

EDIT: BTW, the screenshot above is from version 1.2 (still in process). It adds a column ('RO') showing if the specified DAT file and its associated alert JPG (if it exists) have their Windows 'read-only' file attributes set... Per Ken at Blue Iris support, so-called 'protected' alerts (as marked in the database) will be certain to survive a Database/Regenerate action only if one these files have the read-only attribute set (next version of Blue Iris). So knowing if the read-only attributes are set will be a key piece of information if you are trying to protect special alerts.

Worth mentioning, -ChgAlMemoTags "" (thank you!) also requires Admin...took me a few attempts before I figured that one out lol. Makes sense, but without it the process just hung until interrupted so it was hard to diagnose.

Thanks for the heads up.
In version 1.2, I'll 1) try to trap this condition, and 2) mention this more explicitly in the user-settings file.

 

[EvanVanVan]

Sorry, I'm just able to get back to this and take a real look...

• Arguments -AlertMemos and -AlMemoTags are 'DAT filters'. That is, if used, the script will only output results for those DAT files where the associated Blue Iris alert's database memo matches the specified substring (e.g., '!') anywhere in the alert's memo.
• If you do not use the the -AlertMemos and -AlMemoTags arguments, the output should show results for 'nothing found', etc.

Ok, I was just trying to filter/process a cancelled alert by adding an explanation point to the alert memo (i.e. "nothing found!") (when the AI misses something entirely) and it wasn't processed. I think because the tool ignores (or can't read) the database alert memos of cancelled alerts? (Or I'm just missing something.) If possible, could you add that feature?

• 'IfAi' arguments like -IfAiNothingAny and -IfAiNothingAny are 'JPG filters'. When used they restrict which JPGs are extracted. They do this by looking at the AI labels the JPGs do, or do not, contain... I understand that this can initially seem very complicated (even more so to code!). Please re-read the 'false positives' and 'false negatives' script command examples in post 3; note how they are used to control which JPGs are extracted. Hope this eventually makes sense, lol.

Yeah, those arguments are super confusing lol. I think I eventually got them though...

• For a POSITIVE alert, ifAiNothingAny extracts all the "other images" sumitted to the AI where nothing was found, before/after an object was successfully detected?
• For a POSITIVE alert, ifAISomethingAny extracts images where an object was detected but was different/incorrect from the final AI label after the analysis ended?

lol the examples were a bit more confusing because I didn't understand the -Patterns "FE.","FD.","GT" argument. I think they are your camera names lol?

Suggestion for 1.2: Can you make -SingleFolder an argument that obeys the user set destination path, rather than a folder named SingleFolder? Basically I'd like to extract directly into the destination Images folder. I thought about updating it in the script, but then I would have to remember to change it with every update lol

Thank you