Auto-detect and Fix Blue Iris Memory Leaks

[jaydeel]

I've developed an approach to automatically detect and fix Blue Iris memory leaks.

It requires the following:
1. Creating multiple Blue Iris 'write to a file' actions that log RAM usage to a CSV file every N minutes.
2. Executing a PowerShell script every N minutes (via Task Scheduler) that reads (and acts on) the CSV file.

On my system...

• I use N=6 minutes (10 log entries / checks per hour).
• I've found that restarting the Blue Iris service is usually sufficient to FIX a memory leak.
• Even if the action does not automatically fix the leak, I have still received a notification that I can investigate further.

The following screenshots illustrate two types of leaks detected:
'Continuous' type memory leaks...

'Step' type memory leaks...


The sections below describe how to set this up.
(it's not really as complicated as it may first appear.)

======
'Write-to-a-file' actions

Required...
1. Click Blue Iris Settings > Schedule > 'Execute action set each N min' > 'Edit' button

2. Add actions per the next screenshot, and the Key Point below...

You can copy/paste the following into the respective fields:
Header field: Timestamp,SoftwareUsedRAM,Uptime,RAM_Pct,Tag\n
Parameters field: %Y/%m/%d_%H:%M:%S,%014,%012,%017,"periodic"\n

Key point: Schedule tab action sets are Schedule-specific.
You must repeat steps 1&2 for each Schedule you use (via the 'Now editing' control).

Optional...
This additional action will add entries to the CSV file when the Blue Iris service is restarted.
These entries can be helpful when inspecting the data after a memory leak event.

1. Click the status icon (chart).
2. Select the 'Log' tab.

3. Click the 'Status Alerts...' button.

4. Click the 'Action set' button
5. Repeat Step 2 in the Required section above, replacing the Parameters field with %Y/%m/%d_%H:%M:%S,%014,%012,%017,"status_action"\n

======
Customizing the PowerShell script

To use this script, simply extract the 'BI_memleak_fix.ps1' file from the attached zip file to your desktop, or scripts folder.
(Note: this folder must be present in the Windows environment PATH).
1. Open the script in a text editor.
2. Navigate the 'USER SETTINGS' section.

Each setting is fully described in script.
The settings fall in these categories. Only the first is required.

• [REQUIRED] Location & name of the CSV file in your filesystem.
• Customizable parameters for RAM leak detection.
• Opt in to taking action if a memory leak is detected.. by default, this is a BI service restart (a system reboot is optional)
• Opt in to logging the memory leak event to the Blue Iris logfile, add the BI server address, and add credentials (if needed).
• Opt in to sending a Pushover notification, add API keys, and specify preferences.

Extras in the ZIP file

• The *.csv files in the ZIP file can be used to demo the script from a PowerShell window.
• The *.png files in the ZIP file are charts illustrating the data in the *.csv files.

======
Create/configure Task Scheduler task to run the PowerShell script

This topic is described in the comments at the top of the attached PowerShell script.

# HOW TO AUTOMATE THIS SCRIPT VIA TASK SCHEDULER
#   Open Task Scheduler and create an appropriately named task with two triggers and one action:
#   Task Scheduler
#   'Triggers' tab:
#       'At startup'
#       'Daily' or 'One time' (e.g., at 12:05 AM, every 6 minutes)
#       (Note: I find 'One time' to be more reliable)
#   'Actions' tab:
#       Program/script: powershell.exe
#       Add arguments: {your_powershell_scripts_path)\{script_file_name}.ps1

Tip#1: To make sure the Task Scheduler task is working...
1. Edit the PowerShell script to use one of the demo *.csv files (1, 2 or 3).
2. Also opt in to sending Pushover notifications.
3. Right-click the task, and select 'Run'.
Note: Once you've confirmed that the task is working, don't forget to edit the script, returning variables $path= and $fname back to the original CSV file.

Tip#2: If you are getting spammed with Pushover notifications, you do not need to delete the task.
Rather, right-click the task and select 'Disable'. Then debug the task and/or script.

======
Using/Testing the PowerShell Script

To experiment with this script, simply
1. Extract the BI_memleak_fix.ps1 file from the attached zip file to your desktop, or scripts folder.
2. Run the script from a Admin PowerShell window.
The script can also be executed from an Admin cmd.exe window using a command like powershell.exe "C:\ps_scripts\BI_memleak_fix.ps1"

NOTE: If you've never used PowerShell on your Blue Iris PC, you may first to need to change its default security policy.
Open PowerShell in Administrator mode, and type the following, then hit the Enter key Set-ExecutionPolicy RemoteSigned
(click <here> for a nice beginner's tutorial to using PowerShell).

The following screenshots illustrate the script's output (in a PowerShell window) using the extra *.csv files in the *.zip file.
1. No event detected.
2. 'Continuous' type leak event detected.
3. 'Step' type leak event detected.
4. 'Continuous' type leak event detected (2nd example).
5. Same as #4, triggering a Blue Iris service restart.

 

[jaydeel]

Update...

Yesterday I experienced a memory leak event that defied the methodology described in the 1st post above.

What happened was this... Blue Iris stopped executing Action set actions during the leak event.
Thus the Blue Iris write-to-file action CSV file being used by the PowerShell script was not being updated during the event.

Below is a chart of the write-to-file action memory usage data before, during and after the memory leak event.
Note the lack of data during the timespan enclosed by the red rectangle.

I only discovered this because I use a redundant method to log memory usage.
This data clearly shows the memory leak event.

My alternate method uses the Blue Iris JSON Interface to query and logs the values below.

JSON status
Array
(
    [ram] => 1134108672
    [memphys] => 15.9G
    [memload] => 63%
    [uptime] => 0:00:14:39
)

Key Take-away: The Blue Iris webserver continued to return data during a memory leak event that killed actions in a Blue Iris Action set.

When I get some time, I'll try to write a PowerShell script that logs Blue Iris memory usage data via the JSON Interface.

 

[jaydeel]

Update...

I've updated the utility to V2 in post #1.

V2 - 2023-04-20

• improved demo mode handling
• additional test examples

The provided test examples include an image illustrating the data points (green box) that trigger a memleak event by the script. Every test *.csv is from an actual event.

Example:

 

[jaydeel]

Update...

Early this morning I experienced a memory leak event that flipped my observation for the event in post #2 above.

What happened was this... the redundant CSV log file being updated by Blue Iris JSON interface requests stopped being updated during the leak event.

On the other hand, the original CSV log file being updated by Blue Iris write-to-file actions continued to be updated during the event.

Below is a chart of the write-to-file action memory usage data before, during and after the memory leak event.
Compare this to the 1st chart in post #2 above.

This next screenshot shows the memory usage data collected via the Blue Iris web server JSON interface.
Note the lack of data during the timespan enclosed by the red rectangle, which spans the memleak event in the above chart.

Conclusion: To successfully detect a Blue Iris memory leak event, it appears that I need to use two scheduled tasks running the PowerShell BI_memleak_fix.ps1 script.

1. One reading the CSV file being updated via Blue Iris Action Set actions.
2. The other reading the CSV file being updated via Blue Iris JSON interface requests.