Memory leak with Blue Iris - persisted through fresh install of windows and Blue Iris

[ChrisPDuck]

I'm desperate for help now, I have been trying to solve a memory leak within Blue Iris
A couple of days ago I installed a new main hard drive, reinstalled a fresh copy of windows, re-installed a fresh copy of Blue Iris (imported settings) and still the memory leak is there.

I have a machine which only runs Blue iris and Plex
OptiPlex 5090 SFF
CPU i5 11500
32Gb RAM
Windows (11 pro) and Blue Iris (v 5.9.4.7) on a 4Tb NVME
Alerts and database on a dedicated 250Gb SATA SSD
Clips stored on 2x6Tb WD purple drives (SATA)
4 cameras go to 1 drive, and 4 camera to the other. Clips are never moved

Camera's
5x 8mp Annke cameras
1x 5mp Annke camera
1x 4mp Annke camera
1x5mp Reolink doorbell

All cameras are connected via POE
All cameras are using substream
All cameras are making use fo hardware acceleration

AI is CodeProject, version 2.6.5, using Google's Coral (internal unit, in an M.2 slot)
No 3rd party virus checkers with Windows inbuilt virus and threat protection exclusions set up to ignore the 'Blue Iris 5' folder, the 'dB' and 'Alerts' folders and both WD purple drives

When first started Blue Iris will use about 2Gb - showing about 20% of the total RAM in use.
Over the next ~5 hours the memory in use by Blue Iris will creep up and up. Once it hits about 18Gb in use (which shows about 80% of RAM in use) I start to loose camera feeds. This will continue until all feeds are lost.
At the moment I'm using the WatchDog function on all 8 cameras to run an external program when a video feed is lost for more than a minute. The external program is a script which kills Blue Iris (taskkill /F /IM BlueIris.exe /T). As Blue Iris is running as a service it restarts itself and the RAM used drops back to 2Gb.
This is crude but it works (It means I have ~3minutes of missing footage each 5 hours, prior to this the machine would simply stop recording anything every 5 hours)

I have tried:

• Rolling back to many different Blue Iris versions (some way before the issue started)
• Changing the AI processing from Coral to CPU (YOLOv5 .NET)
• Disabling one camera at a time
• Turning off AI all together (uninstalling Code Project AI)
• Reinstalling windows from scratch on a new NVME drive, installing a fresh copy on Blue Iris and importing the settings (i have far too complex a set up to start from scratch, i need to import settings)

None of the above have had any impact on the memory leak and i still start to loose camera feeds every 5 hours or so

I think the issue started around 19th March 2024

The only other fact which may be of use to those more clever than myself is it's the commit memory that appears to have the issue, not the actual working set (see below, Blue Iris is only has a working set of less than 2Gb but a commit of over 16Gb)



The log files don't appear to contain anything useful to me but here's an hour of the file during an issue.
I lost the first camera feed (Drive_R) at 4:23:05, the WatchDog/kill script executed at 04:23:53 and the feeds started coming back at 04:24:09, the last feed came back at 04:24:19

0 11/07/2024 04:00:02.911 DB DBCompact: started
0 11/07/2024 04:00:28.860 DB DBCompact: finished
0 11/07/2024 04:00:29.146 DB Reindexing: 44096 records 285 msec
10 11/07/2024 04:00:58.010 Server [::ffff:92.249.48.197]: Connected
0 11/07/2024 04:01:12.635 Stored Delete: nothing to do [5548/5550GB, 39.7GB free]
0 11/07/2024 04:01:12.638 Stored_2 Delete: nothing to do [5549/5550GB, 38.6GB free]
0 11/07/2024 04:11:13.085 Stored_2 Delete: over quota 5550/5550GB, 38.5GB free
0 11/07/2024 04:11:13.120 Clips DeleteFile OK: E:\Stored 2\Shed.20230811_050000.bvr
0 11/07/2024 04:11:13.133 Stored_2 Delete: 1 items 287.8MB
3 11/07/2024 04:15:37.920 Shed Triggered: Motion_ABD
0 11/07/2024 04:15:41.124 Shed AI: Alert canceled [nothing found] 78ms
0 11/07/2024 04:16:13.353 Stored Delete: nothing to do [5548/5550GB, 39.6GB free]
0 11/07/2024 04:16:13.367 Stored_2 Delete: nothing to do [5549/5550GB, 38.7GB free]
1 11/07/2024 04:23:32.727 Drive_R Signal: network retry
1 11/07/2024 04:23:36.787 Gate Signal: network retry
1 11/07/2024 04:23:45.655 Drive_L Signal: network retry
1 11/07/2024 04:23:48.324 Shed Signal: network retry
1 11/07/2024 04:23:59.800 House Signal: network retry
1 11/07/2024 04:24:02.844 Drive_R Signal: network retry
2 11/07/2024 04:24:08.025 App Restarted after unexpected shutdown (5.9.4.7/1500/240)
0 11/07/2024 04:24:08.098 App Intel video hardware detected
0 11/07/2024 04:24:08.258 App Current profile: Night
0 11/07/2024 04:24:08.295 Action Web: 200 OK
0 11/07/2024 04:24:08.295 App AI: is being started
4 11/07/2024 04:24:08.906 Shed Signal: restored
4 11/07/2024 04:24:09.275 Drive_R Signal: restored
4 11/07/2024 04:24:09.288 Drive_L Signal: restored
4 11/07/2024 04:24:09.366 House Signal: restored
4 11/07/2024 04:24:09.423 Gate Signal: restored
0 11/07/2024 04:24:09.817 App Today's sunrise: 04:17:00, sunset: 22:19:00, timestamp=1720668249 (GMT-3600s)
0 11/07/2024 04:24:09.818 Log Retaining 1 files (12.6MB)
0 11/07/2024 04:24:09.824 Server Started on port 81
0 11/07/2024 04:24:10.691 AI responding at 127.0.0.1:32168
0 11/07/2024 04:24:10.770 DB DBCompact: started
0 11/07/2024 04:24:11.308 App AI: has been started
0 11/07/2024 04:24:12.744 DB DBCompact: finished
0 11/07/2024 04:24:13.056 DB Reindexing: 44104 records 317 msec
0 11/07/2024 04:24:13.282 New Move: nothing to do [0.00/1.00GB, 52.6GB free]
0 11/07/2024 04:24:13.393 Stored Delete: nothing to do [5548/5550GB, 39.6GB free]
0 11/07/2024 04:24:13.724 Alerts Delete: nothing to do [9.99/10.0GB, 52.6GB free]
0 11/07/2024 04:24:13.828 Stored_2 Delete: nothing to do [5549/5550GB, 38.6GB free]
4 11/07/2024 04:24:13.828 L: 11175 items, 9.99/11.0GB (52.6GB/222.7GB free, +51.6GB)
4 11/07/2024 04:24:13.828 D: 13317 items, 5548/5550GB (39.6GB/5.45TB free, +38.6GB)
4 11/07/2024 04:24:13.820 E: 19385 items, 5549/5550GB (38.6GB/5.45TB free, +38.5GB)
0 11/07/2024 04:24:18.534 Doorbell HW VA not compatible: 51442
0 11/07/2024 04:29:14.020 Stored Delete: nothing to do [5549/5550GB, 39.5GB free]
0 11/07/2024 04:34:14.266 Stored_2 Delete: over quota 5550/5550GB, 38.5GB free
0 11/07/2024 04:34:14.261 Clips DeleteFile OK: E:\Stored 2\Porch.20230811_050000.bvr
0 11/07/2024 04:34:14.265 Stored_2 Delete: 1 items 304.3MB
0 11/07/2024 04:39:14.494 Stored Delete: nothing to do [5549/5550GB, 39.4GB free]
0 11/07/2024 04:39:14.495 Stored_2 Delete: nothing to do [5549/5550GB, 38.7GB free]
0 11/07/2024 04:44:14.721 Stored Delete: nothing to do [5549/5550GB, 39.3GB free]
0 11/07/2024 04:44:14.722 Stored_2 Delete: nothing to do [5549/5550GB, 38.6GB free]
3 11/07/2024 04:47:21.760 Gate Triggered: Motion_AC
10 11/07/2024 04:51:41.784 Server [::ffff:45.156.128.116]: Connected
0 11/07/2024 04:54:15.190 Stored Delete: nothing to do [5549/5550GB, 39.2GB free]
0 11/07/2024 04:54:15.192 Stored_2 Delete: over quota 5550/5550GB, 38.5GB free
0 11/07/2024 04:54:15.223 Clips DeleteFile OK: E:\Stored 2\Porch.20230811_060000.bvr
0 11/07/2024 04:54:15.226 Stored_2 Delete: 1 items 391.2MB
0 11/07/2024 04:59:15.442 Stored Delete: nothing to do [5549/5550GB, 39.1GB free]
0 11/07/2024 04:59:15.455 Stored_2 Delete: nothing to do [5549/5550GB, 38.8GB free]

 

[IAmATeaf]

Try disabling hardware acceleration on all the cams to try and eliminate that as a possible cause?

 

[wittaj]

It would have been best to not import settings because if the memory leak is in there, you simply brought it over.

When I went to another machine, I simply had two monitors (one for each computer) and would open each camera one by one on the old computer and then set up the camera on the new one.

I have over 30 cameras and multiple spotter cameras for PTZs, unique profiles, and some other complicated settings and it didn't take as long as you think.

But yeah, hopefully disabling hardware acceleration takes care of it. With substreams, using hardware acceleration takes more CPU% to offload to the GPU than the savings in CPU%.

 

[ChrisPDuck]

It would have been best to not import settings because if the memory leak is in there, you simply brought it over.

When I went to another machine, I simply had two monitors (one for each computer) and would open each camera one by one on the old computer and then set up the camera on the new one.

I have over 30 cameras and multiple spotter cameras for PTZs, unique profiles, and some other complicated settings and it didn't take as long as you think.

But yeah, hopefully disabling hardware acceleration takes care of it. With substreams, using hardware acceleration takes more CPU% to offload to the GPU than the savings in CPU%.

I know that would have been best but i really expected the fresh install of the OS and Bi to work. It really is quite a complex setup with multiple zones triggering multiple different actions depending on various schedules with quite a few macro's and webhooks etc
My fear is I'd spend a couple of hours replicating all the settings (no doubt realizing at some point a few I'd missed) just to repeat whatever it is that's causing the memory leak (So I'd still have it, just have caused it manually rather than imported it)
I'm 95% certain I didn't make any changes when it started, which is confusing me.

I couldn't use your approach as I only have one machine where I replaced the HD so I can only have one instance powered up at a time

 

[looney2ns]

Was the fresh Windows install performed using the MS Media Creation Tool and NOT from a manufactures backup?
Also, in the past this has been blamed on bad intel video drivers.
Some background here: Memory Leak: Quick Sync (Hardware Acceleration)

 

[ChrisPDuck]

Was the fresh Windows install performed using the MS Media Creation Tool and NOT from a manufactures backup?
Also, in the past this has been blamed on bad intel video drivers.
Some background here: Memory Leak: Quick Sync (Hardware Acceleration)

Yes it was fresh from a freshly created stick using the media creation tool,

Disabling Hardware acceleration did indeed solve it. Though I can't see it was a change in Intel's driver that triggered it as I've just looked and the driver is from November 2023, a good 4 months before the issue started.
Will try some other drivers (there are 5 newer versions available) and see what happens. Hopefully I don't need to find an older version as they don't seem to be available.