CodeProject.AI 2.9.5 w/YOLOv5.NET (GPU/DirectML) memory growth/leak

C

carteriii

Pulling my weight
Jan 8, 2016
164
173
USA
Running on Windows 11 Pro, running about 40 streams through CP.AI, after enough time & events, the memory use of the CodeProject.AI Server through Task Manager still shows only ~150MB on the process tab, BUT the total memory use shown on the performance tab grows over time to 10-15GB at which point I either have to restart the service or bad things happen. When I use the services window to restart "CodeProject.AI Server", the system memory use drops by that 10-15GB, indicating that the memory lost is within CodeProject.AI. Here are two screenshots showing the stated memory usage, and then the memory usage after stopping the CodeProject.AI Server after about 440k calls to the Object Detection (v1.14.0), and 77k calls to the License Plate Reader (3.3.4).
bi-cpai-process-memory-use.jpg
cpai-memory-drop.jpg

Having said that, I realize it's possible this is due to BlueIris' interaction with it, perhaps not freeing up connections, BUT then I would expect the process memory use to show growth as well.

The 10GB+ memory growth typically happens over the course of a week or more, but when the winds are blowing causing lots of false activations, it can happen in just a few days. At a minimum, it does seem directly proportional to the number of calls to CP.AI. I have a second instance of BI with only 4 cameras and never see this kind of growth.

Has anyone else experienced this? Are there specific lines in the CP.AI log file that I can try to match up (e.g. new connection call vs. free'd up connection or call) that might help identify or at least narrow down the root cause of this? I'm not asking anyone to do the work for me, but please help steer me in a direction that might result in a finding that can be useful to everyone. Thanks in advance.
 
I've had some incredible instability lately and made a post on it here:

kjinxx2

Thread 'Very Odd CodeProject.AI/Blue Iris Behavior - Thoughts?'

Hi all - running BI 5.9.9.24 (latest stable update) and CodeProject.AI 2.9.5.0

I don't know when/why this started - but BlueIris is randomly receiving TONS of positive identifications on all of my cameras from CodeProject.AI. Below is just an example, this happens on all of my cameras. I've tried reinstalling CPAI, updating BI (this started happening last stable release), etc. I've seen this issue before where multiple models are triggering, but I've configured BI & CPAI to only use one model.

Below is a picture of the alert image & AI Debugger...

Any thoughts on how I can debug this...

I also am experiencing memory leakage. Every few days CPAI completely stops responding to BI and I must restart CPAI through the BI interface (or restart the machine).

I never solved this issue
 
I'd like to add to this, that this is still an ongoing, and major issue. I have tried all number of combinations to resolve this and none seem to resolve it. My default setup is running CPAI in a docker container on unraid, but I have tried directly in a windows install as well with no change.

In my testing, I have found that no model seems to make any difference, Yolo v6, Yolov8, Yolo .net variants all experience the issue the same, and regardless of GPU (nvidia in my case) or CPU processing, the leak still occurs. I've tested this across multiple servers and the results are the same.

There have been no updates to CPAI in over 8 months, and this issue just cropped up about a month or so ago, maybe less, with no change to CPAI at all, so the issue seems to be in the way Blue Iris is sending requests to CPAI and there's no garbage collection being done. I'm not sure if requests are being sent a new way or something, but from all the variables I've considered and tested, this is a breaking change from Blue Iris's side, not CPAI.

Have messaged support about this and awaiting a response. Hopefully can get this resolved soon as ballooning from 4gb to 15+ and never stopping is hugely problematic lol.
 
  • Like
Reactions: prsmith777
I found the issue to be related to the static object detection (which would prevent movement in a camera from triggering an alert if, say, a car was parked in your driveway and not moving but a shadow triggered movement, etc). I see that in the latest patch this has all been reworked to a 'periodic scan' feature but I haven't yet been able to mess with it. Disabling the 'static object detection' on all of my cameras resolved the issue for me.
 
prsmith777 said:
Also noticing this. My bandaid solution is to reboot every few days
Click to expand...
Agreed. I first saw this last week I think. It was disconcerting because I just stopped getting alerts, with no good way to monitor the behavior.
For a while, I was rebooting the BI machine weekly.
But at one time, I got close to one million inferences.
Same 2.9.5 version.
I didn't realize it has been 8 months.
It's starting to look like abandonware to me.
Having good luck with Frigate on Ubuntu with a Coral USB accelerator.
(I also miss Deepstack)

Sent from my iPlay_50 using Tapatalk
 
  • Like
Reactions: kjinxx2
hapstabu said:
Agreed. I first saw this last week I think. It was disconcerting because I just stopped getting alerts, with no good way to monitor the behavior.
For a while, I was rebooting the BI machine weekly.
But at one time, I got close to one million inferences.
Same 2.9.5 version.
I didn't realize it has been 8 months.
It's starting to look like abandonware to me.
Having good luck with Frigate on Ubuntu with a Coral USB accelerator.
(I also miss Deepstack)

Sent from my iPlay_50 using Tapatalk
Click to expand...

Yes this was the exact same behavior that I experienced.
 
You must log in or register to reply here.
Top Bottom