Unfortunately this wasn't it. I hope it's just some setting I can disable. I don't want to be left behind in updates. Seems no one else is having this problem. It disappears if I disable ai, so it's something related to that, I believe.
CodeProject.AI Version 2.5
- Thread starter MikeLud1
- Start date
Oh good to know!The .NET module works with all GPUs. Depending on your Nvidia GPU .NET might be faster then CUDA
GitHub - microsoft/DirectML: DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported hardware and drivers, including all DirectX 12-capable GPUs from vendors such as AMD, Intel, NVIDIA, and Qualcomm.
DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported ...github.com
View attachment 191627
This partly answers a question I've been thinking of. If .NET is faster or equal as CUDA with some GPUs, what is the point of running CUDA version of the model?
I recently upgraded to 4070 Super Ti on my desktop / BI test machine. While experimenting with it, I noticed that YoloV5.NET is faster than Yolov5 62 (cuda version). Added bonus was that .NET was whole lot easier to get up and running, no need to install specific version of Cuda drivers that are compatible with Cuda version of Yolov5. Yolov5.NET worked out of the box with what ever drivers were supplied when I installed the Nvidia software package while installing the GPU.
Or am I missing something by using .NET?
MikeLud1
IPCT Contributor
You are not missing anything by running the .NET model. The models for the .NET are the 6.2 models converted from a PT model to ONNX model. They are trained using YOLOv5 6.2.
I upgraded to CPAI 2.5.6 back on 3/17 and it hasn't been going very well. I started off running YOLOv5 6.2, but was getting slower response than I thought I should. I switched over to YOLOv5 .NET and things seemed to be better, but now several times a day BI complains about the AI not responding, and eventually restarts CPAI. The BI logs show most AI response times to be between 75-200ms with the occasional response in the 300-500ms range, but then will log a few 10000ms -30000ms response times before going to dozens or hundreds of AI: not responding entries before BI finally restarts CPAI.
Horsepower and memory shouldn't be the issue. The system has a Core i7-13700K, 32GB of DDR5-7200 memory, and an Nvidia RTX 4070.
Upgrading/downgrading/reinstalling CPAI is a task I don't take lightly as I'm on a horrible 12Mbps DSL connection.
@MikeLud1 Is CPAI 2.6.2 stable enough to be a likely improvement over the 2.5.6 version I'm running or are there bug fixes coming soon enough I should hold off?
Server version: 2.5.6
System: Windows
Operating System: Windows (Microsoft Windows 11 version 10.0.22631)
CPUs: 13th Gen Intel(R) Core(TM) i7-13700K (Intel)
1 CPU x 16 cores. 24 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 4070 (12 GiB) (NVIDIA)
Driver: 546.12, CUDA: 12.3 (up to: 12.3), Compute: 8.9, cuDNN: 8.9
System RAM: 32 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 8.0.0
.NET SDK: 8.0.100
Default Python: Not found
Go: Not found
NodeJS: Not found
Video adapter info:
NVIDIA GeForce RTX 4070:
Driver Version 31.0.15.4612
Video Processor NVIDIA GeForce RTX 4070
System GPU info:
GPU 3D Usage 43%
GPU RAM Usage 3.8 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
Horsepower and memory shouldn't be the issue. The system has a Core i7-13700K, 32GB of DDR5-7200 memory, and an Nvidia RTX 4070.
Upgrading/downgrading/reinstalling CPAI is a task I don't take lightly as I'm on a horrible 12Mbps DSL connection.
@MikeLud1 Is CPAI 2.6.2 stable enough to be a likely improvement over the 2.5.6 version I'm running or are there bug fixes coming soon enough I should hold off?
Server version: 2.5.6
System: Windows
Operating System: Windows (Microsoft Windows 11 version 10.0.22631)
CPUs: 13th Gen Intel(R) Core(TM) i7-13700K (Intel)
1 CPU x 16 cores. 24 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 4070 (12 GiB) (NVIDIA)
Driver: 546.12, CUDA: 12.3 (up to: 12.3), Compute: 8.9, cuDNN: 8.9
System RAM: 32 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 8.0.0
.NET SDK: 8.0.100
Default Python: Not found
Go: Not found
NodeJS: Not found
Video adapter info:
NVIDIA GeForce RTX 4070:
Driver Version 31.0.15.4612
Video Processor NVIDIA GeForce RTX 4070
System GPU info:
GPU 3D Usage 43%
GPU RAM Usage 3.8 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
Last edited:
MikeLud1
IPCT Contributor
You are not missing anything by running the .NET model. The models for .NET are the 6.2 models converted from a PT model to ONNX model. They are trained using YOLOv5 6.2.
In response to my own post. I went ahead and upgraded to CPAI 2.6.2. Being able to choose to remove existing modules and only install LPR and YOLOv5 .NET made the install a lot easier.
Unfortunately I've already seen BI report that the AI is not responding, followed by restarts of CPAI, so whatever issue I was having with 2.5.6 seems to also be plaguing me at V2.6.2.
Are there any troubleshooting steps I can take to figure out what is going on when BI thinks CPAI is unresponsive?
Unfortunately I've already seen BI report that the AI is not responding, followed by restarts of CPAI, so whatever issue I was having with 2.5.6 seems to also be plaguing me at V2.6.2.
Are there any troubleshooting steps I can take to figure out what is going on when BI thinks CPAI is unresponsive?
Current System Info:
Code:
Server version: 2.6.2
System: Windows
Operating System: Windows (Microsoft Windows 11 version 10.0.22631)
CPUs: 13th Gen Intel(R) Core(TM) i7-13700K (Intel)
1 CPU x 16 cores. 24 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 4070 (12 GiB) (NVIDIA)
Driver: 546.12, CUDA: 12.3 (up to: 12.3), Compute: 8.9, cuDNN: 8.9
System RAM: 32 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 8.0.0
.NET SDK: 8.0.100
Default Python: Not found
Go: Not found
NodeJS: Not found
Rust: Not found
Video adapter info:
NVIDIA GeForce RTX 4070:
Driver Version 31.0.15.4612
Video Processor NVIDIA GeForce RTX 4070
System GPU info:
GPU 3D Usage 4%
GPU RAM Usage 4.1 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168MikeLud1
IPCT Contributor
I am just wondering if it is a Windows 11capaitablity issue. I am still on Windows 10 and have similar GPUs that are working
On my machine when I tried to use .NET DirectML CP.AI would use my Intel UHD GPU, however when I changed to YOLOv5 6.2 it would use the NVidia CUDA from my RTX 2060. The times to detect were roughly 8-10x longer with .NET than the 6.2 times. It may be something with how my system is setup though. Maybe there’s a setting missing to tell CP.AI to use the NVidia card when I select .NET, I’m not sure.
The issue I kept having with CP.AI not responding and timing out seemed to be caused by Blue Iris restarting the service in my circumstance. I changed the BI restarting of CP.AI to 60 rather than what I had it at before, maybe 30 seconds, and it seems to have resolved that issue.
I’m running Windows 11 and it works fine after setting BI not to restart CP.AI until 60 seconds of timeout.
When I built this machine back in December I did a clean install of Win 11, BI 5.7.8.3, and CPAI 2.3.4.
CPAI was running pretty much rock solid. I probably should have left it alone, but I wanted to bring things up to date and try out LPR's cropped plate feature.
I just bumped the AI timeout to 60 and will see if that helps. Thanks
Sweet.
The only difference I've spotted is that ONNX model file size seems to be roughly double the equivalent PT model file size. I assume that this is just minor "cosmetic" difference, both models have the same number of parameters and should consume about the same amount of GPU memory?
The reason I am asking is that my plan is to run BI on a separate machine with a low powered AMD RX6400 4GB GPU (still on the way) , so memory consumption might be an issue going forward.
Here is my "System Info". I haven't had any issues with .NET using 2.5.6 or 2.6.2. You seem to have more recent version of .NET runtime, maybe using older version could help?
Server version: 2.6.2
System: Windows
Operating System: Windows (Microsoft Windows 11 version 10.0.22631)
CPUs: AMD Ryzen 7 3700X 8-Core Processor (AMD)
1 CPU x 8 cores. 16 logical processors (x64)
GPU (Primary): NVIDIA GeForce RTX 4070 Ti SUPER (16 GiB) (NVIDIA)
Driver: 551.86, CUDA: 12.4 (up to: 12.4), Compute: 8.9, cuDNN:
System RAM: 32 GiB
Platform: Windows
BuildConfig: Release
Execution Env: Native
Runtime Env: Production
Runtimes installed:
.NET runtime: 7.0.17
.NET SDK: Not found
Default Python: Not found
Go: Not found
NodeJS: Not found
Rust: Not found
Video adapter info:
NVIDIA GeForce RTX 4070 Ti SUPER:
Driver Version 31.0.15.5186
Video Processor NVIDIA GeForce RTX 4070 Ti SUPER
System GPU info:
GPU 3D Usage 32%
GPU RAM Usage 1.6 GiB
Global Environment variables:
CPAI_APPROOTPATH = <root>
CPAI_PORT = 32168
16 hours since raising AI timeout to 60 seconds. Not one single "AI: not responding" entry in the BI log.
Still don't understand why BI log file would show response times ranging 200-900ms with the hardware I have.
@snuuba what are the typical response times you see in your BI log WITH your 4070 TI?
Still don't understand why BI log file would show response times ranging 200-900ms with the hardware I have.
@snuuba what are the typical response times you see in your BI log WITH your 4070 TI?
Last edited:
My 2060 is 30 to 200 ms if it identifies something most of the time, but if it’s a more complex scene or nearby colors to the object detected, it can take longer. Usually if it fails to detect anything, such as nothing found, then it could be up to 1200 or even 1500 ms regularly.
From BI phone app I can only see response times for situations where AI did not find anything, ranging from 30-300 ms. Typically 80-100 ms.
Not sure how exactly BI calculates these numbers, but the model size, number of images sent to AI per trigger and image size (main vs substream) will probably effect these numbers, i.e more work will take more time.
I am running two models for each trigger using substreams, ipcam-general for detecting persons/vehicles and my custom built model for detecting my dogs. At least my dog model is based on yolov5 small model and by looking at ipcam-general file size, it seems to be based on yolov5 small as well.
Most of my cameras will send 8-12 images to CPAI. From CPAI log each individual inference is around 7-40 ms.
Which model / model size are you using? How many images per trigger is sent to CPAI?
Maybe lowering the level work of CPAI has to perform would help, i.e using smaller model or sending less images per trigger?
Are you using substreams for AI for all of your cameras?
Your post got me thinking, and after looking at the AI analysis details for a few events on the AI tab of the BI Status window I can see that the time shown in the BI log file is the sum of all of the times for all of the models used for the particular image that was used to produce the AI result reported, whether that was a positive result or a "nothing found".
as @XDRDX mentioned, the times seem to be longer if nothing is found.
I'm using ipcam-general and ipcam-animal on most of my cameras and my LPR camera is using ipcam-general and license-plate. AI details from the LPR camera shows that the total includes times for Plates, license-plate and ipcam-general.
I send Main stream images to CPAI and set it to burn label mark-up onto the alert images because that way I can have Pushover send me the full resolution image of a verified event and open that full resolution image in Pushover and see what's going on when a confirmed alert comes in.
Analysis of 50k response times out of the CPAI log shows that more that 50% of the response times are sub 60ms and 90% are sub 105ms.
I think I can live with that.
Last edited:
This is probably the main factor for longer response times. I've seen typical response time in CPAI log jump from normal ~10-40 ms to ~80-120 ms while analysing main stream images.
I only use main stream once and while when collecting new images to retrain my custom dog model. It seems to help with accuracy if you have some images from each camera, so I typically add images and retrain the model when a new camera is installed or if I notice that particular camera is performing worse than other cameras.
Also with powerful GPU like 4070 it might be that CPAI is not sending enough work to GPU to keep it at full throttle. When nothing is happening my 4070 hovers at around 200 Mhz and ramps up shortly to higher frequency when CPAI is processing images. I noticed this the last time I was retraining my model. GPU utilization was constantly +90% and my gut feeling was that busy GPU might prolong CPAI response times, but nope instead of typical 10-40ms response time, I was getting fairly constant 10-15 ms as the GPU was already running at 2850 Mhz.
I use static object analysis and CPAI response times for these is closer 40ms, i.e it is so little work that GPU doesn't have enough work to ramp up to full potential before all the images are already processed. On the other hand when there is actual movement detected and multiple cameras are triggering and retriggering and sending more images to CPAI to be processed, then CPAI response time is typically closer to 10-15 ms.
Finally my low-powered /low profile / single slot AMD Radeon RX6400 GPU arrived after a month of waiting and I was able to move Blue Iris & CPAI from my desktop computer to dedicated SFF server.
While RX6400 seems to be universally hated by all the gaming youtubers, it is doing surprisingly well in CPAI use. For static object analysis there is not much difference to my Nvidia 4070 Super Ti based desktop, with both GPUs average CPAI response time is around 40 ms per image. Even when there is actual moment CPAI response time with RX6400 is decent enough around 20-30 ms, roughly double what it takes with 4070, but I can live with that.
There seems to be a bug in CPAI 2.6.2 version. Status window shows that the model is running in "CPU (DirectML)" mode even though CPAI is really using GPU (as seen from info).
The best part is that the RX6400 is consuming only around 8-10W of power (according to AMD software). No point of dreaming of low-powered Coral setup anymore.
While RX6400 seems to be universally hated by all the gaming youtubers, it is doing surprisingly well in CPAI use. For static object analysis there is not much difference to my Nvidia 4070 Super Ti based desktop, with both GPUs average CPAI response time is around 40 ms per image. Even when there is actual moment CPAI response time with RX6400 is decent enough around 20-30 ms, roughly double what it takes with 4070, but I can live with that.
There seems to be a bug in CPAI 2.6.2 version. Status window shows that the model is running in "CPU (DirectML)" mode even though CPAI is really using GPU (as seen from info).
The best part is that the RX6400 is consuming only around 8-10W of power (according to AMD software). No point of dreaming of low-powered Coral setup anymore.
Last edited: