AI: error 500 and AI: error 200

BTRKY

Young grasshopper
Joined
Sep 21, 2016
Messages
39
Reaction score
14
Switching to CPU only destroyed my CPU usage to 100%. Switching back to GPU. I'm running an RTX3090.
 

randallc79

n3wb
Joined
Feb 28, 2023
Messages
26
Reaction score
2
Location
Noble, OK
Switching to CPU only destroyed my CPU usage to 100%. Switching back to GPU. I'm running an RTX3090.
I had this same problem... that's why I moved my AI to a dedicated box and as well as using this nVidia tesla p4.
 

randallc79

n3wb
Joined
Feb 28, 2023
Messages
26
Reaction score
2
Location
Noble, OK
So, far since making that last change on the AI side... this has been the only error I've seen pop up now, randomly...

Code:
14:00:14:Response received (...222274)
14:00:14:Request 'detect' dequeued for processing (...2fd08c)
14:00:14:Object Detection (Net): Command completed in 1968 ms.
14:00:14:Client request 'recognize' in the queue (...230a68)
14:00:14:Request 'recognize' dequeued for processing (...230a68)
14:00:14:Client request 'detect' in the queue (...0b9cff)
14:00:14:Face Processing: Retrieved face_queue command
14:00:14:Face Processing: An Error occured during processing: Traceback (most recent call last):
14:00:14:Response received (...230a68)
14:00:14:Face Processing: Queue and Processing Face Processing command 'recognize' (...230a68) took 174ms
 

Village Guy

Pulling my weight
Joined
May 6, 2020
Messages
291
Reaction score
161
Location
UK
So, far since making that last change on the AI side... this has been the only error I've seen pop up now, randomly...

Code:
14:00:14:Response received (...222274)
14:00:14:Request 'detect' dequeued for processing (...2fd08c)
14:00:14:Object Detection (Net): Command completed in 1968 ms.
14:00:14:Client request 'recognize' in the queue (...230a68)
14:00:14:Request 'recognize' dequeued for processing (...230a68)
14:00:14:Client request 'detect' in the queue (...0b9cff)
14:00:14:Face Processing: Retrieved face_queue command
14:00:14:Face Processing: An Error occured during processing: Traceback (most recent call last):
14:00:14:Response received (...230a68)
14:00:14:Face Processing: Queue and Processing Face Processing command 'recognize' (...230a68) took 174ms
BI Face processing is pretty much a demo facility at this time. It is unreliable at best. Try running with face recognition turned off.
 

looney2ns

IPCT Contributor
Joined
Sep 25, 2016
Messages
15,412
Reaction score
22,425
Location
Evansville, In. USA
Everyone having these errors, be sure to "properly" report this to Blue Iris support.
The more that report, the higher chance support will look into it.
See BI help for the proper support method.
 

randallc79

n3wb
Joined
Feb 28, 2023
Messages
26
Reaction score
2
Location
Noble, OK
After talking to Blue Iris and CP AI, heres what I know. The problem originated with CP AI 2.7, 2.8 was intended to fix the hi cpu issues with 2.7, but has apparently made matters worse; they are working the issue although its probably on hold till Monday. . BI is away of the problem, but as the problem originates with CP they (CP) will need to take lead.

I have had some success by rolling back BOTH CP AI and BI by several versions, however that was not a total fix. Rolling back too far and with mismatched versions creates issues with python, .net, and various other services. I finally gave up and will just wait for the fix this week.
Have you by chance heard anything back from CP AI about this issue?
 
Joined
Jul 25, 2021
Messages
6
Reaction score
4
I'm getting the same thing and have tried upgrading my GPU drivers. Anyone know how to downgrade back to deepstack? I tried reinstalling it but I don't see an option to switch it back in BI
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
23,713
Reaction score
45,909
Location
USA
Unless something changed, BI is simply providing images to the AI program and taking the results and putting it back into BI, so whether you have DS or CodeProject, it is pointing to the same area.

Now with that said, if it hasn't completely broke already, at some point all of the issues to get CodeProject working will eventually (if it hasn't yet) break functionality with Deepstack.

That is why many of us have not updated BI to beyond when Deepstack was still the only player in that arena.

So if you are not able to get DS running in your current version, consider rolling back to the update prior to when CodeProject came.
 

kaltertod

Getting the hang of it
Joined
Jul 30, 2022
Messages
64
Reaction score
44
Location
BFE
I have done a little more investigating and logging on the 2 BI systems that I have running, My remote system running 5 cameras, using a GTX 1650 show zero error:500 over the course of time since the last update. My local Server running 9 cameras, and using a RTX2060 12GB shows a total of 3 Error:500 instances over the same time period. I am not sure what is causing it but it does not seem as major of a problem as it had first seemed. I tried shooting support another message this morning hopefully they reply to this one. I also sent the boys over at codeproject a message to see what the progress was on fixing this issue.

The puzzling thing to me is why one system shows symptoms and the other one does not. I will have to look at configurations side by side and see if there is something I missed.
 

BTRKY

Young grasshopper
Joined
Sep 21, 2016
Messages
39
Reaction score
14
Following the latest update, I’m getting almost nothing but 200 and 500 errors. RTX3090
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
23,713
Reaction score
45,909
Location
USA
It just shows how difficult it can be to write a program to be used on sooo many different systems and configurations.

From my observations (since I am holding tight on Deepstack and BI version prior to CodeProject) that many of the issues seem to be GPU related.

While some have had no problem with Code Project, many have nothing but trouble.

We didn't see this kind of trouble with people starting Deepstack before the change was made.
 

BTRKY

Young grasshopper
Joined
Sep 21, 2016
Messages
39
Reaction score
14
CP worked fine until the last update of CP (could be a combo of BI and CP updates). There seems to be no progress in CP development over the last several weeks. When I initially moved from DS to CP, CP was superior. I’m hopeful an update will come soon to resolve.
 

Tinman

Known around here
Joined
Nov 2, 2015
Messages
1,192
Reaction score
1,461
Location
USA
It just shows how difficult it can be to write a program to be used on sooo many different systems and configurations.

From my observations (since I am holding tight on Deepstack and BI version prior to CodeProject) that many of the issues seem to be GPU related.

While some have had no problem with Code Project, many have nothing but trouble.

We didn't see this kind of trouble with people starting Deepstack before the change was made.
I agree completely. You can save yourself a lot of headaches by simply upgrading your BI pc for a couple hundred bucks and forget the GPU. If you are running 10 cams or less on AI I really see no need to use GPU. If you really want to go the GPU route, then do some research and ask what card is working the best. I still run a I7-6700k in CPU mode and have had zero errors, the only time AI restarts is when I update BI (which is always the latest, sometimes not the greatest) :)

47.jpg
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
23,713
Reaction score
45,909
Location
USA
I agree completely. You can save yourself a lot of headaches by simply upgrading your BI pc for a couple hundred bucks and forget the GPU. If you are running 10 cams or less on AI I really see no need to use GPU. If you really want to go the GPU route, then do some research and ask what card is working the best. I still run a I7-6700k in CPU mode and have had zero errors, the only time AI restarts is when I update BI (which is always the latest, sometimes not the greatest) :)

View attachment 158068
Yep and can probably find a more powerful computer cheaper than a GPU!
 

kaltertod

Getting the hang of it
Joined
Jul 30, 2022
Messages
64
Reaction score
44
Location
BFE
I agree completely. You can save yourself a lot of headaches by simply upgrading your BI pc for a couple hundred bucks and forget the GPU. If you are running 10 cams or less on AI I really see no need to use GPU. If you really want to go the GPU route, then do some research and ask what card is working the best. I still run a I7-6700k in CPU mode and have had zero errors, the only time AI restarts is when I update BI (which is always the latest, sometimes not the greatest) :)

View attachment 158068
I agree the cpu way is alot more stable. However the gpu is a hair faster and with the amount of detections that I am getting at both locations the gpu makes more sense. Since the last update the other day I have more than 4 million analyzed images just on one of my machines. And have only popped 3 Error:500 codes. The other machine has analyzed just north of 3 million images and shows no errors using the gpu.

However, depending on the resolutions of the cameras being used it is fairly easy to tax even a higher end cpu with just the transcoding and decoding of the video streams. I use dual streams on my 9 camera system(5820K @ 4.4ghz all core, 32GB ram, RTX 2060 12GB) and it is not uncommon to see the cpu spike up the 100% usage for short durations if there is alot of motion going on. I believe that this is mainly due to the decoding of the video for analysis purposes.

The thing that remains puzzling to me is that a 1650 with 4GB of ram shows no errors what so ever. So I dont think that it is strictly a video card problem or a vram problem anyway.

It is however based on use case and what you need the machine to do as far as analysis goes. If you are trying to capture fast moving objects such as cars then you need to send more frames to be analyzed at a faster pace. I have not broken into ALPR but I am sure that requires even more images to be sent to get a correct detection. With more images being analyzed in a short duration the gpu makes more sense to me. Analysis times for CPU on the above system average 150-175ms on the GPU in the same system they are averaging 60-75ms.

In my case the gpu made sense as I was able to get them on a good deal used before the GPU prices hit the fan. For a new setup I believe that for now the cpu only approach would be cheaper.

I am going to drop the main rig back to cpu analysis for now and see how the cpu usage is compared to the gpu usage just to get away from the errors. (BI sends me a notification when I get errors)

**Edit well that did not take long Multiple Error:200 (No Error:500 though ;) ). Detection times are running on Average 550-650ms on the same machine and CPU pings 100% usage for the duration of the analysis. Back to GPU lol.
 
Last edited:

randallc79

n3wb
Joined
Feb 28, 2023
Messages
26
Reaction score
2
Location
Noble, OK
I'm running about 18 cams and have a dedicated server for BI and one for AI (with direct NIC link between them). The whole point was to spread out the CPU load because it was really hammering my main BI system with the AI installed on it, while also utilizing GPU on dedicated AI server. So after these latest comments, I went to the AI server and switched, "Face Processing" to CPU and Enabled Object Detection (YOLOv5.NET). It's spiked my CPU usage on my AI server, but this has been the longest I haven't seen any 500 errors.
 

kaltertod

Getting the hang of it
Joined
Jul 30, 2022
Messages
64
Reaction score
44
Location
BFE
Just a little update and a bump for this thread. It appears that this was a codeproject problem I am testing out the 2.1 Beta for them at the current moment and it has resolved all of my Error:500 Problems. I am not sure when they plan to release this version out in the wild as it is still in the testing phase but I will try and keep you posted. Just know there is a fix for it coming down the pipe. :D
 

randallc79

n3wb
Joined
Feb 28, 2023
Messages
26
Reaction score
2
Location
Noble, OK
Just a little update and a bump for this thread. It appears that this was a codeproject problem I am testing out the 2.1 Beta for them at the current moment and it has resolved all of my Error:500 Problems. I am not sure when they plan to release this version out in the wild as it is still in the testing phase but I will try and keep you posted. Just know there is a fix for it coming down the pipe. :D
Thats awesome! Thanks for the update.
 

BTRKY

Young grasshopper
Joined
Sep 21, 2016
Messages
39
Reaction score
14
That is fantastic. I hope the release is soon.
 

kaltertod

Getting the hang of it
Joined
Jul 30, 2022
Messages
64
Reaction score
44
Location
BFE
Currently 6+ million images analyzed on the test machine and still no Error:500 or Error:200.. Hopefully the release is soon but overall I am happy with the results.
 
Top