Switching to CPU only destroyed my CPU usage to 100%. Switching back to GPU. I'm running an RTX3090.
AI: error 500 and AI: error 200
- Thread starter BTRKY
- Start date
randallc79
n3wb
I had this same problem... that's why I moved my AI to a dedicated box and as well as using this nVidia tesla p4.
randallc79
n3wb
So, far since making that last change on the AI side... this has been the only error I've seen pop up now, randomly...
Code:
14:00:14:Response received (...222274)
14:00:14:Request 'detect' dequeued for processing (...2fd08c)
14:00:14:Object Detection (Net): Command completed in 1968 ms.
14:00:14:Client request 'recognize' in the queue (...230a68)
14:00:14:Request 'recognize' dequeued for processing (...230a68)
14:00:14:Client request 'detect' in the queue (...0b9cff)
14:00:14:Face Processing: Retrieved face_queue command
14:00:14:Face Processing: An Error occured during processing: Traceback (most recent call last):
14:00:14:Response received (...230a68)
14:00:14:Face Processing: Queue and Processing Face Processing command 'recognize' (...230a68) took 174ms
Village Guy
Pulling my weight
BI Face processing is pretty much a demo facility at this time. It is unreliable at best. Try running with face recognition turned off.
randallc79
n3wb
Have you by chance heard anything back from CP AI about this issue?
I'm getting the same thing and have tried upgrading my GPU drivers. Anyone know how to downgrade back to deepstack? I tried reinstalling it but I don't see an option to switch it back in BI
Unless something changed, BI is simply providing images to the AI program and taking the results and putting it back into BI, so whether you have DS or CodeProject, it is pointing to the same area.
Now with that said, if it hasn't completely broke already, at some point all of the issues to get CodeProject working will eventually (if it hasn't yet) break functionality with Deepstack.
That is why many of us have not updated BI to beyond when Deepstack was still the only player in that arena.
So if you are not able to get DS running in your current version, consider rolling back to the update prior to when CodeProject came.
Now with that said, if it hasn't completely broke already, at some point all of the issues to get CodeProject working will eventually (if it hasn't yet) break functionality with Deepstack.
That is why many of us have not updated BI to beyond when Deepstack was still the only player in that arena.
So if you are not able to get DS running in your current version, consider rolling back to the update prior to when CodeProject came.
I have done a little more investigating and logging on the 2 BI systems that I have running, My remote system running 5 cameras, using a GTX 1650 show zero error:500 over the course of time since the last update. My local Server running 9 cameras, and using a RTX2060 12GB shows a total of 3 Error:500 instances over the same time period. I am not sure what is causing it but it does not seem as major of a problem as it had first seemed. I tried shooting support another message this morning hopefully they reply to this one. I also sent the boys over at codeproject a message to see what the progress was on fixing this issue.
The puzzling thing to me is why one system shows symptoms and the other one does not. I will have to look at configurations side by side and see if there is something I missed.
The puzzling thing to me is why one system shows symptoms and the other one does not. I will have to look at configurations side by side and see if there is something I missed.
It just shows how difficult it can be to write a program to be used on sooo many different systems and configurations.
From my observations (since I am holding tight on Deepstack and BI version prior to CodeProject) that many of the issues seem to be GPU related.
While some have had no problem with Code Project, many have nothing but trouble.
We didn't see this kind of trouble with people starting Deepstack before the change was made.
From my observations (since I am holding tight on Deepstack and BI version prior to CodeProject) that many of the issues seem to be GPU related.
While some have had no problem with Code Project, many have nothing but trouble.
We didn't see this kind of trouble with people starting Deepstack before the change was made.
CP worked fine until the last update of CP (could be a combo of BI and CP updates). There seems to be no progress in CP development over the last several weeks. When I initially moved from DS to CP, CP was superior. I’m hopeful an update will come soon to resolve.
I agree completely. You can save yourself a lot of headaches by simply upgrading your BI pc for a couple hundred bucks and forget the GPU. If you are running 10 cams or less on AI I really see no need to use GPU. If you really want to go the GPU route, then do some research and ask what card is working the best. I still run a I7-6700k in CPU mode and have had zero errors, the only time AI restarts is when I update BI (which is always the latest, sometimes not the greatest)
I agree completely. You can save yourself a lot of headaches by simply upgrading your BI pc for a couple hundred bucks and forget the GPU. If you are running 10 cams or less on AI I really see no need to use GPU. If you really want to go the GPU route, then do some research and ask what card is working the best. I still run a I7-6700k in CPU mode and have had zero errors, the only time AI restarts is when I update BI (which is always the latest, sometimes not the greatest)
View attachment 158068
Yep and can probably find a more powerful computer cheaper than a GPU!
I agree completely. You can save yourself a lot of headaches by simply upgrading your BI pc for a couple hundred bucks and forget the GPU. If you are running 10 cams or less on AI I really see no need to use GPU. If you really want to go the GPU route, then do some research and ask what card is working the best. I still run a I7-6700k in CPU mode and have had zero errors, the only time AI restarts is when I update BI (which is always the latest, sometimes not the greatest)
View attachment 158068
I agree the cpu way is alot more stable. However the gpu is a hair faster and with the amount of detections that I am getting at both locations the gpu makes more sense. Since the last update the other day I have more than 4 million analyzed images just on one of my machines. And have only popped 3 Error:500 codes. The other machine has analyzed just north of 3 million images and shows no errors using the gpu.
However, depending on the resolutions of the cameras being used it is fairly easy to tax even a higher end cpu with just the transcoding and decoding of the video streams. I use dual streams on my 9 camera system(5820K @ 4.4ghz all core, 32GB ram, RTX 2060 12GB) and it is not uncommon to see the cpu spike up the 100% usage for short durations if there is alot of motion going on. I believe that this is mainly due to the decoding of the video for analysis purposes.
The thing that remains puzzling to me is that a 1650 with 4GB of ram shows no errors what so ever. So I dont think that it is strictly a video card problem or a vram problem anyway.
It is however based on use case and what you need the machine to do as far as analysis goes. If you are trying to capture fast moving objects such as cars then you need to send more frames to be analyzed at a faster pace. I have not broken into ALPR but I am sure that requires even more images to be sent to get a correct detection. With more images being analyzed in a short duration the gpu makes more sense to me. Analysis times for CPU on the above system average 150-175ms on the GPU in the same system they are averaging 60-75ms.
In my case the gpu made sense as I was able to get them on a good deal used before the GPU prices hit the fan. For a new setup I believe that for now the cpu only approach would be cheaper.
I am going to drop the main rig back to cpu analysis for now and see how the cpu usage is compared to the gpu usage just to get away from the errors. (BI sends me a notification when I get errors)
**Edit well that did not take long Multiple Error:200 (No Error:500 though
). Detection times are running on Average 550-650ms on the same machine and CPU pings 100% usage for the duration of the analysis. Back to GPU lol.
Last edited:
randallc79
n3wb
I'm running about 18 cams and have a dedicated server for BI and one for AI (with direct NIC link between them). The whole point was to spread out the CPU load because it was really hammering my main BI system with the AI installed on it, while also utilizing GPU on dedicated AI server. So after these latest comments, I went to the AI server and switched, "Face Processing" to CPU and Enabled Object Detection (YOLOv5.NET). It's spiked my CPU usage on my AI server, but this has been the longest I haven't seen any 500 errors.
Just a little update and a bump for this thread. It appears that this was a codeproject problem I am testing out the 2.1 Beta for them at the current moment and it has resolved all of my Error:500 Problems. I am not sure when they plan to release this version out in the wild as it is still in the testing phase but I will try and keep you posted. Just know there is a fix for it coming down the pipe. 
randallc79
n3wb
Just a little update and a bump for this thread. It appears that this was a codeproject problem I am testing out the 2.1 Beta for them at the current moment and it has resolved all of my Error:500 Problems. I am not sure when they plan to release this version out in the wild as it is still in the testing phase but I will try and keep you posted. Just know there is a fix for it coming down the pipe.
Thats awesome! Thanks for the update.
Currently 6+ million images analyzed on the test machine and still no Error:500 or Error:200.. Hopefully the release is soon but overall I am happy with the results.