Sharing setups with CodeProject AI on Coral hardware (Spring 2024)

[koops]

You should be always able to use the multi-TPU implementation

Is there anyway to be able to tell the difference between the two?

I do see the multi_tpu environment variable

21:52:10:CPAI_CORAL_MULTI_TPU = True

My start up logs show the attemping and then fall back messages.

21:52:12objectdetection_coral_adapter.py: TPU detected
21:52:12objectdetection_coral_adapter.py: Attempting multi-TPU initialisation
21:52:12 objectdetection_coral_adapter.py: Failed to init multi-TPU. Falling back to single TPU.
21:52:12 objectdetection_coral_adapter.py: Using Edge TPU

 

[mailseth]

It looks like you are attempting to use multi-TPU by default, but something is wrong so it falls back to the old method. If you’re feeling adventurous you could dig into it and figure out what’s wrong and let me know what to change. (But it may be just the installed coral library version, which is harder to fix.)

 

[AlwaysSomething]

I'm months behind on current projects so don't want to slide down the rabbit hole on a new adventure.

But should I trip into one as I'm walking around, what is the best setup? Mind you, I haven't coded in over a decade (besides a couple of small VBscript or javascript files to help someone automate a task). I used to use MS Visual Studio on Windows back in the day but I know that's OLD and have changed dramatically. I'm not opposed to Linux even though it would be more of a learning curve for me (I mean I would have already tripped into the rabbit hole right ).

Are most developers using Windows or Linux? Also, what IDE are they using?

 

[mailseth]

The Coral TPU code is all in python, although some of the failures may be related to the setup configuration/environment. There really isn’t any hardcore coding that you’d need to do. Just figure out how to make the existing code run more reliably via trial and error. Things like refining the logic to ensure the watchdog is working correctly and recovering from errors

CodeProject.AI-Server/src/modules/ObjectDetectionCoral/tpu_runner.py at rem_runner_loops · seth-planet/CodeProject.AI-Server

CodeProject SenseAI is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI. - seth-planet/C...

github.com

Or logic like this:

CodeProject.AI-Server/src/modules/ObjectDetectionCoral/tpu_runner.py at rem_runner_loops · seth-planet/CodeProject.AI-Server

CodeProject SenseAI is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI. - seth-planet/C...

github.com

Or make sure that the model segment download or module install code is working well, but I didn’t write that so I’m less familiar.

Edit: and I’d just focus on making sure it runs well on your setup first.

 

[mailseth]

As far as an IDE, I’ve used Eclipse, BBEdit, Sublime, and most recently VS Code. My Linux machine that I’ve done my Coral development on I’m using Sublime and the default Linux text editor (simple text?). And EMacs sometimes. Or vi if I have to. And sometimes I’ll work on some prototype code using the GitHub editing interface. Anyway, it depends.

 

[AlwaysSomething]

Yeah I think most of the issues are with the installer and not the actual execution/"running".

I wan't sure how the debugging works. I used to be able to put break points and step through code.

I've been hesitating since the only problem I really have is it says it's using the model I chose but I don't think it really is (based on testing). However, it is a pain point since I only really care about people on certain cameras and it is currently marking a lot of cars as people for some reason. In other words, a car will drive by and it gets marked as a person.

If I fall in the rabbit hole you may hear me screaming for help.

 

[mailseth]

I’ve been doing this for many years, but still prefer print statements over breakpoints for whatever reason. I’ve been doing all of my testing using a script to run and test the tpu_runner.py code, so I’m not really even that familiar with how things interact within CPAI, and I may not be of much use.

 

[silencery]

Software configuration

• CP.AI version: 2.6.2
• Modules: Object Detection (Coral) v2.2.2
• Model: Medium yolov5. Note: my BI configuration says "Default Object Detection : small". Now I manually changed the model from small to medium within the codeproject control panel. I likely probably should have changed this from within BI itself.

Other
I didn't investigate this enough to realize that that there could be issues with single TPU implementations. Specifically the error "Unable to run inference: There is at least 1 reference to internal data". From googling this I think it means that TPU is already busy and is unable to process the current request.
Checked the codeproject AI log window and noticed what appeared to be near constant analysis.
Reduced this by checking my BI settings and seeing that I had "Static object analysis" turned on. This reduced the issue significantly. I'd guess that the dual TPU implementations do not see this nearly as much?

Great share and awesome feedback. I'll add in your findings about BI config to the top post. Good note that static object analysis should be disabled to prevent flooding requests. In my case, I'm not having BI control CPAI, so I wasn't able to adjust the model size from there. Good reminder about that too.

 

[koops]

Good note that static object analysis should be disabled to prevent flooding requests. In my case, I'm not having BI control CPAI, so I wasn't able to adjust the model size from there. Good reminder about that too.

This is documented in the codeproject AI blue iris faq here : Blue Iris Webcam Software - CodeProject.AI Server v2.6.2 under the section marked "CodeProject.AI Server log shows requests every minute or less when there is no motion detection"

 

[AlwaysSomething]

I think static object analysis is dependent on the user/camera needs.

For example, I have a couple of cameras that cover my driveway which can have a couple of cars parked in it. On those cameras I am tracking person, car, truck, cat, and dog. Without the static object analysis if there were was motion, like a bug flying across the camera, it would detect my parked cars and trigger an alert on the cars. So in that case I have detecting the static objects turned on to prevent false positives. This was a real PITA with some previous versions of BI where it wasn't working correctly. You may also want this if you have a camera in a garage with a parked car that may have motion such as a bug (especially if using IR which can attract the bugs).

Conversely, on my backyard and indoor cameras I do not have any static objects that would trigger so I do not have it turned on for those cameras. Oddly enough, there was a stuffed dog in my living room once that would trigger when the shadows caused motion (large window with lots of sun). So again, you have to evaluate based on your camera's situation.

One last thought is just how many cameras are configured for static object analysis. If only a couple (like in my case) you shouldn't be flooding CPAI. If you have a lot of cameras configured (like a parking garage) then you will have a lot traffic for CPAI. I would say turn off and see how many false positives you get to determine if you need it.

 

[silencery]

This is documented in the codeproject AI blue iris faq here : Blue Iris Webcam Software - CodeProject.AI Server v2.6.2 under the section marked "CodeProject.AI Server log shows requests every minute or less when there is no motion detection"

LOL, oops. I didn't even know a FAQ existed! Added this reminder to the top post in case anyone is lost in the future.

I think static object analysis is dependent on the user/camera needs.

100% Everyone's environment is different, so it's all about tuning things to fit your needs

 

[AlwaysSomething]

The Coral TPU code is all in python, although some of the failures may be related to the setup configuration/environment. There really isn’t any hardcore coding that you’d need to do. Just figure out how to make the existing code run more reliably via trial and error. Things like refining the logic to ensure the watchdog is working correctly and recovering from errors

CodeProject.AI-Server/src/modules/ObjectDetectionCoral/tpu_runner.py at rem_runner_loops · seth-planet/CodeProject.AI-Server

CodeProject SenseAI is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI. - seth-planet/C...

github.com

Or logic like this:

CodeProject.AI-Server/src/modules/ObjectDetectionCoral/tpu_runner.py at rem_runner_loops · seth-planet/CodeProject.AI-Server

CodeProject SenseAI is a self contained service that software developers can include in, and distribute with, their applications in order to augment their apps with the power of AI. - seth-planet/C...

github.com

Or make sure that the model segment download or module install code is working well, but I didn’t write that so I’m less familiar.

Edit: and I’d just focus on making sure it runs well on your setup first.

@mailseth

Couple of quick questions.

1. Is the installer part of the source code that can be downloaded from the CPAI website? It seems like most of the problems are usually with the installer/installation. I upgraded to 2.6.5 and having issues with it not saving the settings to autostart, reverting to CPU, and some other errors I can't remember off the top of my head. The CORAL module works fine once I can get it running.

2. I know the TPU is limited on memory (and therefore the model it can hold) but are there any plans to make it work for the LPR module if there is a prerequisite that you need multiple dedicated TPUs? For example, you need n number of TPUs just for the LPR module. If you want to use other TPUs for other modules you need to install them on a separate machine. For example using VMs, have one VM for regular Object detection with TPUs passed through and then a second VM for LPR with separate TPUs passed through.

Just curious as I want to use LPR in the very near future and I would need a GPU for that based on the current requirements. If I have to go the GPU route then I may need to build a new rig since I can't really fit any GPU's in my Dell Optiplex SFF or EliteDesk. I need to determine if I bite the bullet now or do I wait and then maybe have to migrate everything later to new machine.

Thanks

 

[mailseth]

I'm afraid that the module file management & installation stuff is all outside of what I've worked on. I've looked into it before, but haven't made sense of it, unfortunately. I just send Chris a zip file of the compiled models and he has been setting things up to fit in the larger CPAI system.

I don't think that LPR will ever fully run on a TPU because the logic is more complicated and TPUs work best running a single model (not swapping between many models). So it would work to port the LPR over, but it may still not be the best use of TPU resources.

I've been working on a custom model that I think will make a good general use case for security cameras. It includes everything from ipcam-combined, but also includes classes like Fire that I'm going to find useful for my future projects. I've also included classes for Vehicle registration plate and Human face. My intention is that this single model can be run on the TPU, and then someone can allocate work as required to other models running on another device like the CPU or GPU.

I expect that most of the LPR compute time is spent in running plate localization to get a bounding box around the plate, and then later OCR is run only if it finds a car & plate. So my hope is that the TPU is able to do the heavy lifting for running the common-case model, but this is something that has absolutely no support besides having support in a model that I'm still working on training. I don't know what the BI/CPAI logic would look like.

 

[koops]

I've found that the coral ai small model isn't as accurate for my use case (people detection) as what it was before when I was using the ipcam-general.

I found that I had significantly more false positive triggers than ipcam-general so i've had to increase confidence right up (70%) and number of images (9) and i still miss triggers.
So better cpu utilization overall but not quite a 1:1 replacement for what I had ... yet.

 

[mailseth]

Yeah, there are a few available models for Coral, but they all need to be quantized to 8-bits, which reduces accuracy. Hopefully we can get the IPcam-general Coral model released soon.

 

[headpiece]

@mailseth thanks for your work on this project. Im struggling to understand whether MikeLud ipcam-general.tflite model can be used with CPAI?

If yes,

• what is the API call to invoke it?
• in what coral directory should it be stored?

Thanks again

 

[mailseth]

The short answer is no, that model needs to be reworked to run on Coral. The long answer is yes, I’ve done so and you can download it here:

all_segments_ipcam-general-v8_edgetpu.tflite · mailseth/coral at main

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

Unfortunately, I don’t have a good guide for getting it running in the current version of CPAI. You’re welcome to download and play with the model, however. Or any model in that repo, like YOLOv9. If you’re feeling ambitious with some code, I’d love to get someone submitting some GitHub pull requests against my current code that’s been in a state of purgatory since about June.

 

[mailseth]

My current code is fast, but not functional in CPAI. It needs to start downloading models correctly from the hugging face repo I mentioned above.

GitHub - seth-planet/CodeProject.AI-ObjectDetectionCoral at fix_single_fallback

Contribute to seth-planet/CodeProject.AI-ObjectDetectionCoral development by creating an account on GitHub.

github.com

 

[Ryushin]

Nice to find this thread. I just installed my coral pieces last night:

M.2 Accelerator with Dual Edge TPU | Coral

Integrate two Edge TPUs into legacy and new systems using an M.2 (E key) interface.

coral.ai

Dual Edge TPU Adapter - PCIe x1 Low Profile

Dual Edge TPU Adapter is designed for Coral m.2 Accelerator with Dual Edge TPU to be used on a system with PCIe x1 slot available.

www.makerfabs.com

I see two Coral PCIe Accelerator adapters in Device Manager under Windows and I'm running CPAI 2.9.5.

It won't initialize the second TPU though: Starting Coral gives me this error:
06:35:32bjectdetection_coral_adapter.py: ERROR:root:TFLite file C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\assets\yolov5l-int8_edgetpu.tflite doesn't exist
06:35:32bjectdetection_coral_adapter.py: WARNING:root:Model file not found: [Errno 2] No such file or directory: 'C:\\Program Files\\CodeProject\\AI\\modules\\ObjectDetectionCoral\\assets\\yolov5l-int8_edgetpu.tflite'
06:35:32bjectdetection_coral_adapter.py: WARNING:root:No Coral TPUs found or able to be initialized. Using CPU.
06:35:32bjectdetection_coral_adapter.py: ERROR:root:TFLite file C:\Program Files\CodeProject\AI\modules\ObjectDetectionCoral\assets\yolov5l-int8.tflite doesn't exist
06:35:32bjectdetection_coral_adapter.py: WARNING:root:Unable to create interpreter for CPU using edgeTPU library: [Errno 2] No such file or directory: 'C:\\Program Files\\CodeProject\\AI\\modules\\ObjectDetectionCoral\\assets\\yolov5l-int8.tflite'
06:35:32bjectdetection_coral_adapter.py: Using PIL for image manipulation (Either OpenCV or numpy not available for this module)
06:35:32bjectdetection_coral_adapter.py: TPU detected
06:35:32bjectdetection_coral_adapter.py: Attempting multi-TPU initialisation
06:35:32bjectdetection_coral_adapter.py: Failed to init multi-TPU. Falling back to single TPU.
06:35:32bjectdetection_coral_adapter.py: Using Edge TPU

Switching to single TPU results in no startup errors.
06:39:49:Started Object Detection (Coral) module
06:39:51bjectdetection_coral_adapter.py: Using PIL for image manipulation (Either OpenCV or numpy not available for this module)
06:39:51bjectdetection_coral_adapter.py: TPU detected
06:39:51bjectdetection_coral_adapter.py: Using Edge TPU

Reading this thread @koops had this problem as well. Is there any solution for it to get both TPUs working?

I did some testing and tuning last night after getting it installed. Set it up to use the YOLOv5 model at medium model size. At first I though it was broken because it was not detecting anything. I had minimum confidence set to 90% to prevent false positives when I was using CPU for object detection. Once I dropped it to 60% is started detecting. But it rare that it would ever get into 70%range. Speed was good at 35-45ms though. Compared to the CPU which was over 1000ms most of the time. Saw quite a few false positives though detecting pine trees as people. After a bit more tuning, I tried the large model size and the detection confidence rose into the 80's% range. It also no longer detecting the pine trees as people even though the minimum confidence was set to 50%. Downside is it now takes 210-225ms to detect. Compared to the 1000+ms before with using the CPU, it's a huge improvement. The CPU usage of the server itself is also much better running about 20-50% instead of 100% during detection. The CPU is still doing face and LPR processing.

Other things I've noticed with using Coral compared to CPU. Coral has a harder time detecting cats and dogs. Coral can't detect objects as far away as the CPU. My Dahua PTZ AI cameras have 25x zoom so that isn't so much a problem if I'm trying to detect a person or vehicle as it will zoom in on those.

So overall, I'm very happy with the Coral performance improvement. Now to just get that second TPU to work and I can call it done.

Any thoughts on the YOLOv5 vs YOLOv8 models? Both seemed to work but I didn't see any better detection between the two.