Running CPAI on OrangePi/RK3588--RKNN modules are "not available"

[PeteJ]

Newbie here, but I've googled for hours but have not found much success.

I currently run BI and CPAI on the same server, and it works ok. The server does not have a GPU and it takes 300-700ms for AI to run, and this is on only a few cameras. The server is a 11th gen Intel i5, 16G of ram. For comparison, I run Frigate on Linux with a Coral TPU and inference times are sub 10ms, on the same server, so I'm looking for a way to speed things up w/o a GPU.

I want to split the CPAI workload off to an OrangePI 5 Max to see if performance would improve.. but I can't figure out how to get the RKNN modules to install since they are currently listed as "Not Available" on the CPAI server dashboard.

The Orange Pi runs the official Ubuntu distribution from Orange Pi (22.04.5 LTS, 6.1.43 kernel), I am running CPAI in Docker. CPAI is 2.9.7. I am running the container in privileged mode, with /dev/bus/usb passed thru as devices. Within the container, I can see those devices, and Docker confirms the container is privileged. I am running the amr64 image.

my docker compose file:

services:
CodeProjectAI:
image: codeproject/ai-server:arm64
container_name: codeproject-ai-server-arm64
privileged: True
hostname: codeproject-ai-server
restart: unless-stopped
ports:
- "5000:32168"
environment:
- TZ=America/Los_Angeles
volumes:
- /dev/bus/usb:/dev/bus/usb
- codeproject_ai_data:/etc/codeproject/ai
- codeproject_ai_modules:/app/modules
devices:
- /dev/bus/usb:/dev/bus/usb
volumes:
codeproject_ai_data:
codeproject_ai_modules:


If you have CPAI running on RK3588, would love some advise on how to get this going.

Thanks!

 

[wittaj]

Before you go to all that effort, unless you like to tinker, what is the benefit?

Does improving sub 1-second process times result in any additional improvements to your response to a situation? 300ms is faster than the reaction time of most people.

If it were 15,000ms, then yeah giving a perp a 15 second head start on you is bad, but sub 1 second, whether it is 300ms to 5ms, that is human reaction time differences that you might be chasing something that doesn't add any value.

 

[PeteJ]

Yes, sort of.

I do not currently have plans to act on the data, just collecting it is sufficient. I only have 1 camera doing LPR currently and it's point out to a very quiet street. I'd be surprised if I would have more than 30 reads per day. However, where I plan to install this is going to be toward a main street where I'd be in the 1000 reads per day range, in a single direction (that's just a guess, but it is a busy street).

I don't think BI and CPAI running on the same server can keep up, so that's why I am interesting in splitting it up. But, I am not really clear on how BI pipelines the LPR to CPAI--does it just queue it up in a FIFO queue and will process each one as it waits for CPAI to return the results, or does it overflow or start blocking if the requests are not returned quick enough?

From what I have seen, after about 8-10 secs, it looks like BI stops checking? The *.dat data is all grayed out.

And yes, I do like to tinker.

 

[PeteJ]

Okay, looks like the problem was the official Ubuntu 22.04.5 LTS image from Orange Pi's site. Under platform, it comes up as Linux/Arm64--which is accurate, but looks like CPAI actually looks for OrangePi as the platform:



I was able to get this to work by installing Joshua Reik's Ubuntu image: GitHub - Joshua-Riek/ubuntu-rockchip: Ubuntu for Rockchip RK35XX Devices

Once installed, the platform was detected properly and the option to install the RNKK modules was available.

Thanks very much to @MikeLud1 for pointing me in the right direction. Hopefully this info would be useful to someone else in the future.

 

[PeteJ]

In case anyone is considering running CPAI on the RK3588, this is what I am seeing in terms of perf on the RK3588



Below is on CPU.



CPAI avg is slightly under 40ms running average.



I only have a 15w power supply on it, so it's probably running at half of that right now.

 

[AlwaysSomething]

I'm interested in giving this a try but not really familiar with using Docker (I understand the concept just never used it). Did you try without Docker? Reason you used Docker?

Your numbers seem really good. Any negatives or complaints?

 

[PeteJ]

I'm interested in giving this a try but not really familiar with using Docker (I understand the concept just never used it). Did you try without Docker? Reason you used Docker?

Your numbers seem really good. Any negatives or complaints?

No complaints, it's running very well.

Don't let Docker scare you, it's not that hard to learn, and you can install something like Portainer to manage it via a web interface.

 

[AlwaysSomething]

Which of Joshua's image did you use? I see he has 24.04 as the latest but I thought when I read his notes it sounded like there were still errors with that version. He lists 22.04 as deprecated.

Also curious, did you install the desktop version or just server?

I appreciate the help. SBC is all new to me and I'm still a Linux newb. Just realized I bought the Ultra and not Max. Only difference is one HDMI port is IN and not OUT so hopefully I can still use the Max image.

 

[PeteJ]

Which of Joshua's image did you use? I see he has 24.04 as the latest but I thought when I read his notes it sounded like there were still errors with that version. He lists 22.04 as deprecated.

Also curious, did you install the desktop version or just server?

I appreciate the help. SBC is all new to me and I'm still a Linux newb. Just realized I bought the Ultra and not Max. Only difference is one HDMI port is IN and not OUT so hopefully I can still use the Max image.

I'm running 24.04, yeah, it should work fine with the Ultra. I'm running the desktop version, I think the only difference is that the server doesnt come with some apps and X.

 

[AlwaysSomething]

Thanks for info.

So I ended up trying both desktop versions and neither worked for me. They would load and I got the background images but that was it. I noticed if I moved my mouse all the way up, down, or right, I hit the edge of the screen. If I moved the mouse all the way to the left, it was running off the screen (probably where the icons/login was). I'm guessing there is a config different between the MAX and the ULTRA. Bummer

I just ordered a MAX with 16 GB. Pretty crazy this thing idles at 3 watts!!

EDIT: I did try right clicking to see if I could get the display properties but nothing happened.

 

[PeteJ]

Thanks for info.

So I ended up trying both desktop versions and neither worked for me. They would load and I got the background images but that was it. I noticed if I moved my mouse all the way up, down, or right, I hit the edge of the screen. If I moved the mouse all the way to the left, it was running off the screen (probably where the icons/login was). I'm guessing there is a config different between the MAX and the ULTRA. Bummer

I just ordered a MAX with 16 GB. Pretty crazy this thing idles at 3 watts!!

EDIT: I did try right clicking to see if I could get the display properties but nothing happened.

Yeah. that's a little weird, I didn't have any issues installing the image on the MAX. Maybe try a different mouse or monitor to see if it makes a difference? There is a few youtube videos that go over the install, depending if you are booting from microSD card or nvme.

Update:

I found my notes:

To install it on a nvme drive:

sudo fdisk -l
sudo ubuntu-rockchip-install /dev/nvme0n1
sudo u-boot-install-mtd

The Orange Pis seem to be sensitive to power supply issues, when I got my first one, it would crash from time to time. Nothing in the logs, just hard lockup.

I install a few tools to help debug:

sudo apt install htop
sudo apt install sensors
sudo apt install stress

watch sudo stress --cpu 4 --io 3 --vm 2 --vm-bytes 256M --timeout 600s

This caused it to crash. Swapped the USB-C PSU to a better one, and the problem went away

I increased the timeout to 2h once it was stable and used that as a sort of a burn in.

sensors will give you the board temps to make sure it doesn't get too warm.

Hope this helps.