BlueIris and CPAI on Proxmox

[AlwaysSomething]

The short version:

How are people using Proxmox to host BI (on Windows VM) and CPAI (windows VM or Linux VM)?
How are you performing backups/upgrades?
Are there any issues or pain points to share?
Preference over bare metal install?

The long version:

I want to upgrade to newest version of BI and CAPI but the last 2 times I did an upgrade I spent many hours tweaking/testing/troubleshooting. I don't want to call them problems because sometimes it was new functionality which is why I upgraded to begin with. Things need tweaking and/or learning curves by me (I'm fairly new to all of this especially CPAI). Another example that was not a problem was me getting a Coral TPU and testing different CPAI models, model sizes as well as CPU vs GPU vs TPU.

Bottom line is during that time (which again could be days) I don't want to lose/miss any potential videos/alerts.

That being said I want to configure a second PC for BI. I want to have a "stable version" PC and "latest version" PC (or Production and Development PCs). I want to be able to upgrade and test in DEV (for at least a few days so I get good day and night scenarios) before I upgrade my PROD PC. Obviously I'll need at least 2 licenses and twice the hardware, that's OK (I'll be repurposing older hardware).

I've seen a lot of people here running on Proxmox or ESXI and had me thinking that might be better than bare metal because a few times things got so screwed up I had to reinstall everything including the OS. The reason for the OS I think it was because one time I was having issues with drivers for the iGPU and a second time it got an update somehow and screwed things up. This means I was down without cameras and had to spend the time to get it up and running ASAP. Another more relevant example is with testing CPAI with different models and sizes. I don't want to miss events while testing which did happen a couple of times. I record continuously so usually never lose footage but may lose the alerts.

I'm thinking of using Proxmox to host a Windows VM for BI and a Linux VM with docker for CPAI. Probably going to use Ubuntu but open to suggestions/recommendations. The reason to go with the CPAI docker is easier to wipe and create since we need to keep uninstalling previous version and deleting all the file/folders anyway. The PC will still be dedicated for security camera and nothing else (no other VMs or overhead).

Let's assume everything is up and running and stable. Now let's say I want to upgrade BI, I would clone the Windows VM (so I can keep my current version in case I need to revert) and then perform the upgrade on the clone and test. If something goes majorly wrong I can wipe the clone and start over from the original VM. If all goes well (and tested for a few days) the clone is now the latest version. I can then delete the original VM (or keep for xx amount of time and then delete). I know I can use snapshots but I think the clone would be more reliable and better performing especially if testing over many days. I think I actually read that in the Proxmox docs but I could be wrong. I obviously would shutdown the original VM while testing the clone. Once tested and approved, I can migrate the VM from the DEV PC to the PROD PC. Optionally I can swap which is the DEV and PROD PC.

I know there will be overhead of using Proxmox itself but I'm thinking it should be minimal. I will hardware pass the TPU and can even pass the hard drives if need be (separate SATA controller).

For some this may sound like a lot but for others hopefully this make sense. I just want to ensure I'm not losing any footage when I may need it the most (Murphy's law). Unfortunately it did happen to me once but thankfully it wasn't critical (truck was broken into but nothing was stolen - they wanted the truck and not the contents).

Anyway, I can add more details/steps but hopefully that's enough to follow.

So that said, is anyone else doing this or something similar? I'm choosing Proxmox over free EXSI based on learning curve as well as recommendations elsewhere.

For those using Proxmox are you doing something similar? Any paint points?

I know I can just use 1 PC and run both environments there but my hardware is a few years old and I don't think it can handle DEV and PROD on same PC. Again, I need them both running so I don't lose anything and in some cases for comparison (e.g.: comparing CPAI performance)

More details in case anyone wants to know:

Current PC dedicated to BI (5.8.4.5 x64) and CPAI (2.5.1) both running in Windows 10. PC is a Dell Optiplex 7060 with following specs: i5-8500, 16 GB ram, OS and BI DB on NVME and 1 WD purple HD for video. It also has a Coral TPU (going to add a dual TPU once it arrives).

Proposed 2nd PC is a HP EliteDesk 800 G4: i7-8700, 16 GB ram with more drives as well and also it's own TPUs.

I also have a custom built i7-9700k, 64 GB ram PC if need be. If absolutely needed I can build a new low budget PC but that is last resorts. Main constraint is they need to be low power (so no 24 core LOL) and don't want to spend a lot of money.

Thanks for any feedback in advance.

 

[Hetticles]

Yea BI on Proxmox is great.
Instead of your clone/snapshot or Dev/Prod, consider using Proxmox Backup Server (PBS). When I make a change that screws things up, I am able to roll back to a backup (got it setup to backup weekly). You can also do a backup on demand. This also gives you backups in case your promox hardware dies. I would recommend running PBS on other, cheap, low power hardware.
This eliminates the whole dev/prod thing, just make the change, if its bad, roll back. No need for multiple licenses.

I also run CPAI on a Linux VM on the same proxmox host. I started doing this after upgrading CPAI on the shared BI/CPAI box was causing me problems. I do the same routine. Backup the linux VM, make a change. If it broke something? Rollback to the backup. Try again.

You'll need way more RAM to run concurrent machines. Looks like the Optiplex 7060 can support 64 gb max, that should be good enough.
If you current CPU is good enough, with virtualization it will LIKELY good enough.

What is your Coral TPU connected with? "device passthrough" with proxmox can be a bit tricky. USB passthrough generally works ok, PCIE passthrough is a lot trickier and depends on your CPU AND Motherboard. Something to google.

 

[AlwaysSomething]

Thanks!! I like your approach better than the cloning. I just started looking into the Proxmox Backup Server (PBS). I have an older Dell (3040 I think) that I can use for the PBS. From my readings, having the PBS makes it easier to migrate to a new server if you wish to upgrade the PC. If that is true that is also awesome.

Are you running CPAI directly in Linux or in a container in Linux? Originally I was thinking the container but I've read people doing both ways. I'm wondering if those using containers were running other containers as well. I still want this PC to be dedicated to BI and CPAI. With the PBS it probably wouldn't matter anymore unless I want to save on disk space.

I was wondering about the RAM. I was hoping 16 GB might be enough and was thinking I may need to go to 32 GB. I didn't think I would need 64 GB but I'll see how it goes.

I may still get a 2nd PC up since I still like to test/tinker. Having the 2 side by side makes it easier to compare. But that is low priority.

 

[AlwaysSomething]

I forgot to mention that the TPU is connected via PCIe. I've seen posts on other sites where people were passing the GPU and it didn't seem too bad. Maybe those were best case scenarios.

 

[Hetticles]

yeah PBS makes backup/restore to different hosts a breeze.

For PCIE passthrough, look in your bios, you'll need to see if you have an IOMMU. Its called a bunch of different things with different vendors, you'll have to look around. A lot of the Proxmox documentation is outdated, I didn't have to fiddle kernel modules or change kernel args, just enabled iommu and was able to passthrough with the GUI no problem.

I run a linux vm rather than a container. I'll be honest I dont understand the advantage of a container, I guess install is easier if you can grab some off the shelf compose file. Installing CPAI was also pretty easy, idk.

At one site I have 15 cameras, BI VM uses 9gb ram, CPAI (no hw accel) VM uses ~5gb. Not a lot of headroom with 16gb but might be fine. you'll find out

 

[silencery]

I just made this migration several months ago. I will echo everything @Hetticles wrote above. I'll also add that Proxmox is great and really easy to get started, but making sure everything runs optimally can require a lot of time. Some things to consider for planning:

• You can choose "fat" (full VM) or "slim" (LXC containers) virtualization for your clients. Read up on the pros and cons of each method since you can really get into the weeds if you wanna set it up optimally for your situation.
• LXCs are usually preferred since they are much more lightweight, easy to deploy, and proxmox can handle resource sharing more efficiently. This is ideal if you need to share something like a GPU across multiple hosts. However, an LXC can only be a linux variant. Docker inside an LXC is not officially supported, but it works.
• VMs must be used if you need windows. Also, any hardware passthrough for a VM can only be dedicated for that one host.
• This can be a little complex since CPAI can run on linux, but BI still needs windows. Your final deployment architecture will depend on how you want your hardware (GPU, storage) to be accessed.
• In my case, when I migrated, I was using a regular storage disk and iGPU. Since I need the iGPU for everything, I wound up installing BI and CPAI on the same windows VM.
• For now, I'm locked into putting anything that needs the GPU (transcoding, AI, etc) on windows. I'm in the middle of deciding if i want a dedicated video card so I can share it to other LXCs on the host. If I ever go that route, I would likely move CPAI to its own container.

 

[AlwaysSomething]

yeah PBS makes backup/restore to different hosts a breeze.

For PCIE passthrough, look in your bios, you'll need to see if you have an IOMMU. Its called a bunch of different things with different vendors, you'll have to look around. A lot of the Proxmox documentation is outdated, I didn't have to fiddle kernel modules or change kernel args, just enabled iommu and was able to passthrough with the GUI no problem.

I run a linux vm rather than a container. I'll be honest I dont understand the advantage of a container, I guess install is easier if you can grab some off the shelf compose file. Installing CPAI was also pretty easy, idk.

At one site I have 15 cameras, BI VM uses 9gb ram, CPAI (no hw accel) VM uses ~5gb. Not a lot of headroom with 16gb but might be fine. you'll find out

Yeah I found this page regarding the IOMMU:

No IOMMU detected in a IOMMU-capable system

Hello I have installed Proxmox VE on an HP EliteDesk 800 G3 CPU: Intel i5-6500 65W Chipset: Intel Q270 As per intel specs (available at https://www.intel.com/content/www/us/en/products/sku/88184/intel-core-i56500-processor-6m-cache-up-to-3-60-ghz/specifications.html ) the CPU supports all this...

forum.proxmox.com

I plan on doing this on HP EliteDesk 800 G4 and that is the previous generation HP so hopefully the newer generation is capable (should be).

I think the advantage for the container is easier to grab the off the shelf file (more plug n play). I'll probably skip the container to keep it simple for now.

I'll find out on the RAM soon. Hoping to start this soon. Time flies when you are having fun. LOL

 

[AlwaysSomething]

I just made this migration several months ago. I will echo everything @Hetticles wrote above. I'll also add that Proxmox is great and really easy to get started, but making sure everything runs optimally can require a lot of time. Some things to consider for planning:

• You can choose "fat" (full VM) or "slim" (LXC containers) virtualization for your clients. Read up on the pros and cons of each method since you can really get into the weeds if you wanna set it up optimally for your situation.
• LXCs are usually preferred since they are much more lightweight, easy to deploy, and proxmox can handle resource sharing more efficiently. This is ideal if you need to share something like a GPU across multiple hosts. However, an LXC can only be a linux variant. Docker inside an LXC is not officially supported, but it works.
• VMs must be used if you need windows. Also, any hardware passthrough for a VM can only be dedicated for that one host.
• This can be a little complex since CPAI can run on linux, but BI still needs windows. Your final deployment architecture will depend on how you want your hardware (GPU, storage) to be accessed.
• In my case, when I migrated, I was using a regular storage disk and iGPU. Since I need the iGPU for everything, I wound up installing BI and CPAI on the same windows VM.
• For now, I'm locked into putting anything that needs the GPU (transcoding, AI, etc) on windows. I'm in the middle of deciding if i want a dedicated video card so I can share it to other LXCs on the host. If I ever go that route, I would likely move CPAI to its own container.

Interesting. I only have the iGPU but I was leaning towards putting CPAI into a full Linux VM and not shared with BI in the Windows VM. However, if I'm using a TPU for CPAI I might still be OK?

 

[silencery]

Yup. You can do a few combinations. All of which sound like they work for your use case. For example:

• One GPU for VM(A) and one TPU or GPU for VM(B)
• One GPU for VM(A) and one TPU or GPU shared for various LXCs

(On a separate, but related note, this is further complicated by advances in hardware such as SR-IOV. SR-IOV is hardware-level support to share one device to several VMs. It started out as only being available for enterprise grade hardware, but gradually its becoming available in the consumer space too. This means you could have an nvidia card which could be shared by multiple VMs, as long as the card and drivers support SR-IOV.)

 

[AlwaysSomething]

Just thought of another question I had that I forgot to ask. Currently I'm recording continuously to a WD purple drive. I remember reading that Proxmox is not great for intense IO and if doing a lot of IO then it's better to pass the drive or if you have an HD controller/SHB to pass that instead. I think the examples were using databases like mySQL and SQL Server but I would think this would be similar. Are you guys recording continuously? Is so, how are you setup? I have a cheap Sata controller I can add and pass through if need be.

As far as the PBS, do you recomment SSD or HD for the data? I have some old 1 TB WD HD I can repurpose for the data for now. If going SSD for the data I'll probably have buy to some since I don't think I have anything with a great capacity. I may have a couple of 120 GB but that probably won't be enough. Would use one of those for the OS at least.

I really appreciate the info. When reading other posts there wasn't much detail and most were just stating how much they virtualized versus the "how" and the "gotchas".

 

[Hetticles]

I wouldnt bother with HBA passthrough for one drive. This is what I did when I had a single drive.
People who have big arrays of drives connect to an HBA, then pass that HBA through to TrueNAS, then exporting the share to a windows VM for BI. For a single drive thats a lot of overhead.

For my 24/7 recording, its just huge linear writes to the disk. Database IO can be a lot more unpredictable and demanding.

Make sure you have Proxmox on a different drive than your BI storage.

PBS I do a HDD. More GB/$ for old spinning drives. Most PBS activities I have set to do overnight and it being slow has never bothered me. I also exclude my BI storage drive from the PBS backup.

 

[Hetticles]

I'll also add that Proxmox is great and really easy to get started, but making sure everything runs optimally can require a lot of time.

Earlier silencery mentioned this. Bears repeating with the HDD question. You will want to look at "IOThread" for your BI Storage drive. Part of tweaking for running optinally.

 

[silencery]

oh man, yeah, dizzying number of ways to configure storage. In my case, here's how I do it (pretty much what @Hetticles listed under option #2):

• I have an LXC client acting as an NFS/Samba server on the same physical proxmox machine
• The LXC file server manages my entire disk array and blue iris saves all the videos there via a shared drive mounted in windows
• When I setup the new path, BlueIris DID warn about writing videos to a disk not managed by windows (may have trouble deleting old files), but I haven't noticed any problems
• At the time I migrated, I was recording only on alerts, but I switched to 24/7 recording a few weeks ago; working fine as well

Otherwise, if you're just writing to single disk, direct passthrough is likely better

 

[AlwaysSomething]

I actually have room for two 3.5" HDD, one 2.5" SSD, and two M.2 drives.

I was thinking of using the M.2 drives for the Proxmox (Host, ISO, VM, etc.). Debating mirrored or not.

I may install two WD Purple HDD instead of just one and split the cameras between them trying to maintain coverage so if a disk does fail there is another camera that has some visibility. For example, I have 2 cameras that cover the front yard so I would put each camera on a different disk. That was the only reason I was thinking of using a PCIe Sata Controller card and passing it through. It would be used for 2 drives. I tend to over engineer and also have OCD.

Then I can use the 2.5" SSD for additional storage like if I want to export some clips or whatever for longer term storage.

 

[AlwaysSomething]

Just an update.

First, thanks again for the info and help.

Next, I finally got Proxmox 8.2.2 installed and running and this is AWESOME!!! Wish I did this a LONG time ago.

I have a Windows 10 Pro (v22H2) VM up and running succesfully. I haven't played with BI and the WD Purple drives for it yet since I'm more focused on CPAI at the moment.

I have a Ubuntu 20.04 VM up and running successfully. I know there is a newer version 22.04 but a CPAI developer said he had problems with the newer version so sticking with what he is using. I also used the bloated desktop version and not the light version for now until I'm more comfortable with installing apps/packages. I did the bloated version in case I want to use the Libre Office tools to create some spreadsheets or docs. The biggest thing was I was able to pass my Coral TPU using PCIe pass through successfully. The documentation was not correct but was able to figure out what I needed to do. Also, had an issue with the drivers at first but it was because my PC is using Secure Boot and needing to learn how that works. After I got it up I was even able to test passing images from my main BI PC to the VM which was really cool.

Since I don't have everything running fully I can't comment on performance. I will say that Proxmox with just Ubunutu and CPAI running doesn't break a sweat. I have Ubuntu with 4 GB of memory and just 2 cores for now. Looks like it idles at 0.85-1.71%. When I ran a CPAI benchmark the CPU hit 20% and that was it. The memory seems fixed at 3.08 of the 4 GB. Those are the Ubuntu VM numbers. The PVE server CPU idles at under 1% and when I ran the CPAI benchmark the CPU hit 4% and that was it. I know Windows and BI will be the big test.

I did actually clone my Ubuntu so I could test out different versions of CPAI since I am trying to compare the issues between the versions. This was SO EASY. The clone itself only took a couple of minutes (less than 5).

Again, this is awesome and highly recommend if someone is debating this in the future. There are some learning curves for Proxmox but Google is your friend. When it comes to configuring the VM there are different opinions on the settings (especially if it's for Windows). I don't know if what I chose was best but it's working. I actually think I am going to build a separate server with Proxmox so I can learn some new stuff without having to worry about rebuilding the OS in case I mess something up.

Again, this is great and thanks for the help.