To virtualize BI or not?

[beatnik]

We are looking at upgrading our ancient analog camera system here at work as well as virtualizing 4 servers currently running on old repurposed amd desktops. Originally our thought was to set the new camera system up on a i7-4970 system from the Dell Outlet in addition to a true server to replace the old power hogs that have been patchworked together. The server we are looking at is a single Xeon E5-2620 v3 system however we could step that up to a dual processor server giving us 24 virtual cores if need be.

I guess what I am trying to figure out is if 12 to 14 of these virtual cores would be sufficient to run BI recording (12) 3 & 4mp (approx 40 to 45 total megapixels) direct to disk or if perhaps there are some other aspects of this scenario I need to consider.

Thanks much for any insight you may be able to provide.

 

[bp2008]

I strongly discourage virtualizing Blue Iris. It is a huge resource hog, using lots of CPU time, memory bandwidth, and disk space. A layer of virtualization is going to add overhead to an already expensive process and make it use even more resources. In your case specifically, you could easily use half of your server's capacity just with Blue Iris (more than half, if you prefer to run high frame rates). More importantly, if Blue Iris ever does not have all the resources it needs to handle its continuous workload, then you can end up with corrupt video, crashes, and other fun things. So sharing resources with other processes is not usually a good idea.

 

[Zorac]

could have BI run on the base os (assuming its windows based) and still do vms for the rest.

 

[dalepa]

We virtualize everything now days on vmware, 1000s of VMs... Never tried blueiris, but it's just a CPU intensive app, so I would give it a try. You can always vmotion to a larger server if needed. The Xeon E5-2620 v3 is most likely overkill for 12 cams.

 

[bp2008]

A Xeon E5-2620 v3 is comparable to an i7-4790 in multi-threaded processing capability, so I wouldn't call it overkill for 12 cams. Chances are, frame rates will have to be reduced from their defaults to achieve acceptable CPU usage. Feel free to try it on a VM or on the base OS hosting other VMs. Depending on the other loads, it may work just fine. But also remember most of Blue Iris' resource usage is in decoding many video streams simultaneously. If the CPU or the network or any other subsystem gets overloaded even temporarily then Blue Iris may become unstable.

 

[beatnik]

Some good points to consider, thanks much.

 

[Git]

I am running BlueIris in a VM on a single Xeon E5-2630 v3 which has (16) logical processors

OS is Server 2012 Essentials (also used to back up my computers), I currently have 7 cores and 6gb or ram dedicated to this VM. CPU usage averages ~50%. But, my camera's are set to record on motion only and only 3 of them are scheduled to record during the day. (I do have one of those cameras that is cloned and is scheduled to record 24/7)

Current cameras:


Basically (5) Hiks, (2) Huisun and (2) Foscams I have not replaced yet

CPU Usage for this VM over the last week:


Works for me - but your mileage may vary

 

[beatnik]

Thanks for the info Git, very helpful.

 

[ipcamal]

I am running BlueIris in a VM on a single Xeon E5-2630 v3 which has (16) logical processors

OS is Server 2012 Essentials (also used to back up my computers), I currently have 7 cores and 6gb or ram dedicated to this VM. CPU usage averages ~50%. But, my camera's are set to record on motion only and only 3 of them are scheduled to record during the day. (I do have one of those cameras that is cloned and is scheduled to record 24/7)

Current cameras:
View attachment 7215

Basically (5) Hiks, (2) Huisun and (2) Foscams I have not replaced yet

CPU Usage for this VM over the last week:
View attachment 7216

Works for me - but your mileage may vary

It does have a bit of cpu ready %. Did you check the video make sure there are no skipped frames during those times?

 

[Git]

It does have a bit of cpu ready %. Did you check the video make sure there are no skipped frames during those times?

First of all let me say that when it comes to VMWare, I consider myself to be an "enthusiast" at best

I have not experienced any performance problems per say but I looked into it a little further. Here is ESXTOP showing the 4 vm's that I usually have running. BlueIris is running on the 'server' and is showing 2.41 for %RDY. I do realize I have some spikes in the previous chart


One of the other VM's is my Plex Server and when I add a new movie, it can at times be CPU intensive as it is transcoding it. Any suggestions or thoughts?

Edited to add that I added a movie to the Plex Server and %RDY for my BlueIris VM spiked up to 41%

 

[ipcamal]

Cpu ready means that vm is ready to execute but doesn't have cores available to execute it. The higher the number the worse it is. you want to keep that number under 5 but it does also depend on the esxi host and vm cpu configs. 41% is definitely not good but that's expected since transcoding is cpu intensive and esxi allocates all cpu resources to the plex vm. 2.41% is ok but prolong spikes is usually not good.

what is the physical esxi server config, cpu make and model and is HT enabled or not? and what is the cpu config of each vm running on that host. my suggestion would be to add a second esxi host and setup affinity rule to keep both vm separate. this way they will each get one host. Its an overkill but that's the solution. Post the configs and I''ll take a look. Maybe some vms are not sized correctly. Ever hear of NUMA ?


Alex

PS - I support over 700 esxi hosts and about 17000 vms for my day job

First of all let me say that when it comes to VMWare, I consider myself to be an "enthusiast" at best

I have not experienced any performance problems per say but I looked into it a little further. Here is ESXTOP showing the 4 vm's that I usually have running. BlueIris is running on the 'server' and is showing 2.41 for %RDY. I do realize I have some spikes in the previous chart
View attachment 7349

One of the other VM's is my Plex Server and when I add a new movie, it can at times be CPU intensive as it is transcoding it. Any suggestions or thoughts?

Edited to add that I added a movie to the Plex Server and %RDY for my BlueIris VM spiked up to 41%
View attachment 7350

 

[Git]

NUMA - no, never heard of it

Motherboard: Supermicro SNK-P0048AP4
Chassis: SuperMicro CSE-743TQ-865B
CPU Xeon E5-2630 v3 8 CPU's x 2.3999 GHz
Memory 32GB Crucial CT2K16G4RFD4213
HT Active

ESXI 5.5.0 running on a 4GB USB drive, Main Datastore is on a 1TB Samsung 840 SSD
The BlueIris server VM has (4) additional 3TB hard drives that I use as Mapped Raw drives
My Plex Server VM has (1) 4TB hard drive as Mapped Raw drive




"PS - I support over 700 esxi hosts and about 17000 vms for my day job

" WOW - I am sure that keeps your pretty busy - thanks for your help

 

[ipcamal]

Here is more info on NUMA: its basically cpu to local memory mapping which gives best optimized performance vs where vm has to jump numa nodes because of config
Most modern CPU’s, Intel new Nehalem’s and AMD’s veteran Opteron are NUMA architectures. NUMA stands for Non-Uniform Memory Access, but what exactly is NUMA? Each CPU get assigned its own “local” memory, CPU and memory together form a NUMA node. An OS will try to use its local memory as much as possible, but when necessary the OS will use remote memory (memory within another NUMA node). Memory access time can differ due to the memory location relative to a processor, because a CPU can access it own memory faster than remote memory.

ESX is NUMA aware and will use the NUMA CPU scheduler when detecting a NUMA system. On non-NUMA systems the ESX CPU scheduler spreads load across all sockets in a round robin manner. This approach improves performance by utilizing as much as cache as possible. When using a vSMP virtual machine in a non-NUMA system, each vCPU is scheduled on a separate socket.On NUMA systems, the NUMA CPU scheduler kicks in and use the NUMA optimizations to assigns each VM to a NUMA node, the scheduler tries to keep the vCPU and memory located in the same node. When a VM has multiple CPUs, all the vCPUs will be assigned to the same node and will reside in the same socket, this is to support memory locality as much as possible.
At this moment, AMD and Intel offer Quad Core CPU’s, but what if the customer decides to configure an 8-vCPU virtual machine? If a VM cannot fit inside one NUMA node, the vCPUs are scheduled in the traditional way again and are spread across the CPU’s in the system. The VM will not benefit from the local memory optimization and it’s possible that the memory will not reside locally, creating added latency by crossing the intersocket connection to access the memory.

you are actually using 15 cores. BI is 1x6 which is 6 cores not 7
try doing 2x4 for BI
2x2 for Plex
see if that helps.

Leave the others alone.

You only have one physical processor?

Also take a look at the BIOs make sure that its set for high permormance, C1 state is disabled, VT stuff is turned on and HT is on. Does vsphere show 16 cores avaiable on physical processor?

NUMA - no, never heard of it

Motherboard: Supermicro SNK-P0048AP4
Chassis: SuperMicro CSE-743TQ-865B
CPU Xeon E5-2630 v3 8 CPU's x 2.3999 GHz
Memory 32GB Crucial CT2K16G4RFD4213
HT Active

ESXI 5.5.0 running on a 4GB USB drive, Main Datastore is on a 1TB Samsung 840 SSD
The BlueIris server VM has (4) additional 3TB hard drives that I use as Mapped Raw drives
My Plex Server VM has (1) 4TB hard drive as Mapped Raw drive


View attachment 7353

"PS - I support over 700 esxi hosts and about 17000 vms for my day job " WOW - I am sure that keeps your pretty busy - thanks for your help

 

[Git]

you are actually using 15 cores. BI is 1x6 which is 6 cores not 7

That was a typo - I added another core and didn't update my chart so it is 1x7

Yes - I have just the one CPU

I will try those 2x settings and check the bios, etc. Here is a screen shot of the summary page

 

[richtj99]

I am trying to run a Geovision NVR within VMware using a E3-1245v5 Xeon & X11 Supermicro motherboard. I am finding with 2 cores the system is maxed out on the CPU - always running 100%. My old I72600 physical NVR box is running at 40% with 32 cameras. I would love to virtualize it to get rid of the older box.

I am sending all video to a freenas box on my network (E3-1231 v3 supermicro board).

Any suggestions other than beefier hardware?

 

[nayr]

hardware acceleration can crush realtime video with a few watts of power that will take a software solution hundreds of watts of cpu power.. Putting BI in a VM is stupid on so many levels.

/Cloud Engineer

 

[richtj99]

I figured as much but due to electric prices on a recurring basis I figured it was worth a try.

 

[nayr]

HD Video is so highly compressed with h264/h265 that even the beefiest and most powerful processors can be beat down into submission trying to reach scale via software encoding/decoding, if it were not for hardware acceleration nothing consumer grade would be able to decode it in realtime.. a lil tiny Raspberry Pi can decode dozens of h264 streams running off the power a USB port provides... yet that same job will absolutely crush thousands of dollars of virtualized equipment.

I deal with HD video in the cloud, and we pass through high end graphics hardware to instances handling the encoding/decoding of video.. BI dont do CUDA or anything like that, it was not designed for virtualized architecture, it was designed to run on bare metal as is apparent by its hardware support/requirements.. so locking it up in a virtualized environment might as well be the same as trying to get it to run on a Pentium 4, its going to burn electricity at incredible rates and not accomplish very much.

 

[ruppmeister]

I currently run my BI server on Win10 Pro with Hyper-V running. BI is on the base OS and gets the use of the Intel Sync and runs great. With what CPU and ram I have left I allocate to some Linux servers and test machines running within Hyperv.

I agree with the others that running BI on pure bare metal hypervisor is not the best use of resources, running Hyperv works for me.


Sent from my iPhone using Tapatalk

 

[bp2008]

I am trying to run a Geovision NVR within VMware using a E3-1245v5 Xeon & X11 Supermicro motherboard. I am finding with 2 cores the system is maxed out on the CPU - always running 100%. My old I72600 physical NVR box is running at 40% with 32 cameras. I would love to virtualize it to get rid of the older box.

I am sending all video to a freenas box on my network (E3-1231 v3 supermicro board).

Any suggestions other than beefier hardware?

Assign more CPU cores to the VM running the NVR software. Given the virtualization overhead, your e3-1245v5 CPU is about the same speed as the i7-2600 was. So you will need at least half the CPU assigned to this VM. Clearly Geovision isn't one of the more efficient brands.

I don't know anything about Geovision NVR software but if it has support for hardware acceleration it almost certainly can't use that acceleration from within a virtual machine.