What high spec server would you choose for 50 cameras?

[danletkeman]

We have such a setup.
we're running 51 cams all HikVision HD to UHD cams on an HP DL380
64GB RAM
Xeon Gold 2.8Ghz - 16C
Windows Server 2019 Standard
500GB C:\ Partition
6.7TB D:\ Partition for recordings - Holds about 2-3 weeks, give or take.
RAID 6

Camera system at peak day is running about 250 Mbps through the network so, as it's been stated, you better have some primo switches. I only use Extreme Switches. They are flawless and quick.

Any other info, let me know.

Are you running BlurIris on the server? I am guessing that your CPU doesn't have support for hardware encoding? I have run similar systems (45 cameras) with a different NVR software with similar specs as yours without issue, but I can't seem to get BI working on anything other than desktop machines with QuickSync support.

 

[bp2008]

Xeon Gold does not have video hardware acceleration.

Hardware acceleration in Blue Iris is really hit or miss these days anyway. On some systems it makes performance worse instead of better. Even when it makes it better it is not very important anymore thanks to sub streams being supported. A sub stream being decoded in software is way more efficient than a main stream being decoded by hardware acceleration.

 

[fvruvolo]

Are you running BlurIris on the server? I am guessing that your CPU doesn't have support for hardware encoding? I have run similar systems (45 cameras) with a different NVR software with similar specs as yours without issue, but I can't seem to get BI working on anything other than desktop machines with QuickSync support.

No encoding needed. It's fast and reliable, I run about 30 to 40% CPUU utilization
But I only run HP servers. IMHO the best for WIndows servers.

 

[danletkeman]

Interesting. I can't even get one 4k camera working smoothly with these specs.

Latest version of BI - 5.7.6.2
E5-2630 V3
ESXi 7 - Latest Patch
Windows 10 Guest VM (Fresh install)
Raid 6 - 8 x 10K SAS disks
16GB Ram
8 Cores assigned to the VM
IPerf from my desktop to the server shows 980Mbps
ATTO disk benchmark shows 3.2Gbps disk write performance

BI - No encoding enabled, no AI, one 4k camera with both sub and main stream, no firewall, Windows 10 install stripped to the bare bones, Latest VM Tools, Latest Windows Updates, no CRC errors on the Cisco 3560X switch that is connected between the server and the desktop.

I have another instance where I am running an I5-8400 and two 4K cameras, and two 4mp cameras and that system is running great. Same version of Windows 10, just bare metal, I just don't understand what I am missing on the VM. I have many years experience with VMWare in small datacenters and I have run 1000's of VM's. That said I could have missing something.

 

[looktall]

If it works better on lower spec bare metal than it does on a VM guest then I would have to say there is something hardware wise that is available on bare metal that isn't in the VM.
There might be a CPU function or some other hardware feature that you need to pass through to the guest.

 

[danletkeman]

If it works better on lower spec bare metal than it does on a VM guest then I would have to say there is something hardware wise that is available on bare metal that isn't in the VM.
There might be a CPU function or some other hardware feature that you need to pass through to the guest.

I don't think so. There isn't much that isn't passed through by default when it comes to the CPU features on ESXi. I have run different NVR software in the past on ESXi guest VM's before with 45+ cameras and it never missed a beat, so I know it's perfectly capable.

I need to do some more testing next week, but I just thew 24 cores at it and it seems much better, and at least the playback of the clips I took this week are keeping the CPU under 1 percent.

If all else fails throw more power at it.

 

[tmushy]

I don't think so. There isn't much that isn't passed through by default when it comes to the CPU features on ESXi. I have run different NVR software in the past on ESXi guest VM's before with 45+ cameras and it never missed a beat, so I know it's perfectly capable.

I need to do some more testing next week, but I just thew 24 cores at it and it seems much better, and at least the playback of the clips I took this week are keeping the CPU under 1 percent.

If all else fails throw more power at it.

Did you do the cpu optimization guide? Setting up first and second streams etc...? Im running 16 cams on my esxi server. Im running a 16 core i9 on it but only gave blue iris 8 cores. Its using about 20%.

 

[sdkid]

I don't think so. There isn't much that isn't passed through by default when it comes to the CPU features on ESXi. I have run different NVR software in the past on ESXi guest VM's before with 45+ cameras and it never missed a beat, so I know it's perfectly capable.

I need to do some more testing next week, but I just thew 24 cores at it and it seems much better, and at least the playback of the clips I took this week are keeping the CPU under 1 percent.

If all else fails throw more power at it.

I have 9 cameras running on a 9 yr old Dell tower with an I-5 FOURTH generation processor and 8gb of ram.
Stats right NOW...


I could buy an HP ProDesk I-5 8th Gen off ebay for $150 or so-- but there is just no reason to do it. I guess I could probably cut my processor load to 3% or so... LOL I know there are others here running 50+ cameras on BI with older computers and running very low processor load.

Dedicated full-time hardware seems to be very hard to beat both for stability and cost. My question for you guys is this--- with Very capable hardware available for so cheap, why F around with a VM machine for this at all? What advantage is there that outweighs the stability issues described here with VM servers?

 

[danletkeman]

Did you do the cpu optimization guide? Setting up first and second streams etc...? Im running 16 cams on my esxi server. Im running a 16 core i9 on it but only gave blue iris 8 cores. Its using about 20%.

Yes, that is all done. I had a suggestion on another forum to try changing the camera to h264 instead of h265, but I think just setting the VM to have 24 cores should make the difference.

What model CPU do you have?
Are you doing any hardware decoding with quicksync?
Are you playing back any 4k video clips on a 4k monitor? I find that is also a huge CPU hog on the server.

 

[danletkeman]

I have 9 cameras running on a 9 yr old Dell tower with an I-5 FOURTH generation processor and 8gb of ram.
Stats right NOW...
View attachment 162220

I could buy an HP ProDesk I-5 8th Gen off ebay for $150 or so-- but there is just no reason to do it. I guess I could probably cut my processor load to 3% or so... LOL I know there are others here running 50+ cameras on BI with older computers and running very low processor load.

Dedicated full-time hardware seems to be very hard to beat both for stability and cost. My question for you guys is this--- with Very capable hardware available for so cheap, why F around with a VM machine for this at all? What advantage is there that outweighs the stability issues described here with VM servers?

View attachment 162223

Nobody said that virtualization was the issue, that was your assumption. Your also making an assumption that I am running VM's at my house. I am not, only a desktop, no virtualization.

If you don't understand the concepts of virtualization I suggest you read up on that first. Its the single biggest advancement of the server industry in the last 20 years.

I want to use BI in a small business environment, and it's different than what I am use too. I have run 400+ camera systems on 10 hosts where I had 150+ VM's running and only 10 of them were NVR servers (40 cameras per VM). There was never any "stability issues" with any of them VM's.

So for me in particular, I am not F'ing around with VMs, I have a use case for it and I don't want to buy a desktop every time I need to spin up a machine.

 

[sdkid]

Nobody said that virtualization was the issue, that was your assumption. Your also making an assumption that I am running VM's at my house. I am not, only a desktop, no virtualization.

If you don't understand the concepts of virtualization I suggest you read up on that first. Its the single biggest advancement of the server industry in the last 20 years.

I want to use BI in a small business environment, and it's different than what I am use too. I have run 400+ camera systems on 10 hosts where I had 150+ VM's running and only 10 of them were NVR servers (40 cameras per VM). There was never any "stability issues" with any of them VM's.

So for me in particular, I am not F'ing around with VMs, I have a use case for it and I don't want to buy a desktop every time I need to spin up a machine.

Enjoy dude, LOL. I sense you're a little touchy, but you never really answered my question.
Also-- I was talking about MY house, not anyone else's-- no sure where your statement comes from. You had no stability issues with any of the 150+ VM's before--- running BI, or just in general??

Running 400 cameras had to be crazy in the setup you mentioned. My point was that YOU SAID you had issues running BI on a VM. You then noted that you had an 8th gen i-5 running fine, but a BI install on a XEON / VM setup was garbage. Gee.... sounds like the VM is the issue-- doesn't it?-- especially when you consider so many here have BI running fairly easily on old dedicated hardware with up to 60 cams. So my question is, Why F with a VM for BlueIris when you can run a cheap piece of dedicated hardware reliably for 5+ years? You came close when you said you don't want to buy a desktop every time you need to spin up a new machine. Cool. But you can't get a VM to work right with two 4k cams? That doesn't seem to bode well.

Lastly, I wasn't trying to give you shit-- I was trying to understand why you were taking that path to get BI running, so cool your jets son. Maybe if BI is different than what you are used to, it is not the right product to use for whatever you are planning in a "small business environment", OR, maybe you need to buy a desktop and configure it for the BI needs of each small-business environment customer you get.

 

[danletkeman]

Enjoy dude, LOL. I sense you're a little touchy, but you never really answered my question.
Also-- I was talking about MY house, not anyone else's-- no sure where your statement comes from. You had no stability issues with any of the 150+ VM's before--- running BI, or just in general??

Running 400 cameras had to be crazy in the setup you mentioned. My point was that YOU SAID you had issues running BI on a VM. You then noted that you had an 8th gen i-5 running fine, but a BI install on a XEON / VM setup was garbage. Gee.... sounds like the VM is the issue-- doesn't it?-- especially when you consider so many here have BI running fairly easily on old dedicated hardware with up to 60 cams. So my question is, Why F with a VM for BlueIris when you can run a cheap piece of dedicated hardware reliably for 5+ years? You came close when you said you don't want to buy a desktop every time you need to spin up a new machine. Cool. But you can't get a VM to work right with two 4k cams? That doesn't seem to bode well.

Lastly, I wasn't trying to give you shit-- I was trying to understand why you were taking that path to get BI running, so cool your jets son. Maybe if BI is different than what you are used to, it is not the right product to use for whatever you are planning in a "small business environment", OR, maybe you need to buy a desktop and configure it for the BI needs of each small-business environment customer you get.

You were the one putting F bombs in your reply. I was just replying with the same tone you gave me.

The 400+ camera setup was not using BI, it was different software, and I am not familiar with BI in a larger setup. Just as you are not familiar with virtualization.

I never said the VM was the issue, the hardware setup including the Xeon CPU without QuickSync is what I was questioning, I was saying VM to reference the instance.

I am not going to answer your question about why to use a VM. You clearly don't want to learn the use case and for me to teach you all about virtualization would be a waste of my time.

A word of advice. Instead of replying to someone about a concern or question they have where you know nothing about the topic at hand, try learning about it first rather than just implying your way is the only way. Putting a stupid "Change my mind" photo in your reply only indicates that you are arrogant enough to think your way is best.

All of that said, can you please leave this thread, so that all of the rest of us can get back to the topic of talking about high spec servers?

 

[looktall]

There isn't much that isn't passed through by default when it comes to the CPU features on ESXi.

What about the other hardware?
Graphics perhaps.

 

[wittaj]

Not having quicksync isn't the issue. Now that substreams are available, many don't use quicksync/Hardware acceleration and substreams opened up many opportunities for computers without quicksync, so that is not the issue. I disabled mine and ran it for days with no problem just to prove that point LOL.

While people have used BI in a VM and people have used BI in a raid format, I am not aware of anyone doing both.

Further, BI doesn't recommend VM or RAID, so there is always the potential of an issue when trying something it wasn't designed for.

Also, keep in mind that BI is a one-person operation. It was intended to be desktop based for residential uses. Sure small businesses have used it, but they tend to be the mom and pop operations coming from a Lorex NVR to this as an example.

Maybe you are simply trying/expecting too much from it coming from true commercial VMS systems...

 

[danletkeman]

What about the other hardware?
Graphics perhaps.

I don't have a GPU if you were wondering. I did turn off CPAI to see if that was the issue, but with only one camera I don't think that it would be a problem.

I have not had time to go back and try again, but remote playback of the clips I took seemed a lot better now that I went from 8 cores to 24 cores. I think it will go back and review the configuration once more and perhaps even get a different camera to test with.

 

[danletkeman]

Not having quicksync isn't the issue. Now that substreams are available, many don't use quicksync/Hardware acceleration and substreams opened up many opportunities for computers without quicksync, so that is not the issue. I disabled mine and ran it for days with no problem just to prove that point LOL.

While people have used BI in a VM and people have used BI in a raid format, I am not aware of anyone doing both.

Further, BI doesn't recommend VM or RAID, so there is always the potential of an issue when trying something it wasn't designed for.

Also, keep in mind that BI is a one-person operation. It was intended to be desktop based for residential uses. Sure small businesses have used it, but they tend to be the mom and pop operations coming from a Lorex NVR to this as an example.

Maybe you are simply trying/expecting too much from it coming from true commercial VMS systems...

I agree, I turned off Quicksync on my machine at home and it made no difference to the CPU usage. So I left it off.

A VM doesn't know what underlying disk hardware there is so there is no reason not to use a raid configuration or what I have used many times is actually a FiberChannel connected SAN. IOPS and latency is all that matters with disk access and BI doesn't have any issues with the slowest disk (7200 RPM HDD) so any RAID config is only going to be better than that.

The only problem with running a VM would be over provisioning, meaning that if someone doesn't allocate enough resources to the VM, it will result in a slow down when the resources are used by other VMs. This may be the reason why it is not recommended by BI, because the use case for the software is home use and many home users will land up doing just that.

Yes, I can see that the software is defiantly not scalable like I am used to. That said it works very well and I would like to give it a fair shot.

I have read that others are running systems with more cameras that I am planning so I know it's possible. I think I have a software configuration issue with BI that I need to address as all of the results of the benchmark tests I have done on the VM far exceeds the benchmarks of my system at home.

 

[sdkid]

A word of advice. Instead of replying to someone about a concern or question they have where you know nothing about the topic at hand, try learning about it first rather than just implying your way is the only way. Putting a stupid "Change my mind" photo in your reply only indicates that you are arrogant enough to think your way is best.
...

So you got triggered by a meme. LOL. Too F'ing bad. Dedicated hardware IS better for BI than a VM. Change my Mind.

As @wittaj pointed out-- BI is not made for and doesn't play nice with VM or raid. Did you think Maybe that's why it's not working well for you? I'm not asking for advice or info on setting up VMs-- but contrary to what you said-that is NOT the topic here. The topic at hand is "A HIGH SPEC SERVER FOR 50+ Cameras" and I responded, as did many others early in this thread, that you DO NOT NEED A HIGH SPEC SERVER to run BI with 50 cameras. You don't need a massive enterprise class server that can run dozens of VM's and you don't need an enterprise class storage network. An inexpensive desktop computer will do it just fine. THAT'S THE POINT, and me making that point again means I am more ON TOPIC in this thread than you are.

Lastly, Change my mind means just that--- CHANGE MY MIND. I am open to new ideas and ways of doing things. You still have not presented a solid reason why anyone should consider a VM configured BI install for a small business, but since you are intent on pursuing it, maybe you can start a thread on it. And you know what-- have fun figuring it out. Hell-- you might even change my mind. In the meantime, I think @fenderman can point you in the right direction for surveillance software that WILL reliably run in the Enterprise-class environment you describe for very large volumes of cameras.
Good luck!

 

[danletkeman]

So you got triggered by a meme. LOL. Too F'ing bad. Dedicated hardware IS better for BI than a VM. Change my Mind.

As @wittaj pointed out-- BI is not made for and doesn't play nice with VM or raid. Did you think Maybe that's why it's not working well for you? I'm not asking for advice or info on setting up VMs-- but contrary to what you said-that is NOT the topic here. The topic at hand is "A HIGH SPEC SERVER FOR 50+ Cameras" and I responded, as did many others early in this thread, that you DO NOT NEED A HIGH SPEC SERVER to run BI with 50 cameras. You don't need a massive enterprise class server that can run dozens of VM's and you don't need an enterprise class storage network. An inexpensive desktop computer will do it just fine. THAT'S THE POINT, and me making that point again means I am more ON TOPIC in this thread than you are.

Lastly, Change my mind means just that--- CHANGE MY MIND. I am open to new ideas and ways of doing things. You still have not presented a solid reason why anyone should consider a VM configured BI install for a small business, but since you are intent on pursuing it, maybe you can start a thread on it. And you know what-- have fun figuring it out. Hell-- you might even change my mind. In the meantime, I think @fenderman can point you in the right direction for surveillance software that WILL reliably run in the Enterprise-class environment you describe for very large volumes of cameras.
Good luck!

I don't care about changing your mind.

@wittaj pointed out-- BI is not made for and doesn't play nice with VM or raid

No that is not what he said. You misinterpret everything.


Please point me to the document from the developer stating that I cannot run BI on a VM

Please point me to the document from the developer stating that I cannot run BI on a RAID disk

Please point me to the document from the developer stating that BI is only for home use

Please point me to the document from the developer stating that you cannot run BI on a high spec server

 

[sdkid]

No that is not what he said. You misinterpret everything.

"...Further, BI doesn't recommend VM or RAID "

Please point me to the document from the developer stating that I cannot run BI on a VM

Please point me to the document from the developer stating that I cannot run BI on a RAID disk

Please point me to the document from the developer stating that BI is only for home use

Please point me to the document from the developer stating that you cannot run BI on a high spec server

NO.

 

[spuls]

Interesting. I can't even get one 4k camera working smoothly with these specs.

In the end the xeon v3 is a ~10 year old plattform and does not boost very high to todays standards. Also a small number of old sas disks with raid 6 isn´t the best choice for lot´s of small writes. But it should be able to handle a single camera in a vm setup for sure. What exactly does not work "smoothly"? the recording or live view?