3990X & Blueirs thoughts ?

[mrc545]

Well, I'm not very good at technical writeups (You win that one by a mile, @bp2008 !) , but I'll share my findings so far with my 3970x build and answer any questions to the best of my knowledge.

Bottom Line: The Threadripper 3970x eats a beefy BI setup for breakfast.

I am meeting my exact theoretical MP/s of 2380 MP/s. BI MP/s stats stay between 2378-2382 MP/s. I have 4 indoor Amcrest 4MP indoor wifi cams that were giving me connectivity issues long before this build even started that I disabled for my testing, but once I get the bugs worked out, they will add another theoretical 328 MP/s that I can test with that MP/s "wall".

I tested with and without RDP initially, and RDP only introduces a 2-3% CPU usage overhead with this setup. All of the stats below incorporate that overhead because I got tired of the fan noise and had to move it to the other room for my sanity. Other constants aside from RDP are that it is being displayed to a 4k monitor, the 2380 MP/s input, unrestricted live preview rate, and no substreams. Core Performance Boost and PBO are disabled, and it is running only at its base clock speed.

• Fast scaling, 2x RTX 2060 NVDEC : 13% CPU utilization
• Bicubic scaling, 2x RTX 2060 NVDEC: 27% CPU utilization
• Fast scaling, No H/W acceleration: 20% CPU utilization
• Bicubic scaling, No H/W acceleration: 32% CPU utilization

It looks like the additional GPU's aren't of much added value with the 3970x, as it can handle the extra work on its own without adding a ton of overhead. I did notice that over RDP, it was more responsive with NVDEC enabled though. I doled out H/W acceleration between the two GPU's, and they are each seeing around 20% Cuda and 65-70% Video Decode usage.

Average temps for all testing were around 55c with air cooling. Even doing Prime95 stress tests, temps would only go to the low 80's with the Noctua cooler, and no thermal throttling was seen.

UPS load reported is 423 watts. This is with 3.5" 10 spinners, 2x RTX 2060's, and some other stuff crammed in there.

It is apparent that if this thing is overkill, that 3990x build is going to be absolutely insane. I wish I had a box of 8MP cams to throw at this setup, but I've upgraded all I can (thanks Andy, you da man ) and have 360 degree coverage around my house. Very curious to see where that "wall" is with this thing.

 

[bp2008]

Thanks @mrc545!

There's a trick @crw030 was using where I guess you add the same camera multiple times but mark each instance of it as the master and then BI will pull and decode each stream separately?

 

[mrc545]

Ah, I didn't even think about that. I will play around with it once I get some other important configuration squared away.

Managed to get the other 4MP cameras working correctly. The math was right on the money, 2708 MP/s, with no signs of hitting the wall.



At this level of workload, I am getting 19% CPU on fast scaling, and 33% on bicubic, with the GPU's doing the decoding work. 18 GB of memory in total being utilized, with 13.4 of that being used specifically by BI. I have no other non-Windows applications or services running on this at the moment, with the exception of the UPS interface software, and HWinfo64.

 

[IAmATeaf]

Read this thread and am impressed. So you guys have these beast PCs running all the time?

I work in IT and I also have a VM box but that only ever gets turned on when I really need to as the thing burns power and generates a lot of heat. I thought mine was an impressive box, I run dual Xeons, 96Gb of ECC RAM, dual SAS RAID controllers which are battery backed up. For hard disks I have a single 256Gb boot disk and then 8 320Gb 10k drives and 8 500Gb 7.2k disks which are configured in RAID5.

I suppose when you think of the things that are powered it makes some sense, so I have a BI PC, a NAS and also a small HP micro server which are always powered.

 

[CCTVCam]

3080 is out in October. I'd personally get something easy to sell that won't drop in value like a stone when the 3080 comes out like a middle of the road card eg 2060 / 70 super.

I doubt the ti's are going to hold anything like their value given the 3080 is supposedly twice as powerful and can properly implement ray tracing. If youw ant the best 3080, you want the 3090! Yep the 3090 is back with a double sided board and 24gb of memory (vs possibly 11gb for 3080). Sounds like it's the titan under a new name unless of course they call it the 3090 titan on release. Also, note the new cards are power hungry - repored draw is between 320-350w at stock. I also believe SLI is no longer supported. Big Navi from AMD may also be a contender as no-one knows the comaprative performance yet.

 

[crw030]

For me it was more about testing the limits, once I find those I’ll delete the 20-30 duplicate cameras and either just run everything directly on the box or play around with virtualization.

Right now skipping VMs because 1. Learning curve and 2. I didn’t want to introduce issues that wouldn’t be present on bare metal (since bare metal is the recommended approach).

@mrc545 if you have a chance, just add a couple dummy logins to the camera, and add additional “cameras” using those logins and selecting the “Group Master” or w/e you should see your MP/s continue to climb with each new camera and all cameras will have their own kB/s with no clone indicator.

BTW which Windows OS are you running ? And are you running the new sub streams approach? (I was testing the limits without sub streams but will run them In the everyday config).

 

[reflection]

UPS load reported is 423 watts. This is with 3.5" 10 spinners, 2x RTX 2060's, and some other stuff crammed in there.

Watch out for the power bill. Running that 24/7 will cost about $500/year. The GPUs suck power.

 

[reflection]

Right now skipping VMs because 1. Learning curve and 2. I didn’t want to introduce issues that wouldn’t be present on bare metal (since bare metal is the recommended approach).

1) It's worth it
2) I recommend the VM approach. It cost me less than $5/year for power to run BI 24/7 (if I were to allocate a certain percentage of my power draw from my server)

 

[mrc545]

Read this thread and am impressed. So you guys have these beast PCs running all the time?

Yes, it's going to be running all the time. In a back room. With the door closed. Darn thing is LOUD.

@mrc545 if you have a chance, just add a couple dummy logins to the camera, and add additional “cameras” using those logins and selecting the “Group Master” or w/e you should see your MP/s continue to climb with each new camera and all cameras will have their own kB/s with no clone indicator.

BTW which Windows OS are you running ? And are you running the new sub streams approach? (I was testing the limits without sub streams but will run them In the everyday config).

@crw030 , I will do that when I have some extra downtime this weekend. Thanks for the great idea. I'm running Windows 10 Pro. Not doing the substreams yet. That'll be on the docket for further testing down the line.

Watch out for the power bill. Running that 24/7 will cost about $500/year. The GPUs suck power.

You're pretty much spot on with the annual cost. It'll be worth it to me. BI and my cameras have saved me far more than that so far. Caught the mailman doing a hit and run on my mailbox, and one of my neighbors kids doing a hit and run on my car. The $500/year is just insurance to me. My 9900k build dropped frames constantly and I had to neuter the capabilities of the cameras for it to even run well.

I am currently testing a clip export, and noticed it is extremely slow, but there is no bottleneck anywhere that I am seeing. CPU usage hasn't gone up at all. Is this a single-core feature or something?

 

[Slugger]

Hey guys - Couple questions for you since you clearly have more experience than me (I have never done virtualization but understand the concept):

1) How hard is it to convert a running Windows 10 machine (i7-4790k, 16GB RAM, see below specs) to a virtual machine to use it as a Pfsense router? Does it involve reformatting, etc and starting fresh with the PC or can it simply be virtualized at any point without effecting the current config (other than taking aware some resources)? I'm thinking I may eventually virtualize this machine so I can also use it as a PFsense router. But I'm setting it up for Blue Iris now and simply focusing on that piece at the start. Trying to decide if I should go ahead and virtualize now to create less work later. Right now it's pretty much a fresh single install of Windows 10

2) Does it look like this PC will run well for: Blue Iris running 6-8 cameras (4MP) AND as a Pfsense router? Don't laugh at the 2014 prices I paid lol!

I know I'll need to add a big purple hard-drive. Hoping I don't need a GPU (onboard graphics currently). Anything I'm missing?:

 

[mrc545]

Hey guys - Couple questions for you since you clearly have more experience than me (I have never done virtualization but understand the concept):

1) How hard is it to convert a running Windows 10 machine (i7-4790k, 16GB RAM, see below specs) to a virtual machine to use it as a Pfsense router? Does it involve reformatting, etc and starting fresh with the PC or can it simply be virtualized at any point without effecting the current config (other than taking aware some resources)? I'm thinking I may eventually virtualize this machine so I can also use it as a PFsense router. But I'm setting it up for Blue Iris now and simply focusing on that piece at the start. Trying to decide if I should go ahead and virtualize now to create less work later. Right now it's pretty much a fresh single install of Windows 10

2) Does it look like this PC will run well for: Blue Iris running 6-8 cameras (4MP) AND as a Pfsense router? Don't laugh at the 2014 prices I paid lol!

I know I'll need to add a big purple hard-drive. Hoping I don't need a GPU (onboard graphics currently). Anything I'm missing?:

My primary PC has a 4790k. I was using it from 2014-2019 for BI with about the same load as you, and it was fine. I actually had more overhead on it because I had BI running in a VM (super inefficient- bridged the extra NIC I had to the VM and isolated it to mitigate remote access vulnerabilities. I have since implemented PKI and VPN into my network for remote viewing). The 4790k also supports QuickSync, so you should be good to go.

I haven't used pfsense, but you should just be able to put pfsense on VMware Player (free), and add extra NICs that are inline between your WAN and LAN.

 

[crw030]

If I were you, I would read a bunch of threads specific to visualizing pfSense (Netgate Forum) before proceeding in that direction.

By itself, pfSense doesn't need much, in fact it needs so little, that a full computer is always going to be overkill. That would tend to support virtualization where you could allocate like 1-core & 6MB RAM to it, but you are adding complexity that you will have to work through any issues (and while your internet is down most likely).

 

[Slugger]

By itself, pfSense doesn't need much, in fact it needs so little, that a full computer is always going to be overkill. That would tend to support virtualization where you could allocate like 1-core & 6MB RAM to it, but you are adding complexity that you will have to work through any issues (and while your internet is down most likely).

Yea I hate to buy a $150-200 box to run pfsense when I'm already running this PC in the server rack. But I don't want it to be too complicated. I really have no idea how difficult the virtualization aspect will be. And I figured I would probably have enough spare resources. I'd like to eventually implement adblocking and intrusion detection so Pfsenese seemed to be the "route" to go (pun intended). Currently, I just have a couple Velop nodes doing the routing. They don't even support a VPN connection for remote access...

 

[reflection]

Yea I hate to buy a $150-200 box to run pfsense when I'm already running this PC in the server rack. But I don't want it to be too complicated. I really have no idea how difficult the virtualization aspect will be. And I figured I would probably have enough spare resources. I'd like to eventually implement adblocking and intrusion detection so Pfsenese seemed to be the "route" to go (pun intended). Currently, I just have a couple Velop nodes doing the routing. They don't even support a VPN connection for remote access...

Will you be using this machine be just for BI and pfSense? Depending on your needs, you might consider running a type 1 hypervisor instead of a type 2. ESXi is also free.

 

[Slugger]

Will you be using this machine be just for BI and pfSense? Depending on your needs, you might consider running a type 1 hypervisor instead of a type 2. ESXi is also free.

Well the only think I currently have on the PC is a Plex server...but I haven't even powered the computer on in over a year.... so I guess I'm not really using the Plex server. I'd be OK ditching it if it made my life much simpler. If I ditch Plex or move it elsewhere that means the PC could be dedicated to BI and Pfsense

 

[bp2008]

@Slugger

It will be cleaner if you just install Windows fresh on the VM.

But there are several ways to convert physical to virtual. It may be risky or challenging if you don't have a solid grasp of how virtualization or disk imaging works. Remember Windows will lose its activation, but you should be able to re-activate by re-entering the product key.

Perhaps the simplest (and best-performing) way to convert physical to virtual is to leave the Windows OS intact on the SSD (don't install your hypervisor on the SSD), and then you can just map the entire disk to your virtual machine. You can't do this if you have set it up as a RAID array using the onboard RAID controller however. Unless the RAID was a RAID 1 (mirror).

Another approach is to save an image of your Windows OS boot drive and then restore the image onto your virtual machine's virtual hard drive. I recommend Macrium Reflect for this, as they have a very capable free version.

If you decide to use VMWare ESXi, then there is a tool specifically designed to make a virtual disk image from a physical hard drive. VMware vCenter Converter transforms your Windows- and Linux-based physical machines and third-party image formats to VMware virtual machines.


The last option you might not have considered is if you have Win10 Pro then you can install the Hyper-V role and use Windows's built-in hypervisor to run pfsense. I personally would recommend against it because I find Hyper-V to be rather frustrating, especially with how it does networking, and Windows 10 is not designed to run 24/7 -- its biannual feature updates often require user intervention at bootup. And you wouldn't want to lose your home's internet connection every time you rebooted it for an update.

 

[reflection]

Well the only think I currently have on the PC is a Plex server...but I haven't even powered the computer on in over a year.... so I guess I'm not really using the Plex server. I'd be OK ditching it if it made my life much simpler. If I ditch Plex or move it elsewhere that means the PC could be dedicated to BI and Pfsense

Since you have not turned on this machine in a while, you would probably not be using Plex that often in the future. Plex can also leverage the integrated GPU for transcoding. If you do go the VM route, only install one Windows VM and put both Plex Media Server and BI on there and pass through the integrated GPU. I prefer running Plex on linux, but you can't have the GPU passed through to two different VMs (so you can't have a linux VM for Plex and Windows for BI). Plex won't be consuming resources 24/7 like BI will so you will probably be okay for the 2hours that you are watching a movie. You would have a second VM for pfSense.

 

[Slugger]

Another approach is to save an image of your Windows OS boot drive and then restore the image onto your virtual machine's virtual hard drive. I recommend Macrium Reflect for this, as they have a very capable free version.

This seems like a pretty easy option.

But I don't have much on that PC so starting fresh is OK too. I think I may just do that unless the above is actually easier.

If I'm starting fresh what's the best and easiest free way to set up the virtual machine? Is the VMware software free and recommended?

 

[reflection]

This seems like a pretty easy option.

But I don't have much on that PC so starting fresh is OK too. I think I may just do that unless the above is actually easier.

If I'm starting fresh what's the best and easiest free way to set up the virtual machine? Is the VMware software free and recommended?

If you don't have any data or apps you need to keep, installing Windows fresh give you a clean start. Yes, the ESXi hypervisor from VMware is free.

Here are some basic steps. You will have to google for details. I'm going to assume you are doing VLANs. Let's say your VLANs are:
10 WAN (DHCP from your provider)
20 Cameras (192.168.20.0/24)
30 Internal (192.168.30.0/24)
40 IoT (192.168.40.0/24)

On your switch, create four VLANs, 10, 20, 30, 40 for the ones above. Set the interface connected to your ESXi machine to be a trunk port. Set the interface connected to your provider's modem to be on access VLAN 10.

1. Go to my.vmware.com and register for a free account.
2. Download vSphere (ESXi) here: Download VMware vSphere. Get the ISO image.
3. Google how to create a boot image using a USB stick.
4. Install ESXi using that USB stick.
5. Once it reboots, set an IP address using the console. Set the management VLAN to be 30.
6. Using another computer, connect to the switch on a port on VLAN 30. open a web browser and open a page to the IP address of your ESXi host. Login.
7. There should already be a port group called VM Network with VLAN 30. Create three more port groups. One called WAN with VLAN 10. One called Cameras with VLAN 20. One called IoT with VLAN 40. The VM Network will be your internal VLAN. You don't have to rename it unless you want to.
8. Create a new VM for pfSense. One vCPU is probably enough. 1GB ram is enough for basic firewalling. If you want to do ad blocking, allocate 4-6GB ram. Give it four vNICs. One on port group WAN (VLAN 10), one on port group Cameras (VLAN 20), one on port group VM Network (VLAN 30), and one on port group IoT (VLAN 40). Setup pfSense so that the WAN port gets a DHCP address. This is your outside interface. The pfSense interface towards the cameras will be isolated from the world. Have a rule that doesn't let it talk out. The only thing that anything on the Camera subnet can to is an NTP server (and DNS if you need to resolve your NTP server). Internal can talk to anyone. IoT can talk to only admin machines.
9. Create a new Windows 10 VM (my BI machine has 2 vCPU and 8GB ram). You might consider 4 vCPU and 12GB ram. Give it two vNICs. One on port group Cameras (VLAN 20) and one on port group VM Network (VLAN 30) You will need to mount your WIndows 10 ISO image in the virtual DVD and boot and install Windows. BI will talk to the cameras on the directly connected VLAN 20.
10. If you have a wireless AP, it can also connect to your switch as a trunk port. You can create multiple SSIDs that map to different VLANs. This will allow you to have a wireless doorbell camera that will live on VLAN 20. And you can have wireless IoT devices live on VLAN 40. And your internal stuff of course would live on VLAN 30.

pfsense is your router and firewall between your VLANs. You don't need a L3 switch in this case.

 

[Slugger]

If you don't have any data or apps you need to keep, installing Windows fresh give you a clean start. Yes, the ESXi hypervisor from VMware is free.

Here are some basic steps. You will have to google for details. I'm going to assume you are doing VLANs. Let's say your VLANs are:
10 WAN (DHCP from your provider)
20 Cameras (192.168.20.0/24)
30 Internal (192.168.30.0/24)
40 IoT (192.168.40.0/24)

On your switch, create four VLANs, 10, 20, 30, 40 for the ones above. Set the interface connected to your ESXi machine to be a trunk port. Set the interface connected to your provider's modem to be on access VLAN 10.

1. Go to my.vmware.com and register for a free account.
2. Download vSphere (ESXi) here: Download VMware vSphere. Get the ISO image.
3. Google how to create a boot image using a USB stick.
4. Install ESXi using that USB stick.
5. Once it reboots, set an IP address using the console. Set the management VLAN to be 30.
6. Using another computer, connect to the switch on a port on VLAN 30. open a web browser and open a page to the IP address of your ESXi host. Login.
7. There should already be a port group called VM Network with VLAN 30. Create three more port groups. One called WAN with VLAN 10. One called Cameras with VLAN 20. One called IoT with VLAN 40. The VM Network will be your internal VLAN. You don't have to rename it unless you want to.
8. Create a new VM for pfSense. One vCPU is probably enough. 1GB ram is enough for basic firewalling. If you want to do ad blocking, allocate 4-6GB ram. Give it four vNICs. One on port group WAN (VLAN 10), one on port group Cameras (VLAN 20), one on port group VM Network (VLAN 30), and one on port group IoT (VLAN 40). Setup pfSense so that the WAN port gets a DHCP address. This is your outside interface. The pfSense interface towards the cameras will be isolated from the world. Have a rule that doesn't let it talk out. The only thing that anything on the Camera subnet can to is an NTP server (and DNS if you need to resolve your NTP server). Internal can talk to anyone. IoT can talk to only admin machines.
9. Create a new Windows 10 VM (my BI machine has 2 vCPU and 8GB ram). You might consider 4 vCPU and 12GB ram. Give it two vNICs. One on port group Cameras (VLAN 20) and one on port group VM Network (VLAN 30) You will need to mount your WIndows 10 ISO image in the virtual DVD and boot and install Windows. BI will talk to the cameras on the directly connected VLAN 20.
10. If you have a wireless AP, it can also connect to your switch as a trunk port. You can create multiple SSIDs that map to different VLANs. This will allow you to have a wireless doorbell camera that will live on VLAN 20. And you can have wireless IoT devices live on VLAN 40. And your internal stuff of course would live on VLAN 30.

pfsense is your router and firewall between your VLANs. You don't need a L3 switch in this case.

Simply awesome. @reflection once again you have been very generous with your knowledge and time exceeding my expectations. Thank you!

I like this plan. It gives me ultimate flexibility and control but has minimal additional cost for me... So I can put more money towards cameras! I just have to learn about some stuff I wanna know anyways...

I think I’ll just follow your instructions as closely as possible. It’s quite a bit for me to learn and set up but I’m sure I can get through it. I am pretty good with computers and watching youtube lol... I may ping you guys here if I get stuck. It will be a week or two until I have my switch and then I’ll start. But don’t hesitate to check in or provide additional thoughts.

@tech101 sorry if I highjacked your thread. I swear I didn’t mean to!