94 Camera Blue Iris install

[By the Book Networking]

That's right!

96 Cameras, currently installed are 64 cameras.

Thank you for your time in reading this thread, I will post as many photos and answer as many questions that I can!
This is a multi phase project that I have been working on with a company for a few months. Last summer I installed 33 cameras in the company's retail store, we stumbled upon Blue Iris and was impressed.

That being said they where firm on wanting Blue iris for their warehouse, I insisted on Digital Watchdog but licensing was $87 CAD X 94 = Ouch!!! my local DW rep would not break a deal unless I used their cameras...

My main concerns for the install was:
CPU power
Redundancy
Capacity
Network
Reliability

For CPU power I found 2 Dell Poweredge R720XD equipped with 2X E5-2690 10 Core 20Thread CPU at 3GHz.
Those servers came equipped with 64 gigs of ram, 12 3.5" HDD bays and dual 750W Platinum PSU.
They cost $1780 CAD each

For Capacity 5 X 4TB WD purple in RAID 5 ( will expand to 10 X 4TB)

Windows 10 Pro

Quad Gigabit Nic

Network for the building is being upgraded at the same time to 10GB/s

Now to the cameras!

All cameras are Dahua 6MP Turret cameras, (except 10 existing Geovision 360)
6MP only because we got a good deal plus everything is well lit

Cameras are Configured to 2MP at 10FPS

CPU is running at 48% at 2.7 Ghz- CPU will turbo to 3.6

There are 12 users that will watch the cameras periodically, at times 4 simultaneously.

System has been up for 2 weeks, currently building server 2 and still installing cameras.

We are on the second phase of 3, by the end of October there will be 82 cameras live.

I attached some pictures, more to come!

The maps don't include Phase 3

I will try to share as much information without compromising security!

 

[SouthernYankee]

Is BI recording direct to disk?
On average what is the camera frame rate?
What is the camera resolution on average (1920x1080) ?
Is BI doing any motion detection and notification ?
Is system connected to the internet ?
Is BI and cameras on a different subnet than other business computers?

 

[smole]

What router, switches and kind of 10gig wiring are you using?

 

[By the Book Networking]

All cameras are direct to disk, 1920X1080, 10FPS.
All cameras are motion detection with no notifications currently set up, Blue iris is providing a time overlay for each camera.
The servers are connected to a private lan with internet access, all cameras are on a private Vlan and subnet

we are using OM3 Multi Mode LC Duplex manufactured by corning.

Ubiquiti switches are being used, no router yet since the there are existing routers and firewalls in place.

We are building the 10G network along side the current network that is only gigabit and at capacity. Once the time is right we will cut over all the existing traffic to the unifi system

It was really interesting pushing Blue Iris and seeing how it preforms under such a large load, and so far it has been 100%

 

[bctrainers]

On the CPU utilization, you appear to be hitting a NUMA-node cap, where Blue Iris is not "seeing" (or using) the second socket or CPU segment. I have observed this issue on a machine with an AMD Threadripper 1950x, be it Blue Iris and other applications that are multithreaded, or have a form of multi-core support - have 32 "cores", but only up to the first 16 end up being used.

It should be noted, this is a coding limitation of the application (Blue Iris), not a fault on your end.

 

[By the Book Networking]

If we crank the cameras to 6mp, the other cpu will activate, the cpu saves power by scaling back.

 

[bctrainers]

Ah, I see. Regardless, this is an impressive use-case. Will be interesting on seeing how Ubiquiti handles the constant port beating over time.

I do have a question though, apologies if i am picking your brain too much! On the task manager, I see you've three ethernet ports active at the time of the capture. Are you not using LAGG on the windows machine, aka what Microsoft/Intel calls Teaming/Teams? I assume each ethernet port is on a different subnet of sorts for the split ethernet?

 

[By the Book Networking]

No worries!, and you got it, different subnets for the 2 camera manufactures, then the local access. There will be redundancy set up in the future since the dell came with a quad nic.
I believe the Ubiquiti will have no trouble handing the load since its not a flat network and all traffic is isolated from LAN, plus the 10G speeds are crazy!!!!
The project just blew up when we started designing the network, turned out a isolated network would of had to been built since their existing wouldn't handle the traffic.
plus the fiber was mandatory with the supply distance exceeding 500'. so it made sense to just upgrade everything

It was a beautiful thing to see 64 cameras on Blue iris, after 2 computer swaps we finailly got it right with the Dell R720XD

 

[bp2008]

Neat. So did Blue Iris actually prevent you from adding a 65th camera on one system? I've always wondered if that was a hard limit or not.

If you end up hooking up multiple high resolution monitors (particularly 4K ones) keep in mind you may need to limit the "live preview" frame rate to keep CPU usage under control. 4K viewing adds a lot of CPU overhead.

 

[By the Book Networking]

Yes, the add camera becomes greyed out, i wanted to keep going lol

and the clients are viewing the cameras remotely, the dell servers do not support high resolution monitors unfortunately. one is currently stuck at 800X600 as i look for a driver that works with windows 10 lol

 

[bp2008]

If you have a PCI-E socket then you can put in a third-party GPU to get proper video support. Of course if these are slim rack mount servers you might not have a slot for that anyway. USB video outputs probably have a lot of CPU overhead...

 

[By the Book Networking]

They are a 2U, I tried a passive GT710 but it was unrecognized by the dell bios. Turns out I need a GPI riser power cord that is dell specific. I might just live with the poor resolution, even if I teamview into the server with no monitor is stuck at the low res...
Ive never played with a usb video card before... sounds interesting lol

(fun fact!)
we performed a test with 10 of the 15 clients viewing all 64 cameras on full res H.264 and the cpu never surpassed 66%
I should message support and see if their is an unlock for the 64 cam cap

 

[Q™]

...we performed a test with 10 of the 15 clients viewing all 64 cameras on full res H.264 and the cpu never surpassed 66%...

How many cameras had been triggered and were recording at this time?

 

[By the Book Networking]

approximately 20-45 cameras at one time are triggered. The test was during production and a good 40 where triggered. on and off of course, but overall little cpu spike.

 

[JonW]

Something else to consider is whether using RAID 5 is wise or not. If I were running this system, I would be looking at either RAID 6 or using an Unraid server with dual parity drives. With RAID 5 and drives that large, your odds of a successful rebuild when a drive fails is not good.

 

[By the Book Networking]

Something else to consider is whether using RAID 5 is wise or not. If I were running this system, I would be looking at either RAID 6 or using an Unraid server with dual parity drives. With RAID 5 and drives that large, your odds of a successful rebuild when a drive fails is not good.

This is one of the few raid deployments I have done, If I do lose a drive, will all other data be lost? or will just the information on that drive be missing?
We are considering of doing a 10G linked NAS to both servers for external backup, the backup currently is not a big priority to the company, the dell servers are currently attached to their monitoring for stability, hopefully dell can see a drive struggling before failure.

Thank you for the advice!
There are always lots of improvements that can be made.

 

[SouthernYankee]

On the disks:
If a raid 5 has a failure, it takes a while for the raid to rebuild. Is the raid using the Main CPU or is it using a raid controller ? If a raid controller have spare disks assigned to the controller for recovery.

I like to keep things real simple and stupid. I have use a raid one, mirror, so each drive pair is independent of the other drive pair. Also in a raid 1 mirror all data is on two disks, each disk can be removed from the system and read on any other PC, this is not true for a raid 5. The disks must be kept as a set.
For Blue Iris assign each drive pair to a Aux folder. Then split / assign/ record the cameras to different aux folders. So any two drive failure will at most lose a few camaras, at best no lost.

 

[rhui_BC]

curious why you didn't go with a server OS since you can't use quicksync?

 

[JonW]

Take a look at this article:
Why RAID 6 stops working in 2019 | ZDNet

I personally feel that RAID 6 is very reliable yet, although I can see where an array of 10-12TB drives might start getting into the dangerous range when a rebuild is needed. I use Unraid on my storage server with dual parity drives. While it is not as fast as a RAID 5 or 6 array, data is not striped across all of the drives so if a perfect storm happened and a rebuild was impossible, only the data on that individual drive would be lost. So far I've lost one of my parity drives and a data drive in separate incidents. Both times the rebuild was simple and worked flawlessly.

 

[By the Book Networking]

we had a bunch of 10 pro keys, and we used the DWS tool that disabled all the forced updates and bloat.
And a windows server licence that would support that many cores would be costly, and its only running blue iris.