Choosing hardware setup for BI on Threadripper 2950X system with nVidia

prd0000

n3wb
Joined
Sep 6, 2020
Messages
8
Reaction score
3
Location
Jakarta
Good day members, my first post here..

I have just bought BlueIris, after using iVMS4200 from Hikvision.

I have a Threadripper 2950X idling at about 2-3% utilization for my Remote Desktop host most of the time. So I plan to add BlueIris to this computer.
I've seen that BlueIris can use NVEnc, as well as NVDec. Since I will be using Threadripper system, QSync is obviously out of question.

The Threadripper has 64 GB or RAM equipped with NVMe 3GBps R/W for local recording.
It currently has nVidia 9400GT for its display, thus I think it is very inadequate for Blue Iris.

We have about 43 camers, with 15 are 5MP, and 26 are 2MPs, and 2 are 3MP 360 degrees fisheye.
All of them are HikVision, running at 10fps, with about 800Kbps max bit rate.

The monitor window will be off, all of the time, but there is a security guy who will always monitor 10 of our cameras via web browser.

My questions are:
1. I plan to buy 1650 SUPER for this setup, since 9400GT is obviously not enough. But would 1650 SUPER be enough? I have considered 2060, but after seeing the spec on 1650 vs 2060 decoders, I think its stream properties are about the same. What do you think? Will moving to 2060 be helpful?
2. Is my Threadripper enough for this setup? Or should I buy an Intel system for this need?
3. Will monitoring via web browser be good enough? Or should I change to another application?

Thank you for helping.
 
Last edited:

SouthernYankee

IPCT Contributor
Joined
Feb 15, 2018
Messages
5,170
Reaction score
5,320
Location
Houston Tx
Test do not Guess.

I would build the system without the 1650 SUPER. I would get it up and running, tuned to use the substreams. Then see what the system is doing. My concern is that the disk drives are configured. I would have at least 3 drives (more drives better) configured for video, set 15 cameras per disk. The bigger the drives the better.

Writing video continuously from so many cameras will kill an NVMe or SSD. I would not risk my system drive for video writes. There is a limited number of writes. Moving Video for New to stored wastes system resources.

================================================

My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders
8) an alternate way to allocate space on multiple drives is to assign different cameras to different drives, so there is no file movement between new and stored.
9) Never use an External USB drive for the NEW folder. Never use a network drive for the NEW folder.


Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.
Hint:
On the Blue iris status (lighting bolt graph) clip storage tab, if there is any red on the bars you have a allocation problem. If there is no Green, you have no free space, this is bad.
 
Joined
Apr 26, 2016
Messages
1,090
Reaction score
852
Location
Colorado
I'm always too verbose, here's my points summarized.
  1. Don't record to the NVME, you will kill it dead and fast. Get some spinning drives and round-robin the cameras to spread the workload. (oop, @SouthernYankee beat me dangit)
  2. You likely don't need a new GPU, and any consumer GPU will likely hit a vendor imposed driver limits around 24 cameras. See: EPYC Overkill Build
  3. Whether you use sub-streams or not you have enough CPU power for 1400 MP/s or more. See: EPYC Overkill Build (easily 3300 - 6500 MP/s on similar CPU)
  4. Consider whether you want to use your high end machine like this, when you could be running a cheap desktop in a closet recording security video.
  5. If you use sub-streams you really don't appear to need a powerful computer anymore for Blue Iris.

Just be aware that I ran into what I believe is a driver limitation when I tried to add either 13 video streams to a single card (or 25 total) to TWO GTX1660 SUPER configuration here: EPYC Overkill Build so you could have issues leveraging GPU acceleration unless you were to get their Quadro series but you shouldn't need it (RTX3090 may support this since it's supposedly a "Titan Class" GPU, which a couple Tech Youtubers said was total BS). Regardless, you likely DON'T NEED ANOTHER GPU.

Further, if you can use the sub-stream feature in BI5 you won't need anything nearly that powerful imho for what I estimate is 1400 MP/s of video.

When I went CPU-only mode with sub streams I simply didn't have enough cameras to go beyond 53 cameras on Blue Iris 5.2.9.18 x64 (running as service), but I reached 6450 MP/s at 45% CPU and 154 watts system power draw on an EPYC 7302P CPU. My EPYC 7302P is just the "server" version of your Threadripper 2950x (same cores/threads but slower clocks than 2950X). The EPYC does support 8-memory channels compared to the 2950X only supporting 4-memory channels, but I found 4-channel to be adequate to reach about 3300 MP/s without substreams.

If you can use sub streams then that system is probably good for over 6000 MP/s without any GPU acceleration. You need a GPU obviously to drive a display, but I would simply choose based on the display I plan to connect directly to the BI machine. For the UI interface to work I don't believe it even matters, but check with @bp2008, that 9400GT card can handle a 2K display max.
 
Last edited:

bp2008

Staff member
Joined
Mar 10, 2014
Messages
12,666
Reaction score
14,005
Location
USA
Hello.

You're right, the 9400GT is inadequate for hardware acceleration. But you don't need hardware acceleration. That set of cameras and frame rates should come to about 1330 megapixels per second. It should run without issue on the threadripper with only software decoding. Hardware encoding is also unnecessary.

If you don't like how much CPU load it creates, then configure sub streams for some or all of the cameras. That is one of the features described in Choosing Hardware for Blue Iris as linked by @looney2ns.

So to answer your specific questions,
1) No need for a new graphics card at all.
2) The threadripper is more than enough.
3) Monitoring via web browser is probably good enough. You will probably want to increase and fine-tune the resolution of your group streams. This screenshot shows where to find those settings.
 

prd0000

n3wb
Joined
Sep 6, 2020
Messages
8
Reaction score
3
Location
Jakarta
Yes, I've read it. But it didn't details nVidia encoding and decoding. And I was trying to decide which GPU to replace my 9400GT. The guide was too focused on Intel systems. And I don't plan to buy a new PC to record my camera.

Test do not Guess.

I would build the system without the 1650 SUPER. I would get it up and running, tuned to use the substreams. Then see what the system is doing. My concern is that the disk drives are configured. I would have at least 3 drives (more drives better) configured for video, set 15 cameras per disk. The bigger the drives the better.

Writing video continuously from so many cameras will kill an NVMe or SSD. I would not risk my system drive for video writes. There is a limited number of writes. Moving Video for New to stored wastes system resources.

================================================

My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders
8) an alternate way to allocate space on multiple drives is to assign different cameras to different drives, so there is no file movement between new and stored.
9) Never use an External USB drive for the NEW folder. Never use a network drive for the NEW folder.


Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.
Hint:
On the Blue iris status (lighting bolt graph) clip storage tab, if there is any red on the bars you have a allocation problem. If there is no Green, you have no free space, this is bad.
Hmmm.. that raised another problem then.. Ok.. about the drive then..
The threadripper doesn't have any hard drive in it. For every other purposes, it is connected to StarWind SAN via 10GBe. I've allocated 30TB disk from the array for its use, but it had some problem. The record time was a little off. All events are recorded about 1 ot 2 seconds too late. For example, in a room with a door, it should start to record the moment the door opened. But the record started when I was about a few meter walks the room. Anyway, the drive is not connected using file sharing, but it is a iSCSI drive, using MPIO. I've tested an RAM Disk Target and it was topped at about 1800MB/s, while the hard drive target topped at about 150MB/s. Well, it was about 400MB/s, then dropped to 150MB after a while, and stays there. I guess the SAS cache has it full. I can't use RAMDisk as my NEW Folder since I think it would be too risky. So I tried to record to NVMe drive, and didn't experienced the delay problem, but I havent run it as long as the ISCSI hard drive one.

The NVMe is also not a C Drive. The Threadripper was my development server, and it was heavily underutilized. So I wanted to convert this machine to record CCTV. Therefore it has 3 NVMe drives. I've moved my files from the last drive to make way for CCTV records. It is rated at 320TBW. The files would then be moved to the iSCSI drive. I don't mind buying another NVMe after 2 years or so. It is not that expensive anyway, and it would still be in warranty. The one I use for record is about $50. But I need the files moved to the iSCSI drive immediately, since it is the one that is protected. I originally wanted to write to the iSCSI drive directly, but well.. got some record delays.

What do you think causing the delay? I've tried to set the record to the most sensitive one for motion detection. I first thought it was hard drive, so I moved to NVMe drive. But I didn't discount the GPU probability either.

I'm always too verbose, here's my points summarized.
  1. Don't record to the NVME, you will kill it dead and fast. Get some spinning drives and round-robin the cameras to spread the workload. (oop, @SouthernYankee beat me dangit)
  2. You likely don't need a new GPU, and any consumer GPU will likely hit a vendor imposed driver limits around 24 cameras. See: EPYC Overkill Build
  3. Whether you use sub-streams or not you have enough CPU power for 1400 MP/s or more. See: EPYC Overkill Build (easily 3300 - 6500 MP/s on similar CPU)
  4. Consider whether you want to use your high end machine like this, when you could be running a cheap desktop in a closet recording security video.
  5. If you use sub-streams you really don't appear to need a powerful computer anymore for Blue Iris.

Just be aware that I ran into what I believe is a driver limitation when I tried to add either 13 video streams to a single card (or 25 total) to TWO GTX1660 SUPER configuration here: EPYC Overkill Build so you could have issues leveraging GPU acceleration unless you were to get their Quadro series but you shouldn't need it (RTX3090 may support this since it's considered a "Titan Class" GPU). Regardless, you likely DON'T NEED ANOTHER GPU.

Further, if you can use the sub-stream feature in BI5 you won't need anything nearly that powerful imho for what I estimate is 1400 MP/s of video.

When I went CPU-only mode with sub streams I simply didn't have enough cameras to go beyond 53 cameras on Blue Iris 5.2.9.18 x64 (running as service), but I reached 6450 MP/s at 45% CPU and 154 watts system power draw on an EPYC 7302P CPU. My EPYC 7302P is just the "server" version of your Threadripper 2950x (same cores/threads but slower clocks than 2950X). The EPYC does support 8-memory channels compared to the 2950X only supporting 4-memory channels, but I found 4-channel to be adequate to reach about 3300 MP/s without substreams.

If you can use sub streams then that system is probably good for over 6000 MP/s without any GPU acceleration. You need a GPU obviously to drive a display, but I would simply choose based on the display I plan to connect directly to the BI machine. For the UI interface to work I don't believe it even matters, but check with @bp2008, that 9400GT card can handle a 2K display max.
Wow.. this is the answer I need. So.. I don't need to replace my 9400GT? LOL.. that's a huge relief. Been days researching about nvidia encoding/decoding capabilities, and how many camera it can handle. Actually, the machine is headless, thus the 9400GT choice. I connect to the machine via remote desktop, and if something happens, I would connect a monitor to it. Security guy will connect to this machine via web browser. But it also means, my record delays problem lies elsewhere.

Hello.

You're right, the 9400GT is inadequate for hardware acceleration. But you don't need hardware acceleration. That set of cameras and frame rates should come to about 1330 megapixels per second. It should run without issue on the threadripper with only software decoding. Hardware encoding is also unnecessary.

If you don't like how much CPU load it creates, then configure sub streams for some or all of the cameras. That is one of the features described in Choosing Hardware for Blue Iris as linked by @looney2ns.

So to answer your specific questions,
1) No need for a new graphics card at all.
2) The threadripper is more than enough.
3) Monitoring via web browser is probably good enough. You will probably want to increase and fine-tune the resolution of your group streams. This screenshot shows where to find those settings.
Thank you... Yes.. I was worried about that since I had a problem. I deducted that it was hard drive problem, since my drive only topped at 150MB/s. But I also suspect the problem lies with 9400GT. Either one or both should be causing problem. But now I begin to think the problem lies elsewhere.
 

OICU2

BIT Beta Team
Joined
Jan 12, 2016
Messages
821
Reaction score
1,330
Location
USofA
For your delay issue, try increase the pre-trigger video buffer on the "Record" tab of the specific camera:

BI pretrigger.png
 

biggen

Known around here
Joined
May 6, 2018
Messages
2,539
Reaction score
2,765
As others have said, the Threadripper will have plenty of horsepower especially when using substreams. Substreams are truly a game changer.

Blue Iris wasn't/isn't designed to really write to a SAN/NAS. It will, but then problems can arise as you have found out. No idea exactly where the problem with the delay is. Could be an iSCSI issue. Have you tried writing to a NFS share?

Can you not just add a few large spinners to the Threadripper workstation? That would save a lot of headaches. Stuff it full with as many/large drives you can and off you go. With 40-ish cameras you will want to split up drives so that some cameras write to certain drives while others write to other drives.

I will say that you are getting toward the "upper end" of what Blue Iris was designed for with this many cameras.
 

prd0000

n3wb
Joined
Sep 6, 2020
Messages
8
Reaction score
3
Location
Jakarta
Thank you everyone. I've tried all your suggestion. I installed a magnetic drive into the Threadripper server, and it seem solved the problem, but not entirely. It was better though than SAN drive. Adding another "pre-trigger video buffer" and switched algorithm to "simple" solved the problem. I can move the clip to SAN drive after record though.
 
Last edited:
Top