Conversion on network (UNC) Aux

[Tayschrenn]

It's possible/probable I'm doing this wrong... But I can't seem to get conversion working the way I'm intending it.

My setup/method:

• 3x4k Dahua cameras sending H.256
  • 2-Bullet w/ this config
    
  • 1-turret with:
    
• Dell Optiplex Micro
  • i3-7100T 2c/4t @3.4ghz w/qsv
  • 16gb RAM
  • Primary Storage: Samsung m.2 256gb (C:\)
  • Secondary Storage: Toshiba 2.5" 1tb HDD (D:\)
  • Aux/long term storage: Synology NAS - around 10TB usable currently
• Networking is handled by Unifi GbE and eventually will have either a Cisco or Juniper Layer-3 for vlan segmentation (TBD)
• Alternate BI Machines:
  • I have an R320 w/dual Xeon E5-1410's inbound that I'm considering using in the future for BlueIris - wife might not like the noise but I can drop 3.5" in it as well as having 16 Xeon threads
  • I also have a laptop being used for F@H with an E3-1505M (including an Intel P630 and a Quadro M2200) which I could use - less sexy but in theory I could offload multiple conversions into different GPUs, not that a quadro is an ideal conversion for video

My 'plan' in my mind for what I want is this:

• Full recording for 1-2 days
• 14 days of 'alert/flagged' videos kept locally @ 4k (ideally on the 1tb but i'm starting to realize that might be asking a lot)
  • This is one area I'm not confident on config - I can't figure out how to handle the retention based on alerting
• 14+ days on the NAS of any alert/flagged clips @ 2k

Conversion issue/process

1. Camera(s) record direct to disk at New (c:\BlueIris\New)
   • Currently recording constantly - motion detection isn't reliable for some reason I've found, so for the time being I'm going overkill
   • This is probably inflating my storage but I haven't fully figured out how to have it only move 'movement' events
   • Also running BlueIris Tools - so the subtitle track encoding as I understand it has to be part of all of this
     ('New' Config)
     
2. At 100gb/2days it sends the data to D:\Storage as BVR
   (Stored Config)
   
   (Convert config)
   
   (export Profile)
   
3. At 800Gb/14 days send to Aux1(Synology via UNC path) and convert to 2k/MP4, using D:\export as the folder for the converted files
   • This is where i'm getting an encoding error, it doesn't seem to be converting them and I'm not sure if it doesn't support network/UNC, or if i'm just configuring it wrong
     (Aux1 Config)
     

Any insight would be really appreciated.

I'm probably doing something wrong, so don't be afraid to call me an idiot =)

 

[Tayschrenn]

Anybody?

 

[Tayschrenn]

Update:
I've changed my two Bullet Dahua's to Trigger + periodic (every 20 minutes it takes a 2-minute recording) in an attempt to minimize my insane amount of ingest.
I've also - for the time being - stopped trying to convert them on the initial move to the D:\ drive - my CPU just can't keep up with the conversion to 1080p, and was five days behind. Plus I found out this morning that in doing that, the next step of 'moving' it to archive moved the BVR files, not the converted files.

I may just need to move this entire thread to a general discussion area for people to tell me what I'm doing wrong in my planning

My two servers came in as well, so this weekend I'll try booting one of them up and see if the noise will make the Wife too angry to use 24/7; worse case I might need to upgrade my NAS drives and ditch the entire concept of converting, keeping what i need raw and on deeper disk sizes.

 

[samplenhold]

Moving data from the BI recording disks to a longer term storage disk is a real resource hog. Originally I did that but found that the machine was constantly moving data. I decided to increase the primary storage and have BI set to delete as space is needed. This cut down my CPU percent (currently running 15% on an i7-8700). I currently have 19 cameras (17 2MP and two 4MP) recording 24/7 (H264) direct to disk at 15FPS on to two 10TB WD Purple drives. I get about 22 days of storage.

As I am about to add five more cameras (two 4MP and three 2MP) and retire two, I have picked up another 10TB WD Purple drive. Two of those 2MP are for LPR and will probably run them at 30FPS. I figure that will give me about 30 days of recordings. That should be plenty for my particular situation. I probably do not need to keep that much recording time, but I do travel a lot and feel I need that much look-back time.

 

[kklee]

Unless you need to have the BI overlays, turn off the re encoding for archiving.

I've configured my cameras so that the text and time overlays that I want are done in camera, so I don't need to use the BI overlays at all.

This allows me to archive the BVR files as MP4 on a NAS without re encoding. The archiving process is super fast this way. BTW, re encoding to a NAS does work, I've tested it.

Re encoding also has a heavy impact on the BI system, in my case it was adding up to 50% CPU usage on an i7-7700K, which maxed out the system at 100% (prior to substreams, love the new feature). Re encoding to a UNC path on a NAS just adds to the load as well since you've now got NAS traffic competing with camera traffic.

 

[Tayschrenn]

Unless you need to have the BI overlays, turn off the re encoding for archiving.

I've configured my cameras so that the text and time overlays that I want are done in camera, so I don't need to use the BI overlays at all.

This allows me to archive the BVR files as MP4 on a NAS without re encoding. The archiving process is super fast this way. BTW, re encoding to a NAS does work, I've tested it.

Re encoding also has a heavy impact on the BI system, in my case it was adding up to 50% CPU usage on an i7-7700K, which maxed out the system at 100% (prior to substreams, love the new feature). Re encoding to a UNC path on a NAS just adds to the load as well since you've now got NAS traffic competing with camera traffic.

I'm on full 1Gb with a backplane large enough to handle both traffic streams, there's little to no chance I'll max that with 3 cameras, even at 4mp. Even when I put my layer-3 switch in, that'll have 2Gbps LACP.

I do use BITools for the temp/weather but I suppose I could live without that for archiving; though I'm not sure why the re-encode/export on the unc volume doesn't work, still.

This substream feature I noted in the release notes, but don't fully understand; I know my Dahau can have a substream, but it's only like... 640x480 which is super low quality - would it be better to just clone my 3 cameras and take in a 2nd of the main stream and immediately encode that to an archive?

 

[kklee]

The substreams are used for console viewing and motion trigger. The actual captured video clips utilize the main streams, so they're full resolution.

It's an easy configuration change, the hard part is figuring out the sub stream URL, but that should be easy to find for a Dahua camera. You'll also have to edit any trigger zones since it changes due to the substream resolution.

Export your camera config before making any changes and restore it if you decide it doesn't do what you want.

This greatly reduces CPU load, especially now that H265 HW acceleration is also working. My system dropped from averaging around 60% CPU to around 8%.

 

[SouthernYankee]

====================================
My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders

Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.

 

[Tayschrenn]

====================================
My Standard allocation post.

1) Do not use time (limit clip age)to determine when BI video files are moved or deleted, only use space. Using time wastes disk space.
2) If New and stored are on the same disk drive do not used stored, set the stored size to zero, set the new folder to delete, not move. All it does is waste CPU time and increase the number of disk writes. You can leave the stored folder on the drive just do not use it.
3) Never allocate over 90% of the total disk drive to BI.
4) if using continuous recording on the BI camera settings, record tab, set the combine and cut video to 1 hour or 3 GB. Really big files are difficult to transfer.
5) it is recommend to NOT store video on an SSD (the C: drive).
6) Do not run the disk defragmenter on the video storage disk drives.
7) Do not run virus scanners on BI folders

Advanced storage:
If you are using a complete disk for large video file storage (BVR) continuous recording, I recommend formatting the disk, with a windows cluster size of 1024K (1 Megabyte). This is a increase from the 4K default. This will reduce the physical number of disk write, decrease the disk fragmentation, speed up access.

Thanks - I'm pretty sure I looked around and couldn't find allocation size recommendations, so I used 64k (I'm a server admin, so in ReFS land that = 64k)
I've AV excluding the paths, and defrag is disabled; only a couple write directly to the SSD but I can move those.

 

[Tayschrenn]

The substreams are used for console viewing and motion trigger. The actual captured video clips utilize the main streams, so they're full resolution.

It's an easy configuration change, the hard part is figuring out the sub stream URL, but that should be easy to find for a Dahua camera. You'll also have to edit any trigger zones since it changes due to the substream resolution.

Export your camera config before making any changes and restore it if you decide it doesn't do what you want.

This greatly reduces CPU load, especially now that H265 HW acceleration is also working. My system dropped from averaging around 60% CPU to around 8%.

I'll have to dig into this more; I do know that the i3-7100T does support H265 decode, so that's good! Interesting idea to use the substream to trigger on motion - I'll also look into that.