Inconsistant Performance for BlueIris UI3

[eangulus]

Just for background, I have been running my own BI setup for about 4 years now with 9 1080p cameras. I have also been installing for clients on minor systems for business using around 4-8 cameras total all 1080p. I know the usual tweaks, add-ons like UI3, BI Tools etc. And have gone thru all the performance tips etc to squeeze every inch of performance out of the systems.

The latest setup thou has me pulling my hair out.

The new setup is not quite 100% complete, but will total:

• 19 x 1080p 12FPS@2048 2032 Hikvisions - 13 Installed
• 1 x 1080p 24FPS@2048 Mini-PTZ - 1 Installed
• 1 x 2160p 24FPS@4096 2085 Hikvisions - 0 Installed
• 5 x 2160p 12FPS@4096 2085 Hikvisions - 5 Installed

The NVR is as follows:

• i7 8700K Standard Clocking
• 16Gb RAM
• 250Gb SSD
• 1TB SATA
• Blue Iris
• Blue Iris Tools
• BI UI3

Now my issues:

When first setting up, in the first few days, I was getting great results on UI3. I was getting 12FPS viewing all cameras. I would also get 12FPS view one camera (have group settings capped at 12FPS and 1024 x 720).

But now I have added all cameras (I even added the, to be installed ones and have disabled those for now), I am getting weird results. Nothing is consistent where sometimes I just reboot or restart BI and the results change.

Sometimes I get UI3 reporting 12FPS on all cameras group, but you can clearly see it isn't. Then clicking on a camera shows it as 1FPS. Other times I get 3-4 FPS on all camera group, and full 12FPS on each camera alone.

I originally thought that maybe the graphics drivers, which I made the patch to the Intel Drivers as instructed in the Wiki to prevent the memory leak. I have all cameras using GPU decoding just fine, can see 40-50% GPU usage.

CPU usage is also a weird one, most of the time I get 40-50% CPU with BI open and dropping to 20-30 with it closed. I get around 20-30% when running as a service. But sometimes I get 1% CPU usage, and when looking at the Task manager it seems that sometimes it stops using multicore and starts using a single core when showing very low CPU. If I restart BI when like this it sometimes stays only using 1 or 2 cores and really low CPU and after another restart, it might go back to normal 40-50% and multicore again.

RAM usage is only at around 8Gb of the 16Gb so that's never an issue. and I have checked and am not having throttling issues either.

For recording I have Direct to Disk, to the SSD then convert files while moving to the SATA, then after move to a Dual Synology 12 Drive NAS HA setup.

I also have only 2 profiles, Day & Night based on Sunrise and Sunset. Daytime I have all cameras continuous recording, no motion detection or anything. At night, I have all indoor cameras (currently 15) and recording on motion only, and all outdoor ones (currently 4 4K ones) recording continuously still, no change for daytime. No change in performance between the 2 to clearly show that a profile setup is the issue either. A couple of times thou, I have had the PC go 100% CPU and stay there when profiles change. A restart of BI seems to fix that untill the next profile change.

All I want is good frame rates on UI3 for viewing and I can get them sometimes I cannot seem to make it consistent, nor can I make a particular change to settings and get a clear repeatable issue.

Another smaller but still annoying issue is that on 1 camera I have setup line crossing detection in the camera, and it triggers an alert on BI. It is supposed to post an image to a folder, then calls a webhook I built that in turns sends the office PC's a Chrome Notification that someone has entered the entry area and shows a snap from that notification. Thou I had it working on the exact same settings prior on a smaller NVR before this major upgrade, I now cannot get BI to post the images to either Local Drive, Mapped NAS folder or a Direct Network UNC path. It sometimes will but I am talking maybe 1 in a dozen forced triggers (to test). I get same results using FTP upload instead, maybe slightly more hits that misses but still not enough of a difference to clearly say that one has a problem and the other doesn't. Seems that the whole posting an image has a problem.

Can anyone please help. I am running out of ideas to test and check and am thinking that maybe someone on the outside may have fresh eyes.

BTW: I am willing to give access to someone if they need it to take a deeper look into the settings etc.

Forgot to metion: We have 100Mb Down 40Mb Upload Fibre internet connection. I have also got the connection for NVR setup with QoS and such to make it a priority over everything else barring phones. I have also tested with and without QoS turned on and no difference. I get the above issues both locally on the 1Gb network and from Home where I also have the 100/40 Fibre.

 

[bp2008]

Can you test the LAN performance with a file transfer or something to make sure you're getting the speed you should be?

I calculate that the installed camera load above is about 870 MP/s which is almost exactly what I have going at home on the same CPU with (roughly) the same CPU and GPU usage numbers as you. However UI3 works consistently fine on this system. Even my all cameras group which I have set to 2488x1088 resolution @ 15 FPS streams fine. I should note though that if I open the Blue Iris console fullscreen on a 4K monitor it does increase CPU usage significantly and this reduces the group view frame rate in UI3 to about 7-9 FPS. Other single cameras of course are locked at 1280x720 for H.264 streaming to UI3 and at this lower resolution they never have a problem matching whatever frame rate the camera is providing.

If you're seeing 12 FPS for the all cameras group but it doesn't look like 12 FPS, it is possible that your network is behaving poorly and sending out frames in bursts. UI3 only tries to buffer 0-1 frames ahead so any network interruption is likely to affect the stream. Or you could be streaming in jpeg mode which de-couples the streaming frame rate from the source frame rate. What resolution and FPS did you assign to the All cameras group? It is 5 FPS by default.

When you have the files get automatically converted when they are moved, are you allowing the video to be re-encoded? This would be a huge performance hit, about equivalent to turning off direct to disk, and I definitely do not recommend it.

I've never seen so much as a CPU spike when changing profiles, so I really have no clue why it would sometimes get stuck at 100% until you restart BI.

I've also never see BI fail to initialize and be stuck at 1%.

 

[eangulus]

Hence the problem.

I too have never seen the issues I am having with this system.

As already stated I have set the group's to 720 and 12FPS.

Also network is not a problem, this is all on enterprise grade Nortel switches which is monitored closely as it is also used for iSCSI for 2 Virtual Servers.

I even see the network transfers hit the 850+ in task manager, but will do some more tests there just to be sure.

I understand the performance hit with converting formats and after I wrote the post I have turned off any conversions for now. So all my storage is only in bi format.

I am thinking that maybe the CPU is overheating and throttling but can't see it happening when I look. It does have the stock cooler on it but am replacing that next week with a better copper cooler.

Sent from my Pixel XL using Tapatalk

 

[bp2008]

i7-8700K doesn't have a stock cooler...

 

[bp2008]

Sorry I missed the note about your group resolution and frame rate limit. That is definitely doable, I mean my own group view is higher frame rate and over twice the resolution and it works.

 

[eangulus]

i7-8700K doesn't have a stock cooler...

Then what is the Intel fan and heatsink that's sitting on it come from? Has a big Intel stick on it. It's what came with the CPU.

Either way I have read tests the changing it to what I am putting on next week can give around a 20C drop. It currently hovers around the 80C's.

Just a question too, am I better off setting the administrator account and using that for the setup, or it doesn't really matter?

Also am I better off on Windows 7 or 10?

I am thinking at this stage to start the installation again just to make sure something else hasn't gone all weird.

Sent from my Pixel XL using Tapatalk

 

[bp2008]

Maybe the CPU is actually i7-8700 (non-K). The K models haven't come with heatsinks for years.

80-90C should not introduce throttling. That is only supposed to happen when you hit the limit which is a bit higher. Still it might be good for longevity to let it run cooler.

It doesn't matter which account you use to set up Blue Iris.

You can't use hardware acceleration when BI is running as a service in Windows 7, but otherwise there shouldn't be much difference. Windows 7 is a little easier to get along with regarding the updates and reboots, but Windows 10 will be supported much longer.

If you're going to start over, I recommend creating new installation media (USB) using the media creation tool.

 

[eangulus]

Maybe the CPU is actually i7-8700 (non-K). The K models haven't come with heatsinks for years.

80-90C should not introduce throttling. That is only supposed to happen when you hit the limit which is a bit higher. Still it might be good for longevity to let it run cooler.

It doesn't matter which account you use to set up Blue Iris.

You can't use hardware acceleration when BI is running as a service in Windows 7, but otherwise there shouldn't be much difference. Windows 7 is a little easier to get along with regarding the updates and reboots, but Windows 10 will be supported much longer.

If you're going to start over, I recommend creating new installation media (USB) using the media creation tool.

It's 100% a K version. It's written on the CPU when I reapplied better thermal compound.

With the install media if it's to make sure nothing else is installed then I already used it. I used a USB key I made as we got the PC bare and we bought a 10 pro key.

Sent from my Pixel XL using Tapatalk

 

[eangulus]

Just checking it out but seeing K CPUs don't come with coolers I was curious, seems they just put an 8700 Intel cooler on it.

And when I said it sits around 80, it has a cap of 100, and I get the 80 with bi as a service and UI closed. Basically bare minimum everything with bi running. No UI3 or anything else running.

So it wouldn't be a long shot to think that when streaming, moving files, accessing ui etc it may rise to the 100 and start throttling.

Sent from my Pixel XL using Tapatalk

 

[bp2008]

Yeah the K version has a 95 watt TDP and the non-K has only a 65 watt TDP too http://a.co/g9HvVp7 is a good option. Wouldn't be surprised if that is what you chose. Though it won't fit in a lot of smaller cases.

 

[eangulus]

Actually am tight on space. This is all mounted in a 2U Rackmount and have 70mm clearance from top of CPU to lid.

So I picked this: NH-L9x65

65mm but I have 4 large fans blowing thru the case too.

Sent from my Pixel XL using Tapatalk

 

[eangulus]

So I put the new Cooler on, and it made around a 20C difference. No difference to the issues but.

I have noticed that when BI is set as a Service I am more likely to get UI3 all cam view saying 12fps when it's clearly not and then a single camera at 1fps. When I switch it back to a normal program, I get 3fps on all cams and 12fps on a single view.

CPU is still sitting around 30-40% and GPU is around 50%.

Is it possible this could be a driver issue of some sort like for the GPU or something?

I tested the BI web interface too and it seems to work much smoother and faster than UI3, but UI3 is nicer so I want to keep it.

Next week I will back everything up and start from scratch with a clean install, and test along the way to see if I can find where and/or when I hit the issue.

 

[eangulus]

Seem to be the sheer number of Cameras.

I disabled all 4K cameras and got better FPS (8ish). I then re-enabled and disabled an equal number of 1080 cameras, the same result so it's not the 4K specifically.

But I don't understand why it is doing this. I am sitting at 43% CPU with BI minimised and UI3 running. I get same results on any computer I run UI3 on from i7-920 thru to Dual Quad Core Xeon Workstations.

The system is not stressed, I am not even reencoding anything anymore, direct to disk (SSD) then moved to SATA then moved to NAS.

No triggers, no motion, no pre-recording. No FPS capping, have set Groups to 720 and 12FPS. No Live limit (other than 5sec for RDP).

There is obviously a limitation somewhere but can't seem to find it. Also I have tested UI3-4 UI3-6 and UI3-7. No change. Will see if I have an old copy of UI2 just so I can eliminate where the bottleneck is.

 

[bp2008]

It is unlikely that UI3 is the bottleneck, especially if reducing the number of active cameras improves performance, because UI3's video decoder knows nothing and cares nothing for the number of cameras involved.

What do your streaming profiles look like? Make sure you have a bit rate limit set for any H.264 streaming profiles. Mine works great like this:

(Streaming 0, 1, and 2 in Blue Iris match Streaming 0, 1, and 2 in UI3)

 

[eangulus]

It is unlikely that UI3 is the bottleneck, especially if reducing the number of active cameras improves performance, because UI3's video decoder knows nothing and cares nothing for the number of cameras involved.

What do your streaming profiles look like? Make sure you have a bit rate limit set for any H.264 streaming profiles. Mine works great like this:

(Streaming 0, 1, and 2 in Blue Iris match Streaming 0, 1, and 2 in UI3)

Mine nearly exactly the same. Only diff is I have 50 instead of 90 for quality on all 3.

Bit rated I use in order are no limit, 2048 and 1024. But I get no difference regarding frames on any of the 3.

I will double check it thou, just to make sure. I kinda wish I was making out the PC, at least that would be a reason for all this crap.

Sent from my Pixel XL using Tapatalk

 

[eangulus]

I think I am starting to find where the problem is.

After doing some more tweaks to get everything consistent and smooth, I have now noticed that when I view ALL 25 Cameras via UI3 the CPU usage goes from 20% to 100%. Should it be hammering the system that hard? And what can I do to minimise the impact?

 

[bp2008]

Did you turn on the feature "Limit decoding unless required" on all/most of your cameras? That gets turned off for cameras while they are being viewed remotely. It is the one thing I know of that could explain that drastic of an increase in CPU usage.

 

[eangulus]

Yes. Will go thru and turn them off and test again.

Just for more information, There is a massive issue between running as an app vs service.

As a normal program, I get around 20% CPU idle (no viewing or anything). In UI3 I get low frames for all cameras (3-4) then I get full frame rate on each camera view (min 12FPS some are 20FPS and they also show full rate on single camera view).

When I change BI to a Service, I get around 8-10% CPU idle, then in UI3 I get 12FPS in All Camera view, and max 2 FPS in Single camera view.

I even tried setting the service to run as the Administrator account and got no change.

Seems to be an issue with decoding the video between Service or Program modes.

I'll report back in a minute when I change the Limit Decoding.

 

[eangulus]

OK turned off all Limit decoding.

Now I get the same result as described above for a service, but this time in either service mode or program mode.

Also get a message saying my CPU isn't fast enough yet, the Server is 8700K @ 30-40% and my PC is i7.

 

[bp2008]

The CPU not fast enough message means more frames are coming into UI3 than it can decode. Typically it indicates a slow client but sometimes it can come up if there was a long network interruption that caused a lot of video to get buffered and arrive in a big burst.