Cameras Go Offline Randomly, Changing On Each Restart

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
Checked again today, 12 cameras down "No signal".

4 Reolink turret cams.
4 Reolink bullet cams.
1 Reolink dome.
1 Hikvision pinhole cam.
1 Hikvision bullet cam.
1 Hikvision turret cam.

Given that Hik cams are going down as well, I can't blame this on Reolink.

I closed the app again, stopped the service, and started the service. As usual, everything came back up immediately, however two cams stayed offline. I stopped the service again, restarted it again, this time different cams were offline. Finally I stopped the service, waited a few minutes, started the service, some cameras were still offline. I tried to restart one of the cameras manually, then restart it in Blue Iris. The camera is remained in a "No signal" state until about two minutes later when it finally kicked back on by itself.

I will check again in the morning to see if some are still offline and will log any additional weirdness here for reference as I work towards solving this.
have you exluded blue iris program and storage folder from antivirus?
 

TL1096r

IPCT Contributor
Joined
Jan 28, 2017
Messages
1,223
Reaction score
465
Checked again today, 12 cameras down "No signal".

4 Reolink turret cams.
4 Reolink bullet cams.
1 Reolink dome.
1 Hikvision pinhole cam.
1 Hikvision bullet cam.
1 Hikvision turret cam.

Given that Hik cams are going down as well, I can't blame this on Reolink.

I closed the app again, stopped the service, and started the service. As usual, everything came back up immediately, however two cams stayed offline. I stopped the service again, restarted it again, this time different cams were offline. Finally I stopped the service, waited a few minutes, started the service, some cameras were still offline. I tried to restart one of the cameras manually, then restart it in Blue Iris. The camera is remained in a "No signal" state until about two minutes later when it finally kicked back on by itself.

I will check again in the morning to see if some are still offline and will log any additional weirdness here for reference as I work towards solving this.
Yes I notice this too. When blue iris loses connection it goes down it stays down for awhile even if video feed is back up. For example my internet goes down and it will give socket errors and 10 minutes later it will go back online even though the feed will work in other apps. That seems to be 1 issue you have where you reset and stop as service and it comes back-up vs taking forever to reconnect. I wonder why that is - it seems silly that BI cannot pickup the feed as soon as it is backup?
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
I notice with this blue iris when it goes down it stays down even if video feed is back up. For example my internet goes down and it will give socket errors and 10 minutes later it will go back online even though the feed will work in other apps. That seems to be 1 issue you have where you reset and stop as service and it comes back-up vs taking forever to reconnect. I wonder why that is - it seems silly that BI cannot pickup the feed as soon as it is backup?
This is not accurate. You have something malfunctioning. None of these steps are needed. More importantly, if your internet goes does, your cameras should stay up as they dont rely on an internet connection.
 

TL1096r

IPCT Contributor
Joined
Jan 28, 2017
Messages
1,223
Reaction score
465
This is not accurate. You have something malfunctioning. None of these steps are needed. More importantly, if your internet goes does, your cameras should stay up as they dont rely on an internet connection.
... what steps? I don't do anything until it comes back up. This setup I had was pulling feed from reolink NVR which I am talking about. Video feed is up and fine on NVR/And software on computer but BI is still out for easily 10-20 minutes with socket error even though video is getting through other ways. But my new setup with dedicated bi machine no issues. I just talking about past experience and it is similar to Flash6199. I know Flash6199 has different setup I just noticed same thing he is saying about BI not pulling video feed even though it is there until he reset/stopped as service. I just don't get why when video feed is able to be pulled from crap software that BI says no feed socket error or blank.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
... what steps? I don't do anything until it comes back up. This setup I had was pulling feed from reolink NVR which I am talking about. Video feed is up and fine on NVR/And software on computer but BI is still out for easily 10-20 minutes with socket error even though video is getting through other ways. But my new setup with dedicated bi machine no issues. I just talking about past experience and it is similar to Flash6199. I know Flash6199 has different setup I just noticed same thing he is saying about BI not pulling video feed even though it is there until he reset/stopped as service. I just don't get why when video feed is able to be pulled from crap software that BI says no feed socket error or blank.
He is not using an NVR. His issue is completely different. In your case, the "crap software" from reolink is using different protocols than BI. If BI is not getting a feed, its not getting a feed. steps like rebooting.
 

TL1096r

IPCT Contributor
Joined
Jan 28, 2017
Messages
1,223
Reaction score
465
He is not using an NVR. His issue is completely different. In your case, the "crap software" from reolink is using different protocols than BI. If BI is not getting a feed, its not getting a feed. steps like rebooting.
ok let me step out :) you answered me. That makes much more sense. I hope Flash6199 finds his fix with your help. Thanks
 

Flash6199

n3wb
Joined
Jul 11, 2019
Messages
15
Reaction score
3
Location
USA
Tonight in an interesting turn of events. I found several cameras still with "No signal" from this morning. I noted that right clicking the cams, and selecting "Restart camera" did nothing, it did not even show the multi colored placeholder image. Disabling and Enabling had the same effect, or lack there of. I closed the Blue Iris UI, and attempted to re open it. It appeared for a second and disappeared. Upon inspecting the process in Task Manager, I found it was using over 18GB of RAM from under 24 hours of run-time. I killed the service to free RAM, and noted it went to 4GB used. The UI was then able to open properly. Several cameras are still at "No signal" status. This feels stranger than the normal run of the mill connectivity issues I've described so far.

I'll post an update if anything changes drastically. It's worth noting, more cameras are online now than there was during the past 48 hours. I have made no network, or configuration changes.

At this point I'm tempted to backup the Blue Iris config, nuke the servers OS, upgrade to Windows 10, install blue iris, restore the config, and upgrade blue iris to 5. I would love to have this issue fixed before taking these steps though, especially if its corruption that could somehow carry over through the config.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
Tonight in an interesting turn of events. I found several cameras still with "No signal" from this morning. I noted that right clicking the cams, and selecting "Restart camera" did nothing, it did not even show the multi colored placeholder image. Disabling and Enabling had the same effect, or lack there of. I closed the Blue Iris UI, and attempted to re open it. It appeared for a second and disappeared. Upon inspecting the process in Task Manager, I found it was using over 18GB of RAM from under 24 hours of run-time. I killed the service to free RAM, and noted it went to 4GB used. The UI was then able to open properly. Several cameras are still at "No signal" status. This feels stranger than the normal run of the mill connectivity issues I've described so far.

I'll post an update if anything changes drastically. It's worth noting, more cameras are online now than there was during the past 48 hours. I have made no network, or configuration changes.

At this point I'm tempted to backup the Blue Iris config, nuke the servers OS, upgrade to Windows 10, install blue iris, restore the config, and upgrade blue iris to 5. I would love to have this issue fixed before taking these steps though, especially if its corruption that could somehow carry over through the config.
The latest .65 release fixed some blue iris memory leak issues. See wiki for proper intel drivers to avoid intel based memory leaks.
 

Flash6199

n3wb
Joined
Jul 11, 2019
Messages
15
Reaction score
3
Location
USA
The latest .65 release fixed some blue iris memory leak issues. See wiki for proper intel drivers to avoid intel based memory leaks.
I tried following the drivers marked on the wiki for haswell cpus. I was unable to install either recommended drivers as this is a Windows 7 installation, and Intel only supports 10 for gen 4 cpus on that driver version.

Currently version 10.18.14.4264 is installed by Windows and is signed by Microsoft.

Almost every camera was offline tonight when I checked in on it. It's getting harder and harder to keep this running it seems. Killing the service and restarting it again, brought almost everything back. I manually restarted an offline camera and noted it active in Blue Iris. Then as I was looking at it, without warning it went into "No signal" state with a screenshot of its last image. It had only bee online for a few seconds at most. The entire time ping never failed for that camera. I then manually restarted the camera again, restarted the feed several times in Blue Iris, and was finally able to get a video feed.

Does Blue Iris have any more extensive logs other than "No signal" ? It would be useful to see a more verbose error message if one exists.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
I tried following the drivers marked on the wiki for haswell cpus. I was unable to install either recommended drivers as this is a Windows 7 installation, and Intel only supports 10 for gen 4 cpus on that driver version.

Currently version 10.18.14.4264 is installed by Windows and is signed by Microsoft.

Almost every camera was offline tonight when I checked in on it. It's getting harder and harder to keep this running it seems. Killing the service and restarting it again, brought almost everything back. I manually restarted an offline camera and noted it active in Blue Iris. Then as I was looking at it, without warning it went into "No signal" state with a screenshot of its last image. It had only bee online for a few seconds at most. The entire time ping never failed for that camera. I then manually restarted the camera again, restarted the feed several times in Blue Iris, and was finally able to get a video feed.

Does Blue Iris have any more extensive logs other than "No signal" ? It would be useful to see a more verbose error message if one exists.
I would wipe the system clean and start over. You can enable the log file in blue iris status but i doubt it will be more detailed.
 

Flash6199

n3wb
Joined
Jul 11, 2019
Messages
15
Reaction score
3
Location
USA
I would wipe the system clean and start over. You can enable the log file in blue iris status but i doubt it will be more detailed.
Just wanted to provide an update. I've upgraded this installation to Blue Iris 5, which still seems to have the memory leak issue, but seems to have stopped cameras from randomly dropping. I played around with an issue where certain reolink cameras would not come back after restarting Blue Iris. I found this was because those cameras were set to use RTMP and not RTSP. I moved all reolink cameras to RTSP and they are significantly more stable and reliable.

I am currently working towards getting a Windows 10 upgrade so I can install recommended drivers to fix the memory leak issues.
 

fenderman

Staff member
Joined
Mar 9, 2014
Messages
36,897
Reaction score
21,250
Just wanted to provide an update. I've upgraded this installation to Blue Iris 5, which still seems to have the memory leak issue, but seems to have stopped cameras from randomly dropping. I played around with an issue where certain reolink cameras would not come back after restarting Blue Iris. I found this was because those cameras were set to use RTMP and not RTSP. I moved all reolink cameras to RTSP and they are significantly more stable and reliable.

I am currently working towards getting a Windows 10 upgrade so I can install recommended drivers to fix the memory leak issues.
Rtmp was implemented for the reolink cameras as a solution to the I frame interval problem, you're most likely going to start exhibiting tearing in the video in certain clips.
 

Flash6199

n3wb
Joined
Jul 11, 2019
Messages
15
Reaction score
3
Location
USA
Rtmp was implemented for the reolink cameras as a solution to the I frame interval problem, you're most likely going to start exhibiting tearing in the video in certain clips.
I would rather have some tearing than have no video footage at all. I have no idea why RTMP is so unstable, but I also have no means to test or debug it.
 

Flash6199

n3wb
Joined
Jul 11, 2019
Messages
15
Reaction score
3
Location
USA
You must exclude all Bi folders from even Defender.
Done!

I'm happy to find that after several days of disabling RTMP and only using RTSP, all cameras have remained online. I also found a miss configuration where certain cameras were not using hardware acceleration!

After fixing the hardware acceleration setting, and using only RTSP, all memory leak issues also seem to be resolved as Blue Iris is now only using 6.5 GB of RAM (not 16).

I plan to try to find time to run wire shark and monitor packets for RTMP communication on a known issue camera to try to figure out why RTMP is introducing so many problems. More to come!
 

TL1096r

IPCT Contributor
Joined
Jan 28, 2017
Messages
1,223
Reaction score
465
Done!

I'm happy to find that after several days of disabling RTMP and only using RTSP, all cameras have remained online. I also found a miss configuration where certain cameras were not using hardware acceleration!

After fixing the hardware acceleration setting, and using only RTSP, all memory leak issues also seem to be resolved as Blue Iris is now only using 6.5 GB of RAM (not 16).

I plan to try to find time to run wire shark and monitor packets for RTMP communication on a known issue camera to try to figure out why RTMP is introducing so many problems. More to come!
Thanks for the update. How do you disable and choose RTSP in reolink?
 

TL1096r

IPCT Contributor
Joined
Jan 28, 2017
Messages
1,223
Reaction score
465
You don't select RTSP in Reolink. You select RTSP under the camera video source configuration in Blue Iris for the Reolink camera.
Ok that makes more sense. That is the setting I have now in BI. I thought it was like the bitrate/fps where you have to set it on the camera to get it in BI.

Thank you
 

WSCam

Getting the hang of it
Joined
Sep 24, 2018
Messages
32
Reaction score
29
Location
Tacoma, WA
I'm having somewhat similar issues with my Blue Iris 5.x install (5.0.1.0 to be exact). It runs as a service but I do sometimes connect via RDP and run the app as well to monitor the feeds. I don't normally use the web interface.

From time to time, maybe after 24-30 hours of uptime, about half of the cameras go into "no signal" and the only solution is to close the console and stop the service, then start the service back up. About half the time the service can't be stopped at all and I have to reboot the machine.

My cameras are a hodge podge of Amcrest, Dahua and Reolink (I read through this thread and was surprised to hear of issues with Reolink "tearing" since I've never experienced it, but maybe my settings are different, unless I don't understand what that means? I think they're using RTSP but I don't remember). The cameras that show "no signal" vary, not any specific make or model.

This had all been working great up until recently, and the recent change I made was that the Intel Driver & Support Assistant I have installed on there prompted to update the Intel HD 630 drivers. I'm now running version 26.20.100.7000.

When that happened I thought maybe it was a good time to also update the drivers for the Quadro P2000 in there, which is doing a lot of the HWDEC work. So I updated it as well to Nvidia's version 26.21.14.3170

My guess is that one or the other of those drivers may be the problem so I'll probably roll both of them back and see if a) the problem goes away and b) introduce one back at a time and see which one causes it. Because I'm curious to know.

It does sound like some Intel video driver versions have problems, so I'm headed to the wiki now to see if that version I have is mentioned as good/bad. If that is the problem, then the lesson I learned is, don't update just because, or let someone else do it first and report back.
 
Top