BI Clips and Archiving, mainly dealing with New and Alerts

[bug99]

I need a bit of help with regards to BI Clips and Archiving setup, mainly dealing with New and Alerts. I am using DeepStack as a trigger.

I have my cameras all setup as dual stream (main and sub D1), and use direct-to-disk. Does this mean that I should be creating two file sets in the “New” folder for each camera, one at full resolution and one at D1, recording 24/7? If so I would assume that one set of files would be much larger than the other with similar names? If so what are the names, as it doesn’t really appear to be the case when viewing the “new” folder? [Note: I have each camera set to “continuous + triggered” in the record tab to the “New” folder.]

When I click on the calendar icon, I only see about three days of recording. If I look in the “New” folder, I see bvr recordings for over two weeks, but only see two to three days of jpg files in the alerts folder, corresponding to the calendar dots. Is there some reason that I would want to only have alerts for a few days (less than the video files)? I would think that the two file ages and the corresponding database files should all be in sync (cover the same days). How is this best accomplished? Is there a similar way to the clip calendar to see the video recordings that are available in the database/drive? I have historically set up all of my camera streams to be 24/7 for about 3 weeks, then overwrite.

 

[wittaj]

No, the main and sub are one BVR file.

Your alerts folder is either set to a few days or the size is too small.

Post screenshots.

 

[bug99]

@wittaj , I figured that the Alerts was set too small, and bumped it up from 1GB to 15 GB (and age from 7 days to 30) when i discovered the issue and while i waited for more information and someone to comment on the time line synching of the two folders (perhaps also including the db folder, but i suspect that that one is fine). What do most people here set things at?

The New folder is presently at 2TB and no age limit.

 

[fenderman]

Unless you are recording continuously as well, dont use deepstack to trigger. Use deepstack to alert otherwise you will miss events.

 

[bug99]

Unless you are recording continuously as well, dont use deepstack to trigger. Use deepstack to alert otherwise you will miss events.

I plan on recording continuously for at least three weeks. I am not 100% sure that it is happing at this point, but I believe that my setting of "continuous + triggered" causes it to happen. I am also not 100% certain that all of the continuous is in HD (vs only during the triggered alert plus some pre-post buffers), but based on @wittaj comment above, i think both high res and D1 resolution are being placed in the "New" folder under the same name as a bvr file until the space allocation causes the oldest of them to be deleted. When i get the feel of things, i might add in the Dahua tripwire triggers to the mix if that seems like a good idea. I also record continuously to each camera local storage for at least a week.

I can go back and forth to tweak the size and or time to keep numbers, but it seems obvious to me (perhaps mistakenly) that it would be a good idea to have the clip age should match fairly closely to the bvr file age range. If this is a good plan, shouldn't there be a storage choice to have them cover the same time range in BI? What good are DeepStack jpg clips a week older than the bvr video files? Right now i have (or did until yesterday) the opposite, having only 2-3 days of clips.

 

[sebastiantombs]

Additionally, when you record continuously Alerts are nothing more than markers in the database. I see no value to saving a jpg for every alert since the alert is in the bvr file and a screen cap can be made if necessary along with and export of the video.

 

[sebastiantombs]

A jpg is captured and sent, then apparently deleted based on the settings for the Alerts directory. I get alerts all the time without saving jpgs and the Alerts directory takes care of itself very nicely.

 

[bug99]

Additionally, when you record continuously Alerts are nothing more than markers in the database. I see no value to saving a jpg for every alert since the alert is in the bvr file and a screen cap can be made if necessary along with and export of the video.

Perhaps i am missing something with this simplification, but isn't that basically the situation that i had above that created only two days of 'tags" in the calendar, but forces it down to no tags if no jpg is recorded in alerts? or is it inherently different if the selection is database only vs hi-res jpg? With no alert jpgs (i don't see a selection for a few days of clips vs until the allocated space is filled, yet many days of db tags), i assume that you get no alerts in your GUI when the show /hide clips is selected, and this works for you? What is in the alerts folder with "database only" selected?

Note: i now have 6GB of spaced and 20k files used in 10 days of hi-res clips. That is lot of files, likely creating tremendous fragmentation over time on a spinning disk. By comparison i have nearly 2T in the new folder but only 732 files.

One thing i still don't fully get is when i watch a clip for a while, i see it change resolution to what i suspect is the sub-stream of D1 after a few 10s of seconds after motion stops. For continuous recording, this obviously reduces drive usage substantially, but is it the desired compromise? how is this ratio (pre-post trigger time set? I thought i was telling the tool to record both streams continuously (24/7) until full.

 

[wittaj]

If you save the high rez jpgs, then you can physically go to them on your drive and pull them up, delete them, etc. Depending on your setup, that can add up quickly. Most here do not use that option. One exception is for those running LPR to send hi-rez images to a 3rd party plate reader. But in most instances, saving hi-rez is a waste.

When you select the "database only", you cannot see them on a drive somewhere, but they are stored in the alerts folder (but for some reason you cannot access them by going to that folder). So if you don't want those thumbnails, make the storage in alerts 0 and tell it to delete after 0 days, or make it some number that you want to be able to see those alert images in BI to the left. They take very little space. 1GB will save those alerts for much longer than you have storage available.

When you are watching a clip and after 10 seconds of motion stops it goes to the substream - that is because that is the end motion time you placed on the motion trigger (BI default is 10 seconds). If you want a longer time, then you set that in each camera setting in BI.

This is working as intended. By recording both streams, it means it is recording substream until a motion and then goes to mainstream for the pre-trigger time duration, duration of motion, and post-trigger time the user sets and then it returns to substream. This allows us to get 24/7 recording at reduced storage needs so that we can still see if something happened that motion trigger missed, albeit not at a higher resolution. But that then allows us to tweak our settings if that is the desired outcome.

If you want 24/7 recording in mainstream, then simply select continuous. But that eats up the storage.

 

[looney2ns]

Read attached.

 

[bug99]

If you want 24/7 recording in mainstream, then simply select continuous. But that eats up the storage.

I think i want it the way it is now, basically. perhaps with a few optimizations over time. Am i correct that you are referring to the "Record" tab "video" selection pull down for each camera? I am presently choosing "Continuous + Triggered". There are "continuous" and a "continuous + alerts" options (and others, see attached). If i understand you, if i changed my selection from continuous + triggered, i would use much more HDD space by having all of the bvr files be in HD vs primarily in D1 and switching to HD?

 

[wittaj]

Yes.

Continuous = 24/7 mainstream

Continuous + Alerts = substream until a confirmed alert then mainstream for duration of event

Continuous + Triggered = substream until any triggered motion then mainstream for duration of event

So the Cont+Alert would use the least amount of storage (other than the non 24/7 options of when triggered or when alerted), but most here would not recommend cont+alert and would recommend cont+trig because the camera triggered for some reason so you would want mainstream of it.

 

[bug99]

Yes.

Continuous = 24/7 mainstream​

Continuous + Alerts = substream until a confirmed alert then mainstream for duration of event​

Continuous + Triggered = substream until any triggered motion then mainstream for duration of event​

So the Cont+Alert would use the least amount of storage (other than the non 24/7 options of when triggered or when alerted), but most here would not recommend cont+alert and would recommend cont+trig because the camera triggered for some reason so you would want mainstream of it.

Thank you for clearing that up. i looked for that clarity of recording purpose in the docs when setting it up.

On a related note. if i have Deepstack find cars, and i have two cars parked in the driveway in the middle of the frame, but also have a section of road at the top of the frame. how does the trigger and tag sub stream to HD supposed to behave? Obviously the cars in the driveway are stationary. Does it continuously trigger on my cars, or ignore them (stationary), or trigger and re-trigger and then time out after 10 min or something then have a cool down period and repeat, but missing moving cars on the street during cool down?

 

[wittaj]

During the day, it will see them as stationary.

At night, it can be a little problematic. Every car that goes by cast a different headlight shine on it and can make the stationary car appear different and thus log as a non-stationary vehicle. But that should only be happening as cars goes past. No movement outside and it will see them as stationary and no triggers.

 

[bug99]

Ok cool. I think i will add cars back on for the cameras that have my driveway cars in the FOV and see how that goes on false triggers as a trade off to idling cars going by on the street.

If you save the high rez jpgs, then you can physically go to them on your drive and pull them up, delete them, etc. Depending on your setup, that can add up quickly. Most here do not use that option. One exception is for those running LPR to send hi-rez images to a 3rd party plate reader. But in most instances, saving hi-rez is a waste.

When you select the "database only", you cannot see them on a drive somewhere, but they are stored in the alerts folder (but for some reason you cannot access them by going to that folder). So if you don't want those thumbnails, make the storage in alerts 0 and tell it to delete after 0 days, or make it some number that you want to be able to see those alert images in BI to the left. They take very little space. 1GB will save those alerts for much longer than you have storage available.

I want to get this dialed in, but remain unclear as to what does what. As it is now, i am seriously concerned about drive fragmentation due to the writing of 10s of thousands of files (thousands per day). I think the approximate behavior that i want is 1) see triggers on the time line and calendar for the camera(s) selected for at least three weeks; 2) to be able to see thumbnail images in the alerts viewer for a couple of days; 3) to be able to receive text/email/other image notifications at some point in the future (when i set it up) based on specific rules etc. I don't see any reason to have a bunch of high res jpg images in a folder, just in case i want them, when i can just make them when needed several different ways. @sebastiantombs made this last point above.

A couple of weeks ago i was only getting two days of timeline/calendar triggers. I increased the size of my "alerts" folder from 2GB to like 30GB. Now i get many days (i have not filled it up yet), but as noted, approximately 10,000 files per week. I suppose it really wouldn't matter much if i kept two days or twenty days in terms of fragmentation, as it is still writing about 1000 files per day. If this is the only good way to go, i suppose that i can move the alerts folder to the SSD drive to largely fix the fragmentation fears. Alternatively, i can give up the alerts viewing GUI thumbnails if it is the best way to prevent massive drive fragmentation.

What happens if i leave the "alerts" folder at 30 GB and changed the setting from Hi-res JPEG files to Database only? Does the system write snapshots for the alerts GUI in the "db" folder on in the bvr file (or use both)? Or does it write hidden files in the "alerts" folder, still writing thousands of writes per day?

What happens if i reduce the size of the "alerts" folder perhaps to zero? My gut tells me that setting to database only will write nothing to the "alerts" folder, allowing me to delete it and make it zero in size, relying on clever thumbnail image attachments to the bvr file with a meta data record in the db file to its existence (or some such techno jargon equivalent). thoughts?

 

[wittaj]

Exactly. There has to be a specific reason for the hi-rez images like for LPR. Otherwise it is a waste.

Given the purpose of the HDD for this type of activity and since this type of stuff should be the only thing on that drive because you should be using a surveillance rated drive, your fears of drive fragmentation are unfounded. When I added another drive a couple months ago, I took that opportunity since everything was shut down to take a look at that on my oldest drive. I tried to run the defragment and it analyzed it and said it wasn't needed.

I have my alerts set to 1GB (because you cannot go smaller than that other than 0) and the thumbnails go back months. Setting it to 30GB takes away that much GB available for storage.

But at the same time, BI needs that Alerts folder, so do not delete or rename it...

I think you are over analyzing this based on past experiences with computers and drives. BI has been around and operates this storage fairly efficiently.

 

[bug99]

I went in and changed all of the individual camera settings last night from Hi-res JPEG files to Database only and then went to bed. This morning, it appears that thigs did not change in any meaningful way. Does the BI service need to be restarted to take affect? I have attached a Trigger settings screen shot and a snippet of this morning's "alerts" folder with all of the new *.jpg files.

** update: restarting the service didn't change the behavior (as i assumed).

 

[wittaj]

Are you saving them from the Alerts tab too or are you running deepstack and these are the deepstack pics?

 

[bug99]

these all appear to be deep stack post analyzed pics, as they have an analysis percentage on them. Many of the ones that i looked at from yesterday dont, but the number of files from yesterday morning (midnight to 9 am) is about the same as this morning (339 vs 287 files).

 

[bug99]

Ok I just made a change to the Alerts folder settings. I reduced its size back down to 1GB and set the age limit to 7 days. I guess I will see what happens and use some trial and error. The cameras are still set to save triggers to Database only. Hopefully i will see at least three weeks of trigger data in the time line and calendar, and a few days of thumbnails without the 10s of thousands of files. I will report back soon.

*** update: after 15 min or so, i see that all but about 3000 files have been removed from the alerts folder. At 1/3 MB per file, this equates to the 1GB setting. Unfortunately, the time line and calendar now shows only three days of triggers. perhaps that will change somehow in the coming days to a much larger number to align with the database only camera setting, but i have my doubts.