IP Cam Talk Custom Community DeepStack Model

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
23,788
Reaction score
46,045
Location
USA
So moving forward, is that is where we should look instead of Post #1 of this thread or will you update that as well?
 

jaydeel

BIT Beta Team
Joined
Nov 9, 2016
Messages
1,110
Reaction score
1,215
Location
SF Bay Area
@MikeLud1 asked me to publish performance data using the new 'combined.pt' custom model using the methodology I published in August.

This post compares processing times for Blue Iris running Deepstack:
  • with only the 'combined.pt' model for 1 week
  • with only the DS core 'objects' model for the prior week.
Bottom line (for over 1000 events):
  • DS core 'objects' model performance: 142 ±18msec (1 stdev)
  • 'combined.pt' model performance: 102 ±19 msec (1 stdev) ... 29% faster
  • Indistinguishable results for the 'combined.pt' model v2 vs v1
Details follow.

Please note that my observations are system-specific. My relevant system specs:
  • I7-4770 processor
  • 16 GB RAM
  • PNY NVIDIA Quadro P400 V2
  • 9 x 2MP cameras, all continuously dual streaming
  • EDIT: All cameras are sending sub stream resolution D1 (704x576) to DeepStack.
  • 5/9 cameras using DeepStack (all Dahua and triggered via ONVIF using IVS tripwires)
My global DeepStack settings for the study:

1640037704329.png


'COMBINED.PT' CUSTOM MODEL
Full vertical scale
Total data points: 1275
Points >250 msec: 110 (9.4%) ... these points were discarded for the statistical analysis
1640037817331.png

0-250 msec vertical scale + stats
Total data points: 1165
Performance: 102 ±19 msec (1 stdev)
Note: on the 1st day of the study I forgot to configure all profiles to use the custom model.
1640038199943.png
1640038298422.png

DS CORE 'OBJECTS' MODEL
Full vertical scale
Total data points: 2216
Points >250 msec: 289 (15%) ... these points were discarded for the statistical analysis
1640039756756.png

0-250 msec vertical scale + stats
Total data points: 1927
Performance: 142 ±18msec (1 stdev)
1640039807091.png

1640039842775.png
 
Last edited:

jaydeel

BIT Beta Team
Joined
Nov 9, 2016
Messages
1,110
Reaction score
1,215
Location
SF Bay Area
Part II compares confidence levels for Blue Iris running Deepstack:
  • with only the 'combined.pt' model for 1 week
  • with only the DS core 'objects' model for the prior week.
Bottom line (for over 800 confirmed events):
  • Indistinguishable results for the 'combined.pt' model vs the DS core 'objects' model
  • DS core 'objects' model performance: 72±16% (1 stdev)
  • 'combined.pt' model performance: 74 ±14% (1 stdev)
Details follow.

My system specs
: Please see the previous post (#223).

My camera DeepStack settings for the study:
Driveway cams (2 total)
Notes:
  • I do not confirm cars (too many) - just large vehicles.
  • I analyze every 200 ms because objects move faster than my 'entry' cams (below)
1640043752524.png

Entry cams (3 total)
Notes:
  • I include cell phones as I've noticed they are a good proxy for people.
  • I've attempted to use 'mouse' as a possible proxy for critters smaller than a cat (skunks, possums, squirrels). It's never worked yet.
1640043806525.png



'COMBINED.PT' CUSTOM MODEL
Total data points: 824
Confidence (all data points): 74 ±14% (1 stdev)
1640044197903.png
1640044221228.png

DS CORE 'OBJECTS' MODEL
Total data points: 1268
Confidence (all data points): 72±16% (1 stdev)
1640044236985.png
1640044256894.png
 
Last edited:

105437

BIT Beta Team
Joined
Jun 8, 2015
Messages
1,990
Reaction score
872
@jaydeel and @MikeLud1 Do you recommend populating the "To cancel" field for all cameras? If so, I'd like to hear the reasoning. Thanks and great work with all of this data, much appreciated!
 

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,685
Location
New Jersey
I'm wondering if "zebra" in "to cancel" isn't effecting things. I don't think there is a zebra in the combined model unless Mike added it recently. Does DS keep looking for an object that doesn't exist under these circumstances is the question.
 

MikeLud1

IPCT Contributor
Joined
Apr 5, 2017
Messages
2,083
Reaction score
4,040
Location
Brooklyn, NY
I'm wondering if "zebra" in "to cancel" isn't effecting things. I don't think there is a zebra in the combined model unless Mike added it recently. Does DS keep looking for an object that doesn't exist under these circumstances is the question.
BI will keep on sending images to DS because BI does not know what labels (objects) are in the model
 

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,685
Location
New Jersey
So even if DS detects an object from the "to confirm" list it still keeps analyzing. I'd have thought it would stop once the "confirm" criteria was met.
 

wittaj

IPCT Contributor
Joined
Apr 28, 2019
Messages
23,788
Reaction score
46,045
Location
USA
That's why I was asking. So adding an object to the Cancel field that doesn't exist, serves no purpose.
Nope. It helps in certain fields of view, and it helps get a better alert image.

At night, my one camera, that has a straight on angle of the street to get a side profile of a car, would either find a car but the alert image would be the lightshine on the street or just a part of the vehicle, or it would trigger out nothing found due to headlight bounce off the street.

1640050464228.png

Once I added a cancel banana in the field, it now will go thru all the images and select the best one, which gives me the whole vehicle in the frame. It makes for scrubbing video much quicker as I can skip looking at video of known vehicles.

1640050718951.png
 

scp3000

n3wb
Joined
Apr 16, 2019
Messages
15
Reaction score
8
Location
southcoast uk
Well I must say after reading this thread from page start to finish, I thoroughly enjoyed the thread very constructive and a lot of work by Mike have made a few changes now to my BI be interesting to see what happens I hope I have understood it correctley I have put in (objects:0,combined,animal) for daytime and (objects:0,dark,combined,animal) for nightime, is this correct ? again great thread cheers guys..Ricky
 

MikeLud1

IPCT Contributor
Joined
Apr 5, 2017
Messages
2,083
Reaction score
4,040
Location
Brooklyn, NY
Well I must say after reading this thread from page start to finish, I thoroughly enjoyed the thread very constructive and a lot of work by Mike have made a few changes now to my BI be interesting to see what happens I hope I have understood it correctley I have put in (objects:0,combined,animal) for daytime and (objects:0,dark,combined,animal) for nightime, is this correct ? again great thread cheers guys..Ricky
If you are using combined all you need is (objects:0,combined) for daytime and (objects:0,dark,combined) for nightime all of the new animals are in the combined model. In the next several days I am going to add the dark images to the new models so the new models will not need dark any more.
 
Last edited:

105437

BIT Beta Team
Joined
Jun 8, 2015
Messages
1,990
Reaction score
872
Nope. It helps in certain fields of view, and it helps get a better alert image.

At night, my one camera, that has a straight on angle of the street to get a side profile of a car, would either find a car but the alert image would be the lightshine on the street or just a part of the vehicle, or it would trigger out nothing found due to headlight bounce off the street.

View attachment 112947

Once I added a cancel banana in the field, it now will go thru all the images and select the best one, which gives me the whole vehicle in the frame. It makes for scrubbing video much quicker as I can skip looking at video of known vehicles.

View attachment 112949
I'm liking adding "banana" to the cancel field. It's doing exactly as you said by sending all 10 images to deep stack and using the one with the highest percentage. I have a feeling that if I had this setting the other day, it would have identified the deer as a deer instead of a bird.
 

scp3000

n3wb
Joined
Apr 16, 2019
Messages
15
Reaction score
8
Location
southcoast uk
If you are using combined all you need is (objects:0,combined) for daytime and (objects:0,dark,combined) for nightime all of the new animals are in the combined model. In the next several days I am going to add the dark images to the new models so the new models will not need dark any more.
Thanks for that any idea when I play back deepstack log I get three frames around all detected items like my cars myself etc ?
 

Attachments

jaydeel

BIT Beta Team
Joined
Nov 9, 2016
Messages
1,110
Reaction score
1,215
Location
SF Bay Area
@jaydeel and @MikeLud1 Do you recommend populating the "To cancel" field for all cameras? If so, I'd like to hear the reasoning. Thanks and great work with all of this data, much appreciated!
I use it for the same reason as @wittaj in post #231.
I may not need it for my non-street cams, but I’ve not run a study.

I chose ‘zebra’ on the off chance the DS software looped through the labels in alphabetical order and it might matter.
 

105437

BIT Beta Team
Joined
Jun 8, 2015
Messages
1,990
Reaction score
872
@wittaj Follow-up question regarding the "To cancel" setting. If you populate it, does it delay the confirmation and alert until all images have been analyzed?
 

MikeLud1

IPCT Contributor
Joined
Apr 5, 2017
Messages
2,083
Reaction score
4,040
Location
Brooklyn, NY
I started working on adding the images from the dark custom model to the new models.
One question is should I add them as a separate label for example (car and Car) or should I combine them into one label (car)?
My thought is to have two separate label because the image count for each label in the dark model is low compared to new models and if I add them under one label it would be just noise for that label and degrade the accuracy for both.
 
Last edited:

sebastiantombs

Known around here
Joined
Dec 28, 2019
Messages
11,511
Reaction score
27,685
Location
New Jersey
I'd say "car" and "Car" or even "car" and "Vehicle". That way you could add trucks as well and just use a single label.
 
Top