Script to Detect Colour, Make and Model of Car and People from IP Camera (Updated July 2019)

jrf

Getting the hang of it
Joined
Sep 12, 2017
Messages
169
Reaction score
93

James DiBernardo

Young grasshopper
Joined
Nov 14, 2014
Messages
39
Reaction score
5
Location
San Pedro, CA
@jrf id like to learn more of your progress , so i've created an account over at webcore -- im very interested in the car id and licence plate capture -- is there a caveat to where you need a smartthings device? or can i use the code you provided in the webcore post as is?
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
Sorry I've not been back to this thread in a while, it seemed dead for a while after I posted it. I also forgot to update the code because there are errors in it as others pointed out. I can post it here if anyone still wants it. I've done some things like image cropping (to avoid catching the street in front of my driveway) and image optimization (to decrease file upload time/response time).

I've been using it for a few months now. Honestly I don't even pay attention to make/model information because it's horrendously bad. My camera placement does allow for license plates to be picked up some times....although again that's not always 100% accurate, and comes through about 30% of the time.

But what it is good for is just plain vehicle detection. Since i have it SMSing my phone when there is a vehicle, I never get any false positives of motion detection. But it does like to think that my 2016 Mazda CX-9 is a Ferrari 458...which I'll take as a compliment.

Honestly I feel like an enterprising person could easily turn this into a locally run web GUI with TensorFlow to train your cameras with people and cars. That would be great!
 

jrf

Getting the hang of it
Joined
Sep 12, 2017
Messages
169
Reaction score
93
But it does like to think that my 2016 Mazda CX-9 is a Ferrari 458
Hahahaha, ditto. I see that Ferrari reference a lot. Very weird.

Please do share your updated script. And again, thanks for starting this. My false positives are 90% reduced because of this.
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
Hahahaha, ditto. I see that Ferrari reference a lot. Very weird.

Please do share your updated script. And again, thanks for starting this. My false positives are 90% reduced because of this.
I've updated the OP. I think I've removed all the personally identifying stuff and it's coded horribly bad with no global variables...I apologize. I'm not a coder! The way I program is that I run it 100 times until there are no errors lol.
 
  • Like
Reactions: jrf

t_andersen

Young grasshopper
Joined
Oct 5, 2014
Messages
90
Reaction score
17
Location
Sweden
I wrote earlier in this thread, that I was going to try Google's free Tensorflow/mobilenets algorithm that can be trained to detect people, cars, dogs, etc. I just want to report that I got it running well (python too). If I remember well, the processing time for one 1280 x 1920 frame was about 30 ms on a Windows 10 computer with an i7 processor.

I see two ways of using this to avoid the many false alarms from moving tree shadows:

1) The computation time makes it difficult to do live processing, at least with many cameras. On the other hand, it may be ok to do it, say, only once per second or on low-resolution frames. The interfacing to Blue Iris seems hard to figure out (at least to me), but maybe some of you smart guys can tell how to do it.

2) An easy solution would be to have Tensorflow/mobilnets running in the background independently of Blue Iris and then look at all alarm clips and throw away the false alarms. That would at least limit the number of clips one would need to look at afterwards but it would not stop the false alarms coming from Blue Iris.

Cheers from Sweden
 
Last edited:
  • Like
Reactions: jrf

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
I wrote earlier in this thread, that I was going to try Google's free Tensorflow/mobilenets algorithm that can be trained to detect people, cars, dogs, etc. I just want to report that I got it running well (python too). If I remember well, the processing time for one 1280 x 1920 frame was about 30 ms on a Windows 10 computer with an i7 processor.

I see two ways of using this to avoid the many false alarms from moving tree shadows:

1) The computation time makes it difficult to do live processing, at least with many cameras. On the other hand, it may be ok to do it, say, only once per second or on low-resolution frames. The interfacing to Blue Iris seems hard to figure out (at least to me), but maybe some of you smart guys can tell how to do it.

2) An easy solution would be to have Tensorflow/mobilnets running in the background independently of Blue Iris and then look at all alarm clips and throw away the false alarms. That would at least limit the number of clips one would need to look at afterwards but it would not stop the false alarms coming from Blue Iris.

Cheers from Sweden
Oh very cool! Did you train data yourself for specific recognition? Or are you just using a public model for object identification?

Have you looked into YOLO v2? It's probably a faster implementation of object recognition and there is a version that works with Tensorflow. It can readily be used with live video.
 

t_andersen

Young grasshopper
Joined
Oct 5, 2014
Messages
90
Reaction score
17
Location
Sweden
Oh very cool! Did you train data yourself for specific recognition? Or are you just using a public model for object identification?

Have you looked into YOLO v2? It's probably a faster implementation of object recognition and there is a version that works with Tensorflow. It can readily be used with live video.
I did not train it myself, although it shouldn't be that hard. One would of course need to collect a bunch of pictures of cars, people, etc, but (with some effort) they are available for download on the Internet. I also figured that it would not be ok to use my own camera stills for training because many images would be almost identical and make it hard for the algorithms to distinguish between, say, a car and a mail box visible in all pictures. So for the moment I just used the pre-trained models and that worked ok.

The mobilenets/SSD models are supposed to be fast (and less precise) because they were developed for smartphone-use. I did not try the YOLO.

At the moment interfacing to Blue Iris seems to be the main problem. It would be nice if one somehow could use replace the Blue Iris motion detection with the Mobilenets/SSD software but I do not know if that is possible in a simple way? Any ideas?
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
I did not train it myself, although it shouldn't be that hard. One would of course need to collect a bunch of pictures of cars, people, etc, but (with some effort) they are available for download on the Internet. I also figured that it would not be ok to use my own camera stills for training because many images would be almost identical and make it hard for the algorithms to distinguish between, say, a car and a mail box visible in all pictures. So for the moment I just used the pre-trained models and that worked ok.

The mobilenets/SSD models are supposed to be fast (and less precise) because they were developed for smartphone-use. I did not try the YOLO.

At the moment interfacing to Blue Iris seems to be the main problem. It would be nice if one somehow could use replace the Blue Iris motion detection with the Mobilenets/SSD software but I do not know if that is possible in a simple way? Any ideas?
I would probably use BlueIris's built in motion detection with a secondary pass thru your setup to determine if there is truly anything different in the scene, it would be less overhead on the cpu. I guess it would depend on what your scenario is. A front door or driveway would be applicable but a store probably not. This is how at least I've set it up using sighthound's cloud.

OpenCV is another project but I haven't read much into it and I believe it's far simpler.
 

t_andersen

Young grasshopper
Joined
Oct 5, 2014
Messages
90
Reaction score
17
Location
Sweden
So you would keep Tensorflow decoupled from Blue Iris, then running them simultaneously as two separate programs? That would be an easy way to do it, but Blue Iris would still give a bunch of false alarms?
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
So you would keep Tensorflow decoupled from Blue Iris, then running them simultaneously as two separate programs? That would be an easy way to do it, but Blue Iris would still give a bunch of false alarms?
Yes BlueIris would give lots of false alarms, but at least then you have a secondary filter to make sense of the false alarms. The downside is that you could only use tensor flow or another computer vision for custom alerts, not part of BlueIris's recording system....so within BlueIris you'd have lots of false motion recordings, but at least if you have the alert sent to your phone/email or whatever else, you will only get relevant ones once it passes through image recognition first.
 

iandrews

Young grasshopper
Joined
Oct 30, 2017
Messages
30
Reaction score
10
I have been looking at programme / scripts called ImageAI, it allows object detection with "a few lines of code" and it built on top of/ using Tensorflow. Don't think it suitable for real time, but I plan to hock it up to the alerts (somehow) and then only e-mail me when there is a "object" in the picture. It's still a work in progress, and I have to figure out how to get Blue Iris calling it etc. Thread with more info here Object (person) detection
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
Again bringing this back from the dead lol

I've further updated my script, as like JRF I also use this with Smartthings and Webcore

I wanted to use Google's AutoML Vision machine learning to teach it common alert images that it sees, namely:
- My car and my wife's car "arriving" (BI almost always has the same Trigger image of our cars at the end of the driveway); this plays audio in the home who is arriving (only if someone is home to hear it) and sends a push notification
- My car and mine "leaving" (again, it almost always triggers when it sees the back of my car pulling out of the garage); this checks the doors are locked.
- Last but not least, my mailman's vehicle...too much Amazon, poor mailman; sends a push notification so I know a package has arrived.

AutoML is fairly intuitive to setup, you just have to spend a couple minutes organizing old trigger images (that's another requirement, you need to have months and months of images to get a good set of training images), but once that's done, you're set.

If anyone is interested let me know. I am curious what other specific use cases could be used to "teach" certain images for certain outcomes.
 

hayrone

n3wb
Joined
Nov 22, 2015
Messages
18
Reaction score
4
Again bringing this back from the dead lol

I've further updated my script, as like JRF I also use this with Smartthings and Webcore

I wanted to use Google's AutoML Vision machine learning to teach it common alert images that it sees, namely:
- My car and my wife's car "arriving" (BI almost always has the same Trigger image of our cars at the end of the driveway); this plays audio in the home who is arriving (only if someone is home to hear it) and sends a push notification
- My car and mine "leaving" (again, it almost always triggers when it sees the back of my car pulling out of the garage); this checks the doors are locked.
- Last but not least, my mailman's vehicle...too much Amazon, poor mailman; sends a push notification so I know a package has arrived.

AutoML is fairly intuitive to setup, you just have to spend a couple minutes organizing old trigger images (that's another requirement, you need to have months and months of images to get a good set of training images), but once that's done, you're set.

If anyone is interested let me know. I am curious what other specific use cases could be used to "teach" certain images for certain outcomes.
I’m very interested. Is the script available? Do you grace a quick tutorial or high level view of how to get everything set up?
 

devastator

Getting the hang of it
Joined
May 22, 2015
Messages
169
Reaction score
28
Location
Belgium
Could you put the script on GitHub so I can contribute ? I used AWS Rekognition in the past. Because I live next to a busy road, it would always find cars in the image. I've created a transparant image (frontdoor_mask.png), that I overlay on top of the image snapshot. This way it doesn't detect cars on the road anymore.
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
Could you put the script on GitHub so I can contribute ? I used AWS Rekognition in the past. Because I live next to a busy road, it would always find cars in the image. I've created a transparant image (frontdoor_mask.png), that I overlay on top of the image snapshot. This way it doesn't detect cars on the road anymore.
I’m very interested. Is the script available? Do you grace a quick tutorial or high level view of how to get everything set up?
I have posted it on mattkri/BlueIris-SmartDetection

I've tried to comment the heck out of it for it to make sense. It's designed particularly for me, but hopefully it's useful to other people.
 

hayrone

n3wb
Joined
Nov 22, 2015
Messages
18
Reaction score
4
I have posted it on mattkri/BlueIris-SmartDetection

I've tried to comment the heck out of it for it to make sense. It's designed particularly for me, but hopefully it's useful to other people.
Awesome thanks. Have you incurred any cost due to using AutoML? I see there is a specific limit on how many can be classified before you get charged. In addition, it looks like you may have to pay after training a certain amount?
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
Awesome thanks. Have you incurred any cost due to using AutoML? I see there is a specific limit on how many can be classified before you get charged. In addition, it looks like you may have to pay after training a certain amount?
I have done at least 9 model trainings, and made probably 300 calls to the api. I feel like I'd have to hit several thousand before having any cost, and even then it would probably be cents.
 
Last edited:

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
On second note, do not do more than 10 model trainings a month. 10 are included. After you start training more than 10 the cost adds up significantly.

I'm going to figure out how to use the offline version of Tensorflow.
 

ilovecoffee

Getting the hang of it
Joined
Nov 3, 2017
Messages
30
Reaction score
31
I figured out how to do the first round of predictions completely without a cloud service, using Tensorflow on Windows in Python. It's not a complicated setup at all. Why no one has made a GUI for this type of use-case is beyond me...

You do require an Nvidia Graphics Card (a fairly modern one, I think GeForce 960 onwards, I'm using a GTX 1060) in order to do the training and predictions. Predictions are not as fast as doing it in the cloud, but approximately 3.2 seconds to get a result from a GTX 1060 card. This is ideal if you want to train for certain trigger images you know happen in your BlueIris setup, such as a familiar face, or car, or a person in a specific area, or a package on a door step, or even a horde of rats.

I can write it up so that all you need to do is install a few things, and then just run a batch file to train images, and another one to return a prediction via Pushbullet or an HTTP Request of your choice (so you can use IFTTT or Smartthings for example). Because everyone here runs Windows by default, it makes it easy.

If anyone is interested let me know.
 
Top