Script to Detect Colour, Make and Model of Car and People from IP Camera (Updated July 2019)

[ilovecoffee]

Hey Everyone,

Updated July 18, 2019

TL;DR: I wrote several python scripts that can detect the make and model of a car in the driveway. I used both Machine Learning through Google's easy to use AutoML Image Classification and Sighthound's API for vehicle make and model detection. Both also allow the detection of people, although I prefer creating my own model for less false positives.

I decided to share this code, sorry it's a bit rough because I honestly have never coded anything in Python before, and I'm not a coder, just clever enough to get it working. And it does work.

I thought it would be great to have Blue Iris send me a text message of the make model and colour of a car in my driveway. I have also used it in another setup where there was a gate at the end of a long driveway, where it was particularly helpful.

This uses Sighthound's Cloud API which uses machine learning to determine the vehicle in the image. This API is free for the first 5000 calls, which my alerts fit well within. You can sign up on their website.

The script will check the previous vehicle detected and quit if it's the same. This is because Alert images from Blueiris may be set off by other factors and you may have a car parked in your driveway....you don't want to be alerted every minute that there's a White Ford F Series.

Updated Info:

I highly recommend you create your own model at Google's AutoML. It will probably take you about 30 minutes of work to set up, but it is not too complicated. It's all done via a GUI on their website. Pre-requisite: You need to have about 20 to 50 images of each category you want to train. Such as:

Bob Arriving
Bob Leaving
Alice Arriving
Alice Leaving
Mailman
Other Car

Just create folders on your desktop with each of these names, and drag and drop them, as you go through your alert images.

Steps:
1) You have to make sure BlueIris's service is not running as the local account. Open the start menu, type Services. Open Services and find BlueIris's service, and assign it to something other than the local account. Otherwise you can't run this python script.
2) Create a bat file in Word path with the following:

python C:\directory\of\your\script.py

3) I have this set to check images based on the Camera Properties>Post tab, so I can lower the resolution of the jpeg manually, so that the upload is faster and thus the response is faster. Turn on Jpeg snapshots on trigger with a time that's much longer than the trigger time (So it only snaps one jpeg) and turn the quality down to around 60%. Set the camera name to just &CAM so it always has the same filename.
4) Alternatively, you can turn on Store Alerts as High-Res images and uncomment the lines that will look for the latest alert image.
5) In the Camera Alerts tab, turn on Execute Script on trigger, and select the bat file you created.
6) That's it. You should play with your trigger settings so that it works as intended, this is really only meant to detect vehicles, you will probably have some odd results if you have too many alerts from trigger settings. The API will also pick off cars in the background (say parked on the street), so you may want to have a Privacy Mask enabled on the camera.

Please if there is a more proficient coder and you make changes for the better, please do share them here to help me. Thanks.


mattkri/BlueIris-SmartDetection

 

[riceandbeans]

This is great, thanks for doing and sharing. I was actually planning on doing the exact same thing for the past few months but have been like a one legged bobcat trying to cover up a crap on a frozen pond.

Throw this up on github and post a link and I'd be happy to be a contributor. I've actually been working on a system that does recognition locally with tensorflow instead of via a remote API like sighthound. I'd also love to use the sighthound API to get folks who approach the door automatically registered into a face database, friend/foe style, triggering on foe. Would be great to work with other folks here to get more stuff done faster.

 

[ilovecoffee]

This is great, thanks for doing and sharing. I was actually planning on doing the exact same thing for the past few months but have been like a one legged bobcat trying to cover up a crap on a frozen pond.

Throw this up on github and post a link and I'd be happy to be a contributor. I've actually been working on a system that does recognition locally with tensorflow instead of via a remote API like sighthound. I'd also love to use the sighthound API to get folks who approach the door automatically registered into a face database, friend/foe style, triggering on foe. Would be great to work with other folks here to get more stuff done faster.

I'd love to do this locally, I don't have much experience with tensorflow, for me with my limited coding knowledge I just threw something together with an API.

My dream set up would have a web interface where you could train snapshots to assign to people or particular vehicles. Then it could call out a TTS announcement to my Google Homes or send a text alert with things such as "Mail delivery alert" or "Unknown individual at front door", etc, etc.

 

[jason0]

1) You have to make sure BlueIris's service is running as the local account. Open the start menu, type Services. Open Services and find BlueIris's service, and assign it to something other than the local account. Otherwise you can't run this python script.

I am confused, is it supposed to be the local account or not?

 

[actran]

@iloveco Your python script provides a nice, understandable working example. Thank you for sharing. Questions:

#1: How is it working for you so far? In terms of getting the right car color and license plate number?

#2: What camera model are you using for this and at what distance are you taking snapshots of cars?

#3: Have you considered doing local license plate recognition with something like OpenALPR?

LPR Walk Through / Tutorial

#4: What about people recognition with local instance of OpenCV?

 

[Matt L.]

I was gonna say you're over your head. I see not that you are using the API to machine-learning. I wanted facial recognition and that may be possible...

This is great, thanks for doing and sharing. I was actually planning on doing the exact same thing for the past few months but have been like a one legged bobcat trying to cover up a crap on a frozen pond.

Throw this up on github and post a link and I'd be happy to be a contributor. I've actually been working on a system that does recognition locally with tensorflow instead of via a remote API like sighthound. I'd also love to use the sighthound API to get folks who approach the door automatically registered into a face database, friend/foe style, triggering on foe. Would be great to work with other folks here to get more stuff done faster.

Triggering what, say, the hounds?

 

[t_andersen]

Hi guys, this is really interesting. In my BI system, I get a lot of false alerts due to tree shadows moving, snow, etc etc. I have been dreaming of setting up some software that only gives alerts when the system sees people, cars and (maybe) animals. Google has released "tensorflow" and "mobilenet", and it should not be too difficult to get it going. I have talked to a guy who trained mobilenet to detect roads and signs from a moving vehicle and it seemed rather straightforward. I have installed tensorflow on my computer and I am now trying to learn how to use it, so I can train mobilenet to detect cars and people. I still have no idea as to how to interface to BI. Computation time may also be an issue but apparently it should not be a show-stopper when using mobilenet.

If anybody has been thinking along the same line, then lets keep this channel open for exchange of ideas and experience?

 

[Matt L.]

Using Python you can HTTP POST some JSON command sets, it does most things a web client of blue iris can. Mr Coffee has a nice setup to talk to him, just don't plan on JSONing BI to talk through cameras...

 

[jrf]

Was going through getting this to work...little issue is that data.pickle doesn't exist the first run.

with open('data.pickle', 'rb') as f:
previous_data = pickle.load(f)

had to create it with:
with open('data.pickle', 'wb') as f:
pickle.dump(current_data, f, pickle.HIGHEST_PROTOCOL)

then comment out. Excuse my poor python programming....so far so good. I'm working on modifying mine to send JSON data to webcore which is used with my smartthings hub. Then I can do whatever I want with that data.

THANKS!

edit: I got this working with webcore/smartthings no problem. I'm having it pass the make/model/color info into webcore. However it never seems to get the license plate. Might play with that a bit. Thanks for your HUGE running start on this.

 

[Matt L.]

I found sighthound to be rather simplistic, it bombed on age, called my accord a UPS truck, and my Camry it called a van. I hope google's tensorflow or whatever is better. The Google Pictures facial sorting is scary how adept it is.

 

[jrf]

@ilovecoffee Thank you, Thank you, Thank you for driving me down this road. I have now taken this and use it for "most" of my cameras. I'm submitting the image to both "detections" and "vehicles" and making a count of people, faces and vehicles. If I have vehicles I record make/model/color. I send all of this and the CAM name to my smarthome system (smartthings using webcore) which is really powerful for decisions. Talk about getting rid of false positives from shadows and storms!!! I'm still tweaking the logic in webcore...but holy crap this is nice. Now I can actually trigger the smarthome to say someone is at the front door...when there REALLY is a person at the front door. I'm basically integrating this logic into most of my cameras and letting BI do it's thing since I record 24x7 anyways. But THIS...takes alerting to a whole new level of awesome.

Link to thread with current python script:
Security Image Processing to determine if vehicle or person is in the image

 

[hayrone]

@ilovecoffee Thank you, Thank you, Thank you for driving me down this road. I have now taken this and use it for "most" of my cameras. I'm submitting the image to both "detections" and "vehicles" and making a count of people, faces and vehicles. If I have vehicles I record make/model/color. I send all of this and the CAM name to my smarthome system (smartthings using webcore) which is really powerful for decisions. Talk about getting rid of false positives from shadows and storms!!! I'm still tweaking the logic in webcore...but holy crap this is nice. Now I can actually trigger the smarthome to say someone is at the front door...when there REALLY is a person at the front door. I'm basically integrating this logic into most of my cameras and letting BI do it's thing since I record 24x7 anyways. But THIS...takes alerting to a whole new level of awesome.

import base64
import http.client
import json
import os
import ssl
import smtplib
import glob
import pickle
import datetime
import requests
import sys

##
## Send in as an argument the SHORT CAM NAME
##
CAM = sys.argv[1]

#CHANGE THESE FOR YOUR ENVIRONMENT
sighthoundapi = '{YOUR_SIGHTHOUND_API_KEY}'
homeassistanturl = "{SMARTTHINGS URL AND WEBCORE_PISTON_ID}"
filelocation = "C:\\BI-Sighthound\\" + (CAM) + ".sighthound.jpg"

# Reset Variables
person = 0
face = 0
cars = 0
modelconf = 0
make = 0
model = 0
color = 0
makeconf = 0
colorconf = 0
plate = 0
licenseplate = ''

##
## CREATE data file if it doesn't exist so we can compare if this is a duplicate vehicle on subsequent runs
##
if not os.path.exists('data.pickle'):
    new_data = {'make' : "null", 'model' : "null", 'color' : "null"}
    with open('data.pickle', 'wb') as f:
     pickle.dump(new_data, f, pickle.HIGHEST_PROTOCOL)
     print("Created file...")


# Uncomment the below if you store Hi-Res Alert images, and prefer to use those. This will search the latest jpeg to use.
#list_of_files = glob.glob('C:\BlueIris\Alerts\*.jpg')
#latest_file = max(list_of_files, key=os.path.getctime)
#print(latest_file)
#filelocation = latest_file

# CREATE SIGHTHOUND REQUEST
headers = {"Content-type": "application/json", "X-Access-Token": sighthoundapi}
image_data = base64.b64encode(open(filelocation, 'rb').read())
params = json.dumps({"image": image_data.decode('ascii')})
conn = http.client.HTTPSConnection("dev.sighthoundapi.com", context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))
conn.request("POST", "/v1/detections?type=person,face", params, headers)

# Parse the response and print the make and model for each vehicle found
response = conn.getresponse()
results = json.loads(response.read())

for obj in results["objects"]:
    if obj["type"] == "person":
        person = person + 1
print("Number of People Detected: %s " % (person))

for obj in results["objects"]:
    if obj["type"] == "face":
        face = face + 1
print("Number of Faces Detected: %s " % (face))

##
## CHECK TO SEE IF THERE IS A CAR AND THE MAKE AND MODEL
##
headers = {"Content-type": "application/json", "X-Access-Token": sighthoundapi}
image_data = base64.b64encode(open(filelocation, 'rb').read())
params = json.dumps({"image": image_data.decode('ascii')})
conn = http.client.HTTPSConnection("dev.sighthoundapi.com", context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))
conn.request("POST", "/v1/recognition?objectType=vehicle,licenseplate", params, headers)

##
## Parse the response and print the make and model for each vehicle found
## This really only gets the data from the last vehicle object. but does count them
##
response = conn.getresponse()
results = json.loads(response.read())
for obj in results["objects"]:
    if obj["objectType"] == "vehicle":
        make  = obj["vehicleAnnotation"]["attributes"]["system"]["make"]["name"]
        model = obj["vehicleAnnotation"]["attributes"]["system"]["model"]["name"]
        color = obj["vehicleAnnotation"]["attributes"]["system"]["color"]["name"]
        makeconf = obj["vehicleAnnotation"]["attributes"]["system"]["make"]["confidence"]
        modelconf = obj["vehicleAnnotation"]["attributes"]["system"]["model"]["confidence"]
        colorconf = obj["vehicleAnnotation"]["attributes"]["system"]["color"]["confidence"]
        cars = cars + 1
    if licenseplate in obj["vehicleAnnotation"]:
        plate = obj["vehicleAnnotation"]["licenseplate"]["attributes"]["system"]["string"]["name"]
    print("Detected: %s %s %s %s" % (color, make, model, plate))
 
    current_data = {
     'make' : (make),
     'model' : (model),
     'color' : (color)
    }

##
## Check to see if this is the same vehicle as the last run (multiple cameras)
##
    with open('data.pickle', 'rb') as f:
     previous_data = pickle.load(f)

    if current_data == previous_data:
     print('Data is the same!')
     cars = 0
    with open('data.pickle', 'wb') as f:
     pickle.dump(current_data, f, pickle.HIGHEST_PROTOCOL)
print("Detected %s People %s Faces and %s Cars" % (person, face, cars))

##
## SEND URL
##
url = homeassistanturl
payload = json.dumps({
"person" : (person),
"face" : (face),
"cars" : (cars),
"make" : (make),
"model" : (model),
"color" : (color),
"makeconf" : (makeconf),
"modelconf" : (modelconf),
"colorconf" : (colorconf),
"plate" : (plate),
"CAM" : (CAM)
})
headers = {
"content-type": "application/json"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.status_code)
print(response.text)

I'm also a SmartThings / hass.io user and am looking to do something extremely similar. Any chance you could put together a quick overview of how you have your smartthings/webcore/BI setup? My outdoor cameras also trigger like crazy.

 

[jrf]

I'm also a SmartThings / hass.io user and am looking to do something extremely similar. Any chance you could put together a quick overview of how you have your smartthings/webcore/BI setup? My outdoor cameras also trigger like crazy.

Link to post in the webcore forum:

Security Image Processing to determine if vehicle or person is in the image

That takes where this thread left off and continues with Smartthings and webcore. (If you aren't using webcore for smartthings start there and get that installed and learn about pistons. great stuff.) My current python script is there along with an "older" but working piston in webcore. Once into webcore it's really up to you what you want to do with the information. I'll update that post with my current webcore piston. Any questions, fire away.

First as above create your sighthound account.

Then setup BI to save a current image to a folder when triggered. I personally use the record tab and set JPG snapshot only when triggered and set the storage to AUX 7 of which I have setup to a folder just for my python script. I change the filename to be "&CAM.sighthound" which gives me filenames like "CAM1.sighthound.jpg" It then overwrites every time a new snapshot is taken which is fine for me as I record 24x7 and don't really need to keep the JPGs.

So example
AUX 7 Storage is set to: C:\Sighthound

so in that folder you will only ever have 1 image per camera.

My python script (linked in the other thread above) then sits in that directory also. In addition I created a batch file called "sighthound.cmd" that simply runs the python script with an argument

C:
cd \Sighthound
"C:\Program Files\Python36\python.exe" C:\Sighthound\object.py %1

Obviously i named my python script object.py

Then for each camera I want to use it with, under the "alerts" tab for "run a program or execute a script" I set it to be my "sighthound.cmd" batchfile. with the "parameters" being "&CAM". That way the CAM shortname gets sent to the python script...which then turns around and sends it as an argument to webcore.

I do all my decisions about what to do after that in webcore.

I documented the python script pretty well so you can tell what it's doing and I even included it writing a log file that looks like:

2018-02-01 20:42:38,391 INFO CAM2 - People:0 Faces:0 Cars:1 Make:Ram Model:pickup Color:silver/grey Conf:0.0871 - Status Code:200  Response:{"result":"OK","timestamp":1517535758332}

That's basically all of the information it sends to webcore and webcores response code.

 

[wayner]

This is great - thanks for sharing @ilovecoffee and @jrf. This looks fascinating. In terms of the people recognition can you "train" it with photos of a person and then just log the person that is captured by the camera?

 

[hayrone]

Link to post in the webcore forum:

Security Image Processing to determine if vehicle or person is in the image

That takes where this thread left off and continues with Smartthings and webcore. (If you aren't using webcore for smartthings start there and get that installed and learn about pistons. great stuff.) My current python script is there along with an "older" but working piston in webcore. Once into webcore it's really up to you what you want to do with the information. I'll update that post with my current webcore piston. Any questions, fire away.

First as above create your sighthound account.

Then setup BI to save a current image to a folder when triggered. I personally use the record tab and set JPG snapshot only when triggered and set the storage to AUX 7 of which I have setup to a folder just for my python script. I change the filename to be "&CAM.sighthound" which gives me filenames like "CAM1.sighthound.jpg" It then overwrites every time a new snapshot is taken which is fine for me as I record 24x7 and don't really need to keep the JPGs.

So example
AUX 7 Storage is set to: C:\Sighthound

so in that folder you will only ever have 1 image per camera.

My python script (linked in the other thread above) then sits in that directory also. In addition I created a batch file called "sighthound.cmd" that simply runs the python script with an argument

C:
cd \Sighthound
"C:\Program Files\Python36\python.exe" C:\Sighthound\object.py %1

Obviously i named my python script object.py

Then for each camera I want to use it with, under the "alerts" tab for "run a program or execute a script" I set it to be my "sighthound.cmd" batchfile. with the "parameters" being "&CAM". That way the CAM shortname gets sent to the python script...which then turns around and sends it as an argument to webcore.

I do all my decisions about what to do after that in webcore.

I documented the python script pretty well so you can tell what it's doing and I even included it writing a log file that looks like:

2018-02-01 20:42:38,391 INFO CAM2 - People:0 Faces:0 Cars:1 Make:Ram Model:pickup Color:silver/grey Conf:0.0871 - Status Code:200  Response:{"result":"OK","timestamp":1517535758332}

That's basically all of the information it sends to webcore and webcores response code.

Thanks for all of this. i got it mostly working until I realized that the way two of my cameras are positioned, they are likely to pick up cars parked along the street. Thus anytime i send along the screenshot, I will most likely always have cars in the scene. I can't think of a good way to re-size the screenshot or force it to only pick up "new" vehicles or people. The second camera will likely pick up my vehicle that is parked at the back of the driveway... so i envision a fly or bug triggering the motion, then Sighthound finding my vehicle in the scene and it alerting me that there is a vehicle in my driveway, which there technically is.

I need to come up with a good way for it to ignore pre-existing vehicles (from alert to alert) or to have it send motion in a very specific area ONLY.

 

[wayner]

Any advice on positioning the camera for those that have been using this. What is the optimal shot? Head on? Three quarters with front license plate visible? FYI - Here in Ontario we have front license plates so that should help.

p.s. The Privacy ramifications of this are pretty scary. It seems that it would be pretty easy to set up a system that could log all traffic on my street and potentially track people. All with less than $1000 of hardware. Imagine would the government, etc can do with the resources that they have at hand?

 

[jrf]

I need to come up with a good way for it to ignore pre-existing vehicles (from alert to alert) or to have it send motion in a very specific area ONLY.

I did not mention this. This is accounted for in a few ways. First the python script remembers the LAST vehicle detected. If triggered again and the last vehicle is the current vehicle it does not count it! This also works for situations I have where up to 3 cameras cover the driveway. Also in my webcore script I record and remember the last car for EACH camera. Again not counting it again if it's a duplicate. However...do note it gets color different sometimes so I only match on make/model.

Also...I think there is a way and I've thought about this also when there is a street also in the shot to block out the JPG. I'm pretty sure BlueIris can save that JPG with a mask also. Might have to setup a 2nd camera in BI for your physical camera. But I think you can use the masking? Not positive. Worth investigating. That way you could concentrate on only the part of the frame you want to.

 

[jrf]

In terms of the people recognition can you "train" it with photos of a person and then just log the person that is captured by the camera?

With a full sighthound account yet you can program it to learn people. If you haven't visited their site it's worth it for just what the capabilities are. Obviously when it's indoors with good lighting the recognition is way better for faces.

 

[wayner]

Thanks - I did check out the site and I am eager to play around with the API. But I suspect I need to get a better camera up and running first - but I have one on order and expect to get it next week.

 

[James DiBernardo]

Hi guys, so what's the latest version of the script? the one posted here or the posted to Security Image Processing to determine if vehicle or person is in the image ?