Yet another AI detection tool

[bnose]

Posted this in the BI forum but it applies to any setup that can write a image files to a directory.

I was inspired by GentlePumkin’s AI Tool to add artificial intelligence to Blue Iris. After experimenting with AI Tool for sometime I found that (although wonderful) on my system it would often miss the event that I was interested in. By the time that BI detected an event, DeepStack identified a target object and AI Tool triggered an alert in BI often the alert missed the action.

As a result of these missed events I decided to write my own tool using C# and Core. Currently the software monitors a folder for new files of a specific type (e.g. *.jpg files) and on creation of a file with a matching extension it will send that file to an endpoint (e.g. deepstack) and evaluate the results against a configurable list of objects and confidence levels. If a matching object is found equal to or greater than the confidence level the image and corresponding evaluation data are passed to a repository for further action.

Currently I have provided implementations for connections to DeepStack as an AI system and MQTT as the repository. These dependencies are abstracted and injected. Integration of other systems should be fairly simple if you’d like to ‘roll your own’. I’m also using BI but only as the instigator for creating images in the source directory. Any logic performed after a discovery is up to the repository (e.g. MQTT) and downstream systems.

My system is configured as depicted below:
BI -> images to folder -> AIGuardian (this software) -> DeepStack -> AIGuardian (this software) [if an object of interest is detected] -> MQTT -> Node Red -> [what ever I want (telegram, lights …)]

You could use the software as it is currently or provide implementations of your own AI tool (IDetectObjects) and repository (IPublishDetections). I’m using NodeRed for ‘what-happens’ when an object is detected rather than trying to build this logic into AIGuardian.

Source code and configuration details can be found at the GitHub repo below. My hope is I can find a few contributors to help me build and expand it’s feature set.

GitHub: johnhinz/AIGatekeeper

Enjoy!

 

[pmcross]

Posted this in the BI forum but it applies to any setup that can write a image files to a directory.

I was inspired by GentlePumkin’s AI Tool to add artificial intelligence to Blue Iris. After experimenting with AI Tool for sometime I found that (although wonderful) on my system it would often miss the event that I was interested in. By the time that BI detected an event, DeepStack identified a target object and AI Tool triggered an alert in BI often the alert missed the action.

As a result of these missed events I decided to write my own tool using C# and Core. Currently the software monitors a folder for new files of a specific type (e.g. *.jpg files) and on creation of a file with a matching extension it will send that file to an endpoint (e.g. deepstack) and evaluate the results against a configurable list of objects and confidence levels. If a matching object is found equal to or greater than the confidence level the image and corresponding evaluation data are passed to a repository for further action.

Currently I have provided implementations for connections to DeepStack as an AI system and MQTT as the repository. These dependencies are abstracted and injected. Integration of other systems should be fairly simple if you’d like to ‘roll your own’. I’m also using BI but only as the instigator for creating images in the source directory. Any logic performed after a discovery is up to the repository (e.g. MQTT) and downstream systems.

My system is configured as depicted below:
BI -> images to folder -> AIGuardian (this software) -> DeepStack -> AIGuardian (this software) [if an object of interest is detected] -> MQTT -> Node Red -> [what ever I want (telegram, lights …)]

You could use the software as it is currently or provide implementations of your own AI tool (IDetectObjects) and repository (IPublishDetections). I’m using NodeRed for ‘what-happens’ when an object is detected rather than trying to build this logic into AIGuardian.

Source code and configuration details can be found at the GitHub repo below. My hope is I can find a few contributors to help me build and expand it’s feature set.

GitHub: johnhinz/AIGatekeeper

Enjoy!

Sorry to ask a dumb question, but is this compiled anywhere so that I can download the exe and try it out? I see the source code on GitHub but I don't see an exe, etc. Is this meant to be ran on Windows?

 

[bnose]

Not a dumb question, I should have clarified in my OP.
No, it is only source right now, you'd need to download VSCode or the like to compile. It is cross-plaform so it could be run on any OS the is a .NET Core framework for.
I will look at finding a spot to put the binaries (preferably a container).

 

[IAmATeaf]

I use AITOOLS and find that the bulk of time is spent by DQ processing the image so am a bit confused by how you’ve managed to speed things up?

 

[bnose]

What this avoids (and AI Tool may have changed this since I looked at it) is the second call back to BI to create the alert, after a detection was positive. I found that by the time BI was notified of the alert the the event of importance was over. This software looks at the original detection to see what the issue was and uses that original detection in the rest of the pipeline.

 

[IAmATeaf]

You work around that by working out the pre-trigger recording interval. So for example I had my cams set originally to 5.5 seconds so that when a trigger occurred BI would start the recording with the prior 5.5 seconds. With AITools I just added the time it takes to process the image, which in my case was around 3.5 seconds, so I changed it to 8 seconds as way of a compromise.

So how does your software work? Does BI trigger which starts the recording and also a snapshot, it then examines the snapshot to work out if the trigger is valid and if not cancels the recording?

Not too sure overall how it works.

 

[bnose]

Good point on the on the 5.5 second buffer.

The software watches for image files written to a folder. Once an image is created it sends it to the AI software (this is by design easy to swap out for another AI tool). If the AI tool sends back a positive match for an object that is of interest (configurable) this software will then pass the image along with the metadata (confidence, match type, coordinates) to another system. I've created an implementation that passes the image info to MQTT (a lightweight message bus). Once received by MQTT you can create any sequence you'd like. NodeRed is a very popular IFFTTT type software that can be run locally and connects to MQTT to receive (or transmit) messages.

If you setup NodeRed to receive MQTT messages and call the BI API then certainly you could cancel the recording. You're only limit would be what BI exposes as an API. Or you could swap out the MQTT implementation I created to call the BI API directly.

For example, I have NodeRed listen to MQTT on certain topics. If give the proper conditions (I'm not home, sleeping or the sun is down) it sends the image to telegram and for my driveway camera it will turn my front lights on if a person is detected and announce the event over my Google speaker.

Hope this helps clear the purpose up.