Exactly, the IR view is definitely the issue; my house is near a street light and plenty of neighbors have house lights on, so the non-IR photo has no issues. I included the non-IR photo to show a comparison.
The material to the light is a light colored brick, the reflection of the IR may be...