Earlier this year I was on the other side of this argument (sort of). A neighbor I'd never met before came over one morning to ask why I had a camera looking in his bedroom window. This was a ridiculous claim which I denied straight away. I asked if he could show me what he was talking about, and
he led me to a thermometer that I have taped to the back wall of my shed.

This thermometer has been there for years, and its red blinking status LED must have drawn their attention the previous night. I tried to reassure him that this was a thermometer, not a camera, and that I would never spy on my neighbors like that.
To be fair, I do have cameras that can incidentally see this neighbor's house and window, but those aren't what he was referring to, and they see that far about as well as an 80-year-old with cataracts.
I can't speak for your neighbor, but unless you've done something to piss him off or make him distrustful of you (pointing spotlights or lasers at his camera would be a good start), then the camera and floodlight positioning is probably coincidental. Lights are just good sense for night-time cameras, and extending the camera down from the soffit is an effective way of distancing the camera from edges that would cause IR scattering to wash out the image. In the world we live in today, you can't point a camera or light in any useful direction without it seeing someone else's property.
Of course none of that is likely to make you feel better. I'm just trying to point out that this guy probably doesn't deserve your ire, even if you find his lights annoying. Planting tall trees as recommended by
@Dramus may be the best solution all around.