Those sounds like CPU response times rather than GPU response times, unless you're using huge models.
I'm have a lowly GT1030 and I'm seeing response times around ~150ms.
What models are you using?
Edit: Also check whether you're sending your cameras main stream, or sub stream for analysis. If...