Do not run default/auto settings. It looks like you are basically at that.
It is a matter of getting the brightness/contrast and target ratio settings correct, and these can change during different seasons for some FoVs.
I always knew that you shouldn't chase a bright picture - it looks nice and people migrate towards a brighter TV for example, but upon closer examination, most images need to be toned down in order to get all the details. You will be surprised how much changing a parameter like gamma could impact tracking. For example, if you have a pesky tree or something in the middle of the view during an autotrack, just by changing some image parameters you can get autotrack to pass it. Making the image a little darker at night actually helped with tracking someone across the street, which was opposite of what I thought you would think to do. So add some contrast to your image and see if it improves.
I have a yard lamp post that more times than not autotrack would get stuck on it as someone was walking and the autotrack would only go so far. Because my image has soo much contrast (bright white concrete a third, blacktop road a third, grass a third), knocking down the gamma made the lamp post not be so "trackable" lol, and along with that I turned of PFA and that gave it just enough time to retrack the person walking past the lamp post. The camera may still autotrack the lamp post when a small kid goes by, but an adult it was autotracking past the lamp post. Most see better results if the contrast number is 8-10 higher than the brightness number.
Ideally for an intrusion box or tripwires, you should have the initial field of view be such that the camera doesn't have to initially pan too much up/down or left/right to get the object in the center of the screen to start tracking. The closer the object is to the center of the image, the better the chance that it will track correctly. An entire Field of View intrusion box can cause it to latch on to the wrong item.
The reason it starts looking upward or left or right is usually because the intrusion box is too big so the camera identifies the object before it is in the center of the field of view and then sometimes something else matches the "algorithm signature" of the initial object and then starts trying to track something that isn't there. Adjusting the field of view and the locations of the IVS rules to be closer to the center can fix that.
Autotracking PTZs are great, but they have limitations like everything else. Installed in a wrong location or using default/auto settings or with fields of view that do not give it a chance will be problematic.
/cdn.vox-cdn.com/assets/1241225/internetexplorer.png)