I think your first step should be to verify if this is a camera stream stutter or a viewing device decoding stutter, as your video is apparently a screen capture of a mobile device. I recommend using a Windows PC to view the footage in the camera's web interface, or the RTSP stream in VLC. See if it stutters like that when viewed that way.
I get similar stutter on my windows 10 laptop. I've looked at the video on a laptop with IE11, on an old Nexus 7 tablet as well as my android smart phone with TinycamPro. It's worse on the Nexus 7 but certainly still apparent on the laptop and smart phone. Maybe that would indicate it's at least partially the device?
Regarding the maxed out zoom with additional digital zoom, it suggests you purchased the wrong camera for this application. Is there a comparable model with longer focal length available?
The cam I bought was an 8MP varifocal with 4x zoom. I really wasn't sure how wide the FOV at full optical zoom of the cam would be and thought I would try it and see. It's about 15 feet wide at the distance the camera is from the feeder. (16 feet) For the remaining zoom I've been using digital. I'm sure this cam will get used elsewhere when I upgrade. This isn't used at night and my thought was that to obtain the highest detail I could get was to go with the 8MP unit. After playing with the IPVM camera calculator It turns out that a 12x zoom camera would be better suited. I've since learned that an 8MP camera with 12x or greater zoom is a bit more than I care to invest in this. I've been looking at a Hikvision DS-2DE4A425IW-DE. It's a 4MP, 25x zoom PTZ dome for around $600.
Personally I would recommend turning on Smart Codec, then try turning down the quality by 1 number at a time to see if you get better results.
I've tried this. Same (or similar) results.
These scenes with random 'things' (snow, rain, wind) moving quickly through the view are tough for the encoders to deal with, especially when they are trying to run at the limit of a constrained variable bitrate. So by turning down the quality and turning on Smart Codec you might be able to get bitrate under the constraint, thereby keeping the stream looking smooth.