The RTSP stream I was using was coming in via UDP, so there's no retrying to fix any network transmission errors. My best guess, based mostly on mjpeg working fine but h.264/h.265 not, is that it needs the cache as a buffer to acquire enough data to feed into the decoder in the right size chunks.