Been struggling with this lately. When either tracking a shape or moving it manually, I will often hit a keyframe where a few of the spline tangents just flip out, sending the spline curves way off into the off-screen artboard to the point I can’t control them. I’ve tried turning off Shear and Perspective, which can occasionally help, but as mentioned, it can happen just when moving the transform box. Seems totally random.
(Mac Pro, High Sierra, After Effects plugin)
edit: I should add, there usually has to be a track of at least one frame for this to happen, even if I abandon the tracking and manually roto for a section. So I do think it has something to do with the tracking data. Just can’t figure out how to predict or avoid.