Activation/Deactivation of Layers

I have a question. With me using a G5 2Ghz PPC I use the deativation of layers to speed up the processing of my tracking of another layer. Thing is I’m finding that for some reason that when I’ve ‘deactivated’ a layer and roto’d a selected objected (say an arm) other layers that I’ve deactivated are turning on and off seemingly at random during the tracking process.

Am I doing something wrong? I thought that if I selected a layer and deactivate it that it would remain deactivated during entire tracking through of another layer. I have missed something about this process probably. Any help would be appreciated.