That is correct: Non-visible layers are still active and contribute to the track. If you want to have it not contribute you have two options:
1) Drag the layer below the track layer you want
2) Deactivate the layer by right-clicking and choosing "deactivate on current frame" (this is keyframed, so be careful!)
If you'd prefer it NOT to work like this, let us know.