It looks to me that your bottom track point is slipping all over the place because your search area is not focused on valid pixels to track. Understanding the relationship between the search area and the center track point is important for motion tracking and even more important for planar tracking. So - you might get a better track on the bottom hand, instead of at the bottom of the pole… then offsetting the point down to the bottom of the pole. As the shot proceeds, most of this data disappears as the shot goes on, making it difficult to get a solid track.
Looking at the shot, another option might be to use the BCC remover filter - it is similar to the wire remove concept but can be driven by masks instead of the 2 point line. In the Mocha portion, you could track just the top hand and than animate the linked Mocha mask to isolate the pole with a lot more control. Than back in Continuum, you can adjust the clone offset. You will also have more control to mask around the hand, depending on how perfect you would want to make the shot.
Hope this helps give you some more options or things to think about.