I think it is impossible to stabilize everything in a scene when there is a moving camera that is not following a single forward facing vector. I’ll do my best to describe why I think this below. I hope I’m wrong, because I would love to hear a solution to the problem I’m about to describe.
Lets say we have a foreground object (a pole) and a background object (a wall) that are, say, 10 ft apart, and both objects are not moving. The camera is, say, 5 feet in front of the pole and pointing directly at the pole. There is a dot on the wall directly behind the pole which the camera does not currently see because it is pointing directly at the pole; so, the pole is blocking the view of the dot. We’ll take three pictures all pointing directly in the direction of the dot. First picture, we take a picture with the camera moved one foot to the left and pointing at the dot on the back wall which it now sees because we moved left. Second picture, we take a picture of the scene from the front with both objects inline, therefore the dot we’re pointing at on the wall is blocked by the pole Third picture, we take a picture with the camera moved one foot to the right, also pointing at the dot on the back wall. Now, if we “play” the pictures as a video in the order 1, 2, 3, frame 1: the pole will be on the right of the dot, frame 2: the pole will block the dot, frame 3: the pole will be to the left of the dot. Since we were focusing on the dot the whole time, the background will not be moving, or in other words it will already be stable, so there would be no need to stabilize the background wall. If we stabilize the the shots to the foreground (the pole), that would mean the pole would not be moving in the seen, but this would cause the background to move as follows: frame 1: the dot on the wall would be to the left of the pole, frame 2: the dot would move to the right and be blocked by the pole, frame 3 the dot would move further to the right and be to the right of the pole. In other words the wall would be moving from left to right. So, if the scene is stabilized to the background, the foreground moves and if the scene is stabilized to the foreground, the background moves. This is inherent in any scene with foreground and background objects with a moving camera that is not following a single forward facing vector. It is impossible to stabilize both the foreground and background at the same time. It is only possible to stabilize to a single depth in a scene. To add to this, there is not just a foreground and background, any chosen depth can be chosen to stabilize to. For example, if we stabilize to a depth half way between the pole and wall, then both the pole and the wall would be moving, but only half the distance of the above described scenarios. Basically, if you stabilize closer to the foreground, the more the background will be moving, and if you stabilize closer to the background, the more the foreground will be moving.
To summarize, any scene that has objects at different depths will display parallax when there is a moving camera that is not following a single forward facing vector capturing the scene. Due to the parallax, it is not possible to stabilize the scene at different depths. Therefore something in the scene will always be “shaking”.
I hope I described all this accurately and in an understandable way. Thoughts anyone? This seems like a real limitation. Is there a way to work around this issue? Am I just clearly wrong? I would love if there was a way to stabilize all depths in a scene with a moving a camera! Ever since I discovered the above through experimentation, I’ve been a bit depressed about it.