Unsolvable 3D Camera?


I am trying to solve a 3D camera track of a little complicated camera movement for our short movie that I am working on right now.

Camera faces down and while moving downwards is changing perspective to looking straight forward.
Here is the clip in question:
I tracked the wall, the closet to the left, the wine box with the lamp, the floor next to it.
Solving the camera is hanging at 97% or (with other tracks, or just partially selected) finished with a bad solve percentage.

Any tips on what I can do to get a decent camera solve to integrate a camera and 3d objects in AE?=