Reorient 360 video using multiple tracks

I have a two minute 360 mono video project with an unstabilized handheld Insta360 Pro2 camera. The camera holder walks the entire length of a driveway, turns a corner through a gate into a backyard, and then turns again into a garden area.

There are a number of useful horizon level areas to track for a Reorient process, but none are visible throughout the entire video.

I assume it’s possible to do multiple tracks that overlap in time to provide tracking data over the entire video.

My two questions:

  1. Can I use the Mocha Pro - Reorient - Render to render the full video in After Effects or do I need to break the render into subsets for each individual track?

  2. Is there an optimal point in the video (a specific frame) to align the horizon or is it just in an area of the video with a good clean track?

If you have any trouble, let me know and I will be happy to help you.

Cheers,
Mary

  1. Does this seem like a sufficient number of tracks? Having trouble finding areas to track as the dancers move all around the lead singer walking with the camera, occluding/interrupting what could have decent consistent track regions.

  2. Because the camera was handheld with no Z-axis preservation, and the lead singer holding the camera also turns three corners, yaw orientation changes quite a bit. The horizon moves on all three axis quite a lot throughout the video. Is it worth creating keyframed Horizon Orientation points at different frames in time in the video to accommodate for the widely varied camera movement in different scene areas? Current tests on the footage seem to show that setting a single Horizon Orientation keyframe isn’t solving across the entire video.

  3. After creating what seems like a decent set of tracks for a relatively stable section of the video, with the camera in one position, I’m getting single frames that pop out of position when previewing in After Effects with all tacks enabled in Mocha Pro. MPro doesn’t seem to be handling these bad frames correctly. Could this be related to using Large Motion when the sections were tracked? Would I need to retrack a spline to switch from Large to Medium/Small Motion?

  4. General questions
    a. is there any relationship between how many track regions are required for stabilizing an equirect and how jittery the footage is?
    b. Does the stabilization process reduce sharpness due to translating/rescaling content?

As an example of the frame popping I mentioned in Q3., here’s a quick playback video I shot of the scene in After Effects, with all tracks as shown enabled in Mocha Pro.

As typical for my posts/questions, they keep evolving.

Re: the popping in the video above - a few of my tracked regions have small occlusions like the top of a head, or hands swinging up into them while jumping, or a basketball throw through the air. They all are small occlusions but they seem to exist in the same ares the video pops.

Q1: is Reorientation popping a possible issue when a moving object enters a tracked area?

Q2: Do some areas lend themselves better for tracking (high contrast? vertical or horizontal lines? crisp detail?)

Hi Steve,
From looking at your Mocha screen shot, it seems to me that the areas that are good to track: Windows, the bushes on either side of steps (try to include a little edge of steps).

Areas that are not so good to track: columns (not much texture), the shape on top left is very large and should focus more on the window and edge of house. The one on the right does not look very planar.

Generally, try to focus your areas on textures that all feel part of the same flat plane. The shape on the right includes a person in the foreground and trees at different depths. depending on camera motion, this would most likely be a bad track area.

to answer questions:

  1. Does this seem like a sufficient number of tracks? Having trouble finding areas to track as the dancers move all around the lead singer walking with the camera, occluding/interrupting what could have decent consistent track regions.

You need to avoid the dancers and focus the tracker in the flat background planes. Sometimes it helps to stop the track and move the shape to avoid a FG and then pick it back up. As long as you move the shape to an area that moves relative to the that tracked plane, it can be good.

  1. Because the camera was handheld with no Z-axis preservation, and the lead singer holding the camera also turns three corners, yaw orientation changes quite a bit. The horizon moves on all three axis quite a lot throughout the video. Is it worth creating keyframed Horizon Orientation points at different frames in time in the video to accommodate for the widely varied camera movement in different scene areas? Current tests on the footage seem to show that setting a single Horizon Orientation keyframe isn’t solving across the entire video

Yes - I would break up the shot with many overlapping track areas and then keyframe the reorient side (yellow) in that module.

a. is there any relationship between how many track regions are required for stabilizing an equirect and how jittery the footage is?

Not necessarily. Generally the less tracked regions the better. But for long complex moves, you do need to track multiple areas that overlap.

b. Does the stabilization process reduce sharpness due to translating/rescaling content?
Stabilization is a render/filtering process, but should not reduce sharpness too much. If you have to resort to multi-passes of stabilization, certainly you will see loss. Use BCC VR Sharpo to compensate?

Q1: is Reorientation popping a possible issue when a moving object enters a tracked area?
Yes absolutely. The reorientation will only be as good as your tracks. If you view the Surface of each track, look to see if the blue box jumps. This will point to bad tracks for each layer.

Q2: Do some areas lend themselves better for tracking (high contrast? vertical or horizontal lines? crisp detail?). Generally, texture is what you want. Vertical & horizontal lines are not always great textures. This is why the columns might not track well, as there is not much texture. In your image the windows, the bushes seem to look good. Sometimes trees can give you good tracks but in 360 with a lot of motion might have too much depth. Again think about textures that move together… textures at different dephs and move at different speeds are the areas to avoid.

Hope some of this helps

1 Like

Hey Ross - thank you for that info. Certainly helps.

Is there an upper limit to how many tracks can be used in a single video? I’m at ~60 far. Yikes, this is a complex capture with very few planar elements…