Remove parameter and chroma subsampling

I am curious how much of the parameters in the remove module is dependent on chroma subsampling? How similar is the footage information used in a remove to the info rises in a key?

We don’t use a keyer in the remove. What do you mean? We use the track for the remove and the pixels surrounding the shape to judge illumination modeling.

Sorry if I sounded confused (and it doesn’t help that there’s an auto-correction that I did not catch that further garbled my thoughts). I realize there is no keyer in the remove module.

I was trying to understand what information, at the pixel level, the remove parameters use. As I understand, for each pixel there is a luminance value and a chrominance value. I was wondering if that pixel information surrounding the removed shape is informed, in part, by that chrominance value? If so, wouldn’t the quality of chroma subsampling (4:4:4 versus 4:2:2 versus 4:2:0) affect the quality of the remove? If there is more chroma subsampling information (say 4:4:4 versus 4:2:2), at the pixel level, wouldn’t trickier removes more likely be successful?

But from your answer (“illumination modeling”), it sounds like the only pixel information used is the luminance value. Do I have that correct?

We don’t tend to discuss the nitty gritty of the under the hood calculations, but I can ask the dev team what they are comfortable with sharing. Stand by. Why do you ask?

Oh, I’m not looking for any secrets, and it’s ok by me if you all rather not say. This is more nerdy pondering that anything else…

Now that I think about it, the remove must use chrominance pixel values as well as luminance values. Otherwise there would be no color modeled onto the removed shape. I just wonder how much the quality of this chrominance information impacts the quality of the remove.

I ask based on an exchange with someone on another forum who was having trouble keying out hair locks. He decided to use mocha to remove the locks instead, but said he was having a terrible time getting that accomplished, particularly in finding the right tweak with the remove parameters. I suggested he try to get a clean plate to use in the remove, but wondered if there was a connection between pixel information that the remove parameters and the keyer both use: The chrominance value. That is, if he was having trouble getting a good key due to lack of chroma subsampling pixel information, would he also have trouble getting a good remove for the same reason?

If an impossible key could predict a difficult remove… I don’t know if that would be a time saver (I’d still probably try). It certainly could be helpful to know this so you could choose a camera capable of higher chroma subsampling to help the remove go more smoothly. Then again, if you know you’re going to need a remove, I would get a clean plate.

…i’ve kind of talk myself out of needing to know for practical reasons. (smile)

Hi Anthony,

The biggest factor in the quality of a remove is the tracking. When tracking, we have two modes that are relevant to your question: Luminance and Auto Channel.

Mocha works entirely in RGB space. So if the source material is YCbCr 4:2:2 this is converted on ingest into RGB with colour information at every pixel.

In Luminance mode, the tracker converts the RGB to a Luminance value, using the BT.601 (SD) coefficients. If the input material was decoded using these coefficients then this will be effectively lossless and the original luminance will be recovered. BT.709 (HD) has different coefficients so the effect of chroma subsampling will have an extremely minor softening effect. Given the way the rest of the tracker works, this is very unlikely to have any impact on the tracking accuracy. You could probably construct an artificial image to break this but it’s not going to happen in real life.

In auto-channel mode, the tracker assesses each colour channel to see which has the greatest contrast and uses that. This is helpful in situations where the scene contains objects that have distinct colours with similar luminance. Here, because a specific colour channel is being used, chroma subsampling will always have the effect of reducing image resolution and could make the track more difficult.

The remove module basically warps pixels from one frame to another, so chroma subsampling doesn’t have an impact. It also won’t affect the illumination modelling.

The reason chroma subsampling causes problems for keyers is that keying generates a matte based on the colour channels. If chroma subsampling has been used, objects which are visible in the luminance channel (e.g. a 1px wide hair) will be indistinct in any one colour channel, meaning the key doesn’t include them, or if it does, they are spread over a larger area than they should. Provided you track in luminance mode, this won’t be a problem for Mocha.

Hope that helps,

J-P

Thanks, JP!

I thought I was confused, but now I’m not that certain. (Smile)

…At least on my first read. …It’s starting to sink in now. This was thorough and very helpful. Much appreciated, JP. …And thank you too, Mary, for initially taking this on and facilitating.