איך להעלים אובייקטים מוידאו ברגע? גלו את Omnimat Zero!

So hi everyone, my name is DVIR Samuel, I'm a research scientist at Nvidia Research and I will be presenting a paper I worked on during my time at Origin AI titled Omnimat zero Fast Training Free Omnimat with Pre trained Video Diffusion Models. This work is the result of a collaboration between Origin AI, Bar Ilan University, the Hebrew University and Nvidia. So imagine you want to take a character from one video and place them into another. To do this realistically, we first need to solve a fundamental problem separating the object from its background. This is known as video matting, where we break a video down into a clean background and a foreground layer. In we take video matting into another level, we get as input a video and object masks and and output a set of layered videos. First, a clean background layer with all objects and their effects removed and also multiple foreground layers, one for each object, where each layer includes not just the object itself but also its effects on the scene. For example, in a video of kids running on the beach, each kid carries their shadow reflection and footprints which need to be separated from the background as part the of part of the omnimat. Existing approaches fall into two categories. First, optimization based methods that train a small network per video, minimizing reconstruction loss on the input and thus learn to separate a video into semantic layers. And second, training based methods that collect data sets of paired videos with and without an object and its effects and train a diffusion model to erase the object. This produces the background layer. However, they still require extra optimization to recover each foreground layer. Across both approaches, we keep running into the same three issues. First, they are expensive to compute. Optimization based methods can take hours just to learn the layer, the composition for a video that's only a few seconds long. Training based methods still need hours to train the backbone, plus additional minutes to hours of per video optimization. Second, they are slow at inference time. Rendering can take three to nine seconds per frame, which is far from real time. And finally, they don't generalize well. These methods tend to work only within an error setting and often break down on new videos or out of distribution objects. So we propose a three step end to end solution for OmniMATTE, which first removes the object center effects, then extract foreground layers and lastly compose layers together based only on pre trained video diffusion models. But the best part is that our approach is training free optimization, free and fast, close to real time. So since we are aiming for a training free solution, let me quickly recap how diffusion based image and painting works without any Additional training Given an image and its mask, say we want to remove the Eiffel Tower. We encode the image and its mask into the latent space of the diffusion models. Using the vae, we identify latent tokens belonging to the masked region and replace them with random noise. We then run the diffusion denoising process while preserving unmasked background tokens. This simple technique works beautifully for images, but if we directly apply this idea to video diffusion models, it breaks the model, removes the object, but replaces it with unstable blobby artifacts as you can see in the image. The core issue here is consistency. Image diffusion models mainly need to preserve spatial consistency within a single frame, while video diffusion models must preserve both spatial consistency and and temporal consistency. Those extra constraints makes clean object removal much harder, which is why we end up with these unwanted results. We aim to remove the object and all its effects while preserving the original background over time. Our key insight is to guide the diffusion model at the attention level. First, for each object token, we find background candidates from neighboring frames that can serve as replacements for the object. For example, the dots on the person in the middle frame are matched with the background points from other frames. We then override the attention between object tokens and these background tokens with the mean attention score of background to background pairs. This encourages the model to pull information from consistent background patches across time. We call this temporal attention guidance. Similarly, within a frame, we can replace attention from object tokens to nearby background tokens with the mean special background attention and we call this special attention guidance. Together, these two guidance mechanisms enforce both temporal and spatial coherence during generation. To further identify object effects, we observed that these effects naturally appear in the self attention maps of video diffusion models. We expand user provided mask by detecting high correlation regions in the attention maps, effectively capturing all object related effects. Interestingly, these effects do not appear in image diffusion models. We link this to the common fate principle in psychology that says that elements that move together are perceived as one unit. Video models seem to learn the same idea. The overall pipeline for object removal mirrors the image in painting pipeline. We encode both the image and the mask with the vae, replace the object tokens with random noise and just run inference using temporal attention guidance. The special attention guidance that we saw before using our approach. Using our object removal strategy, we generate a high quality temporally consistent background video with minimal artifacts. So for example, on the top our method removes the ball in the find the ball game make the woman playing with nothing. On the left, our method cleanly removes the biker and its shadow while preserving the original scene. On the right, it removes the people on the trampoline including the dent that they leave behind and at the bottom the rightmost swan is removed and including its reflection on the water while the rest of the scene and the other swans are not changed. But the biggest takeaway is onimat zero improves background preservation while being dramatically faster than prior methods. Unlike existing OmniMAT approaches that require heavy training or pair video optimization, our method is training free, requires no pair video optimization, and runs in real time. We evaluate omnimat zero on standard OMNIMAT benchmarks using both real footage and synthetic scenes, and compare it against three families of methods per frame image inpainting, video inpainting and dedicated omnimat approaches. Across these settings, omnimat zero achieves the best reconstruction quality with higher PSNR than previous state of the art and roughly 40% lower LP while running about 225 times faster. We also showed the model is not tied to a single video diffusion backbone. It works with both LTX video, which enables real time performance, and one which provides higher fidelity results compared to other approaches. Omnimat zero is able to remove all the target objects cleanly by while other methods struggle, they leave behind ghosts or artifacts, removing objects only partially or not at all or sometimes failing completely and outputting a gray screen. In the first example on the top, the field contains multiple dogs and omnimat zero removes all of them at once. In the second, a cat moves in front of the mirror and omnimat zero removes both the cat and its reflection while cleanly reconstructing the scene behind. So once we obtain the background layer, extracting the foreground layer is simple. We compute the difference between the original video latence and the background latence and this gives us a full foreground layer without any optimization or extra inference passes. This is exactly the example shown here. In the beach video, the target object is the dog once it is removed to produce the clean background layer, the middle video. Subtracting the background latents from the original video latents recovers the dog together with its shadow on the sand. And finally, to compose video layers, we simply add the two latent layers together. To make the blend even cleaner, we run a few quick noise denoising steps. This naturally smooth boundaries and merge effects. For example, it softens the dog's shadow on the grass and it helps to add details like the car's backlight on the water. You can find the project page and code here. Thank you for listening. Any question we have can you shout or you pretend. So the question was what is the reason for the special guidance? And the reason is that sometimes you can't get like background features from because the camera didn't move in the video. So you have two stationary cameras and you can use the information around the object to make the completion when the object is removed. Okay. Another questions? Yeah. So when working on this project ltx video and one couldn't really complete objects that were hidden behind other objects. So if you have two dogs playing together and one includes the other and you want to remove someone, one of the dogs didn't really complete the the dog correctly because I guess that the models doesn't have like really in painting for objects it can replace with a background, but not with the object itself. And hopefully now when using with newer video diffusion models probably will work like occlusions and stuff like this. Yeah.