איך לשלוט בווידאו שנוצר ב-AI? הסוד נמצא כאן!

I am pleased to present our work Latent Space Jam Layout Guided Video Synthesis So first I would like to introduce the talented team behind this work. This work was a collaborative effort. I was fortunate to work with my friend Asaf Joseph from Lytrix and Professor Ohad Fried from the Canvas Lab at Reichman University and the lab's researchers and professors. The Canvas Lab at Reichman University explores the intersection of computer graphics, computer vision and artificial intelligence. And I'm thankful for the opportunity to have worked with them. Okay, so a little bit of motivation. First. Current text to video models are very powerful, but they still lack the ability for precise control. A text prompt can describe what should happen, but it is much harder to specify where things should happen in the frame and how their locations should evolve across time. For example, suppose we try to control the layout with only a prompt. A cow walking on a grassy field following a diagonal path across the frame from the top left corner to the bottom right corner. This sounds like a clear instruction, but in practice text alone is not enough. The model may generate a cow in a grassy field, but it may not follow the exact diagonal path. It might place the cow in a different region, move it in the wrong direction, or ignore the intended trajectory entirely, as can be seen here in the generated video. Okay, so the goal of our work is pretty simple. We want to control objects placement over time in a generated video. And we want to be able to do it without any training or fine tuning the model. Here's an example of our input and output. On the left, the user provides a simple layout. Each box is paired with a set of one or more words noted here by color. In this example, the prompt describes two koi fish, one deep blue mapped to the blue box and one bright orange mapped to the orange box. The boxes only need to define approximate object regions. On the right, we want the generated video to respect both the text prompt and the layout. The final result should contain the correct objects with the correct appearance and style while following the user provided spatio temporal arrangement. Okay, so how does our method work? Here's a general overview of the method. The user provides a text prompt and the layout specification. Here you can see it on the left. First, we generate a guide video using layout guided latent optimization through cross attention here in the middle. Then we refine that guide using the same video model. The final output combines the advantages of both stages. Here on the right, it follows a requested layout and it has a visual quality of a state of the art text to video model. Okay, so during our research we Found a trade off in video generation. Aggressive layout constraints leads to lower visual quality. This led us to design a two stage process. I will start with the last step. The key observation is that video models can accept a visual guide as a prior. This guide can be rough, noisy or visually imperfect. As long as the guide captures the desired spatial structure and motion, it can serve as a strong prior for the final generation. This allows us to aggressively force a guide to follow our intended layout even at the price of visual artifacts. Similar to SD Edit, we take the preliminary guide, add noise to it and insert it into an early step of the diffusion process. On the left, we have a generated guide after we added noise to it. As you can see here, you can see that the guide is with artifacts. It doesn't look very high quality. We then path it through a video to video refinement pipeline. On the right, we obtain a refined output. The final video is more realistic while still preserving the course layout and motion from the guide. So the guide can be rough and with visual artifacts as long as it contains the global structure. This is important for the first step and that allows us to aggressively force the layout even at the cost of visual artifacts. Now we return to the first stage. How do we generate such a guide? We control the guide layout by manipulating the cross attention. Inspired by the paper Attend and Excite by Hila Heifer et al. Where attention maps are used to influence the presence and location of objects in image generation. We apply a related idea to video diffusion. We then model cross attention maps to connect text tokens with spatial regions in the video. So here's the layout generation pipeline. The input includes two parts, the text prompt and the target layout over time. Here you see it on the left. Text. Together with a layout moving across time, we run a pre trained video diffusion transformer. During generation, we examine the cross attention maps between prompt tokens and the video latents. The model is frozen. We do not train it and we do not change its weights. Instead, we optimize the latent variables during sampling so that the attention maps corresponding to specific words align with the target layout masks. The output of this stage is a generated guide. You can see it on the right. This is just a guide as a preliminary stage. Okay, a few notes. The optimization is performed iteratively at test time. At selected denoising steps, we compute a KL divergence loss between the attention map and the target layout. Since video models are big, we optimize on selected layers. So let's dive into some details. The important point is that we are not supervising pixels directly, we are steering the latent through the model's own internal attention mechanism. This makes the approach train free and compatible with pre trained diffusion models. On the left we have the target layout and on the right we have the cross attention maps at the beginning and end of the diffusion process. At the beginning of the diffusion process, attention mapping still be misaligned with the desired layout. You can see it in the middle. And then at the end of the diffusion process. After multiple optimization, the cross attention is aligned to the target layout. You can see it on the right. Okay, so let's recap on the left. Here we have the input, a text prompt and a layout trajectory. In this example, Spiderman swooping in. The layout generation model produces a coarse guide by optimizing latent variables using the cross attention layout loss. The guide can then be upscaled if needed. Finally, the video refinement stage uses the guide at the structural prior and produces the final refined video. The key advantage is that layout control happens without model training while visual quality is recovered through refinement. So let's see some results. We evaluated three categories. The first is a static layout control where an object should appear in fixed region. The second is dynamic layout control where an object follows a trajectory over time. The third is multi object layout control where several objects or regions are controlled simultaneously. This last setting is especially challenging because each object needs to follow its own spatial temporal constraint while remaining coherent with the others. Here's a static layout example. This is the easiest setting. The prompt is a cow walking on a grassy field. The input you see on the left, the mask. The blue mask is not moving too much. The output here is on the right and you can see like the cow is where it's supposed to be. Here's a dynamic example. The prompt is a car driving around a roundabout. And you could see the blue box with the motion. This is more difficult than static placement. The results show that the method can guide the object trajectory, not just its initial position. Here's an example from earlier and the corresponding cross attention map. Here's a more complex movement. Here we have some multi object examples. This is the most difficult setting, as it should control multiple objects simultaneously. The prompt describes Spider man swooping in and sticking to a skyscraper wall. You can see the blue box is supposed to map to Spiderman and the orange box at the wall. This example contains two distinct visual entities. A collision between two objects in two different styles. This is another multi object scene. A skier carving a slalom through trees. And last but not least. Great. Now we Compare to prior methods the prior methods performed somewhat decent on static and dynamic examples. Here you can see on the left. On the right is R method. So the prior method performed pretty much okay here. With one caveat. Since they were using older video generation models, they suffer from low quality. So we tried to refine the output from the prior method using a state of the art generation model like 1.2.2. In this case, we took the output from the prior method and we used it as input and refined it using 1.2.2. This didn't improve, didn't help. The layout does not follow. Then we also tried to change the backbone of the prior method. Meaning we implemented Trailblazer on the new video model. You can see it still achieved poor results. It's somewhat okay, but you can see that ours is much better. The camel follows the box much closer. But with the multi object settings, the prior method fits fail completely even out of the box. And even when we tried to improve it, it did not succeed at all. Whereas in our case you could see that it follows the layout pretty closely. So one more thing, including on top of controlling layouts beyond layout guided video generation, the same framework can support additional applications. Because the method works by generating or using structural prior, it can be adapted to editing and stylization tasks. In these cases, we are still using the same general idea. Preserve structure and motion while modifying appearance or attributes. Here's an editing example. So the prompt describes Joker speaking in front of the camera while his hair is on fire. But you could see the original video is just the Joker with no hair on fire. And the goal is not to completely regenerate the video from scratch. Instead we want to preserve the original character pose in motion while adding a new attribute in a controlled region. So here you can see on the right that we successfully edit the video and light his hair on fire. Also, another application which is nice is a stylization example where we would like to emphasize a certain element or style in the video. For example, you have a film noir black and white film and we want to emphasize a red dress which is a different style than the background. So you can see here that we emphasize a certain style that is contrast to the overall style in the video. Thank you.