פענוח סרטונים: ה-AI של NVIDIA שישנה את הכל!

good to work with, valid to work with in terms of data and licensing and everything. Okay, let's go a little bit faster. So on the left you can see Agentic Search. Agentic Search is Just use the prompt to use all the elements. The detection, captioning, embedding to find an object, video summarization and report, and real time verified contextual alerts. The search is using all the elements that the VSS is capable of. It's using an LLM to understand the prompt and break it down automatically. Okay. It's a very simple approach, but very efficient. Okay, so here you can see also the pipelines of embedding, separate pipelines for embedding in computer vision with the Cosmos embed and cradle. And after that, using the retrieval of both, you can apply your own metrics to combine both of them. Okay. If you want to combine the computer vision, give more weight to the computer vision or to the embedding, it's your call. Okay, very simple. You can see another example of the actual elements used in real time. Okay, so you see the players. Just a second, let's see. Yeah, so, okay, just get it faster here. So, yeah, once you ask the question, the elements are being activated on the fly. Okay. So the VLM is being called to verify the results. The embedding models. See here The C radio CGLIP2 being used on the fly based on the prompt that was, was used. Okay. And you get all the results and you know, apply your own metrics to it. Okay. Video summarization. Video summarization. You could implement it with the vlm, just calling in a VLM time after time, you know, getting the captions and then activate another LLM on that to get the summarization. But that's not effective, right? You want, eventually you want to be super effective to work with the real systems, minimize the compute costs. Whether you're working in the cloud or on the edge, wherever you're using the compute, you want to minimize the consumption. So we implemented it in very, very efficient way, starting with the ingestion and aggregation of the metadata and everything. Okay. So for reports, you can also define your own templates for reports. Very simple. And get it really tailored for your use. Let's talk about performance a little bit. So I know this is the element that is a little bit, I can say, a bottleneck. Right. Everybody wants to use the biggest vlm, the LLM, in real time in production. But really it's about compute and cost. So you can see that the results here for VSS is really excellent. We have video summarization. You can see the video summarization with a very low latency. This is a 10 minute video summarization. You can see in one minute in RTX 6000 Pro. Okay. You know, this card, this is like a game changer. It has 96 gigabytes of memory and is very cost effective. And a lot of video decoders and encoders. So this is very efficient. The 2L 40s, also very good setup. And the Jetson, you can see the Jetson, Thor and digixpark. Who knows Digic Spark, by the way? Who owns a Digic Spark? Okay, we need to fix that. So Digic Spark is like, you know, the best compute for a researcher because you have 128 gigabytes of memory. Okay. It's like a small digit just, you know, on your desktop. So not. It's a real thing. Okay. So I recommend it. Okay, let's look at another example. So in this case, we're asking for a report. Let's generate a report. Was there some safety issue in the video? Okay. Really cutting edge. So we're just asking free text, the object that we're interested in, the actions that we're interested in, and we get the report. Okay. Very efficiently. Yeah, yeah. So this is, you know, again, you can do it with ChatGPT, with Gemini, you can do it, but this is an on prem solution running very efficiently. Okay. So really speed up your time to market, that's for sure. Okay. Contextual alerts. So in the real world, we want to find things fast, right? We don't want to just process in offline. So for that we really again optimized the complete workflow, the complete pipeline, end to end from the ingestion to the reasoning. You can see the number of streams here. Okay, so for Jetson Thor, you have 14 streams with latency of 09 seconds. It's all, you know, great, great performance, no question. And it uses all the elements, right? You have today you have yolo real time det C, radio, Cosmos embed. And you can plug in your own models into this. Of course, it's better if you have it in VLLM or tensort. It's better for your performance. Okay, let's take a look at an example of real time alerts. So real time alerts, you define the alerts in a free text. Okay. Again, of course. And the blueprint knows how to apply each the alert breakdown to the various elements. Let's say it's a person crossing the line, person running whatever features, in this case a box falling on the ground. Just process it in real time. And you can ask the system to also give it in specific feature like give it actual confidence, the confidence per alert. Okay. And apply more Rules into it. Okay. Performance Overall for alerts, 51 streams with 101 milliseconds. Okay, so this is super fast, super fast. Also for Edge, with a little bit of an effort, you can also squeeze it into a smaller Jetson. Not only Thor, we have a partner here, crg, showing some demos you can see also take a look there and some examples. And eventually again, Nvidia, we are very focused on physical AI. So this is the part, what I showed you is the part of reasoning, right? Understanding videos. But eventually this is really connected to the complete flow of physical AI and robotics. We relate to infrastructure, buildings and cameras and cameras in cities and warehouse as a robot by itself. So you can think about a building set of robots, using models to understand what's happening in real time and also generating predictions in real time for activating actions. Okay, so this is Cosmos, Cosmos from Nvidia. It's really about aggregating everything, consolidating everything together. The reasoning plus the generation plus the action. Action, models, vision, language, action. So we really opened the internal knowledge from Nvidia, where we already created a lot of data sets and models and flow for tuning models. Okay, so we released everything as part of Cosmos. Okay, you can see 15 terabytes of data, 320,000 trajectories, lots of assets for synthetic data, for Omniverse, for the domain where you create synthetic environments. All of this for training robots, testing robots and physical infrastructure. Again, focus on performance. Okay, so Cosmos is a single unified two tower mixture of transformer with two towers trained together, which has its benefits. It has a unified domain that can be used for better understanding. Also for the robot to understand what's happening in real time, predict what is going to happen next and take the action. Let's say an example of. First of all, it takes the lead in the benchmarks, a lot of the benchmarks. And we released, as I said, the physical AI data factory. With this you can go and create your own data and train the models very efficiently. This is an augmentation also for automotive, for robotics and other domains available right now. That's it. That's it for me. I'll be here if you have any questions. Feel free.

Did you include also changing in morphology and kinetics and also forecasting of the motility of the people and the players can hear you. Can I come again? I'm asking if you included morphological and kinetic analysis and forecasting to your metrics. So in the Omniverse, it's already baked in to the models. Okay. Everything is physically grounded. So the data that we train the model with is all physically grounded from simulation itself. Okay. So it includes trajectories and everything. But can you forecast the next step of the football player? Yeah, yeah, yeah. Forecast, you know, multiple scenarios. Right. It's not a single scenario. It can be multiple scenarios based on the frames that we have so far. But this is the actual, you know, today it's not running in real time, you know, the forecast and the action decision. But this is the future. Right. We need the robot to understand what's happening and also forecast multiple scenarios and take action per scenario and then do some another reasoning all in real time. Okay, so this is Cosmos is doing this, but it will take some several steps to make it really real time. We can talk about this also. Thank you. Thank you.