הסוד האפל של נהיגה אוטונומית: למה קנה מידה לא עובד ומה הפתרון?

I'm going to talk about an approach for how to scale more efficiently end to end methods for autonomous driving. Let's start with the motivation, the bitter lesson, the principle that Rich Sutton wrote about scale. General methods eventually win if we apply it to self driving. Self driving is a difficult problem. We need to deal with open world driving, perception, scene variation, interaction with other road users. It's complicated. So let's learn everything from data, either end to end learning or rely on a foundation model like vision language model and just start with a generic approach and scale. More data, more compute and learning will solve everything. It's a very attractive approach. There is just one problem with this approach. Scaling has diminishing returns. So to understand it, let's look at the empirical scaling laws for end to end system, for example the chinchilla law for a foundation text model. And these laws try to figure out what is the rate of convergence. The rate of decrease of the loss function as a function of the number of parameters in the network, the model size and the number of samples in the training set. So these laws tell us that there is a power law with exponents alpha p for the parameters and alpha S for the sample size. And the laws show something very interesting. The exponents are very small, they are around 0.1. And what is the implications if we want to improve the loss from 10 to the minus 2 to 10 to the minus 6, which is roughly what you need for an application like a fully self driving system, then because you want to improve by four orders of magnitude and the exponent is 0.1, it means that the number of samples that you need and the number of parameters that you need scale up like 10 to the 40, which is insane. It's very, very large. Even with the amazing speed of computers, it's still too large. So scaling helps, but it's a very, very poor approach for reaching a very low error. So the question is, why is it slow? Why is it so slow? Because if we look at classical laws for statistical learning, they look like n to the minus alpha, where alpha is around 1. So for example, in the realizable binary classification setting, maybe the most simple classical statistical learning setting, the exponent is 1. So we have 1 versus 0.1. Why is there such a big difference between what happens in classical statistical learning and what happens in classical large end to end models? And we claim that the root cause of the slow scaling is a poor signal to noise ratio in the gradient. And this poor signal to noise ratio in the gradient is a result of Tail distribution plus uncertainty. So let's explain these two things, tail distribution and uncertainty. Starting with tail distribution, what is a heavy tail distribution? Following classical computer science, expander graphs and the like, I like to call it like a hay in a haystack problem. It's not a needle in a haystack. There is no, just one needle that we need to look for in the haystack. It's many, many needles or a hay hidden inside the haystack. Because there are a huge number of rare but very important scenarios. And together their distribution mass is what makes the problem difficult. So we worked on trying to figure out more rigorously why long tail plus uncertainty leads to poor signal, to noise ratio and therefore to slow scaling. So we took a toy model where we just look at a simple mlp, which is one of the building blocks of transformers, and we create Zipfian law distribution. So a heavy tail distribution plus create some uncertainty. And we prove that the combination of these two things, the heavy tail and, and the uncertainty, leads to a very, very slow scaling. And we prove it mathematically. There will be links. I will not go over the proof. There will be links to a blog post where we published the theorem and the proof and the implications. But the implications for self driving is that in self driving we have both of these issues. We have uncertainty and we have heavy tail distribution. So we already talked about heavy tail distribution, long tail distribution. It means that when you are applying stochastic gradient descent, then most of the time after you get to a decent accuracy, most of the time you just waste energy when you train the model because most of the examples are not informative anymore. And also there is intrinsic uncertainty in end to end training. Why? Because what we are doing in end to end training of self driving, we are imitating human drivers. Now, at every scenario there are more than one correct answer as to how is appropriate to drive. Okay, so one driver will perform a lane change here and another one will not. One driver will keep a little bit shorter distance to the heading vehicle and another one will take a higher distance, a longer distance, and both of them are reasonable. Okay, There are extreme cases which looks bad, but most of the time there are more than one correct answers, which creates uncertainty. The supervision is very, very weak and using the words of Karpathy, it's like sucking supervision bits through a straw. And in addition, there is a phenomenon that happens a lot in deep learning, which is called shortcut learning. For example, if you try to train a model end to end, to stop or go when it enters a junction. So the junction has a traffic light and the model needs to decide whether to stop or to go. So you would imagine that the model will look at the traffic light and figure out by the color of the traffic light whether to stop or go. And what happens more often is that the model doesn't look at the traffic light at all. It looks what other drivers are doing. And based on this, it figures out, okay, if everyone is going to stop, I'm going to stop as well. I just want to drive like everyone else. Why the model does it? Because maybe because cars and other objects are larger, they are not a very small element in the image. So it learns faster that it's a good heuristic, it's a spurious feature, it's not a causal feature, but it works most of the time. So it's a good heuristic. We call it a shortcut. The problem with these shortcuts is again that if you think about stochastic gradient descent, once your shortcut sort of works, then every time it works, you don't get any meaningful gradient information. So, so you learn nothing from all the cases in which the shortcut works. So you can learn only from the rare events in which the shortcut doesn't work. And then you get a very, very weak signal as opposed to all the other noise that we have in the system in the learning. So again, it's a source of poor signal to noise ratio. So to summarize, if we want to efficiently scale large model, we need to solve the poor SNR ratio. Otherwise we will be with a very poor exponent of 0.1 or around it. And so this is something that people figured out and there are a bunch of heuristics how to improve this snr, Add structure, auxiliary tasks. This helps because then you get more signal curriculum learning. Again, you start with learning what is a traffic light before understanding whether to stop or go mining rare events from large fleets using simulators in order to increase the number of examples, or relying on foundation models or word models. Again we get more data in order to learn what we want. So all of these are good approaches. But what I want to talk about today is trying to tackle this problem directly and not indirectly. So can we optimize a signal to noise ratio directly? And our main idea is that we need to separate failure discovery from resolution. If you think about sgd, it mixes up both finding issues and fixing them. You just randomly pick examples. If the loss is bad, you get a gradient information. So finding what are the bad examples and, and Fixing updating. The model is entangled and we want to disentangle it. And I argue that this will increase the useful gradient signal. So let's see how it works. So the inspiration comes from an ancient model from 1995 called AdaBoost. The young people in the crowd probably don't know what I'm talking about, but this was maybe one of the most influential model, late 90s, early 2000. It's a very strong model. And actually this model, this boosting approach, is a method for disentangling discovery from resolution. So let me explain. If we look at sgd, what is the objective that SGD tries to solve? It tries to solve the expected loss. So we have expectation over examples, samples from our underlying distribution over the world. And then we have a loss function that takes the prediction of our model F theta of X. So theta is the vector of parameters. X is the input, Y is the output. There is some loss function on F theta X. And we want to optimize the expected value of the loss. And stochastic gradient descent is a very favorable approach to do it because we can estimate the gradient of this objective by just sampling from the distribution. So this is how stochastic gradient descent works. Boosting replaces the objective. Instead of minimizing the expectation, it minimizes the maximal loss over all of the examples. So we replace expectation by maximum. And how it works. So the adaboost has loops and each loop has a separation between discovery and resolution. What is discovery? We collect failures over our training set. So we have a training data. We calculate the loss function over the the entire training data and collect failures. Then we change the distribution over the data to focus more on the failed examples and less on the examples that we already understand. And then we optimize. You can do it based on so cassette descent, but on a modified distribution. So we train a new model on the modified distribution. So the discovery is finding the errors in the training set and changing the distribution accordingly. And the resolution is retraining the model on the newly created distribution. So this is Adaboost from 1995. So if it's so ancient, why nobody uses it? Because it has a very fundamental limitation. It has zero tolerance to noise. The algorithm really breaks down completely when we have noise. Why? Because we looked at the maximum over the example. So it suffices to have a single bad apple in the training set. And this is sufficient for breaking the algorithm. Now why do we have bad examples? Because when we work with large data, it is inevitable to have some bad examples. So here on the bottom you see images of failures of some pedestrian detection system. And these failures are actually not so bad for the model. So one of them is there is a drop of water that causes a reflection. So it's hard to see that there is a pedestrian here at all. Then we have a wiper, we don't see anything. And then we have a labeling error on the other side, just an error for ground truth creation, which is a system by itself and also has errors. So we don't want to focus all of the energy, all of the distribution on these examples because there is nothing to learn from these examples. These are noise. So we want to apply boosting. But boosting by itself, adaboost by itself is not robust to noise. In addition, adaboost looks only on the examples in the training set. And sometimes we want to bring examples outside of the training set. So here is the idea of a new boosting approach which, which we call scenario boosting. We look again at minimizing a worst case. But now the worst case is not over a single example. The worst case is over a scenario, a complete failed scenario. So we maximize S is a semantic scenario in a set of semantic scenarios. And for each scenario we have expectation of examples taken from a generator of examples according to the scenario. Okay? And then we have the loss term. So what did we change? Instead of looking at single examples, we look at meaningful failed scenario and then we apply boosting as before. Okay, so again we have max 4 discovery and expectation for fixing. Before I continue, it's interesting to look at this also from a safety perspective. Take a rare example, like a fallen motorcycle rider facing traffic. This happens. This is very rare scenario by itself. So the probability to see it is already very small. If we look at SGD objective that looks at expectation, then there is very very little pressure from fixing this example because to begin with, the probability of this example is very very small. But of course it's unexpected to fail consistently on this scenario. So if we have a system that consistently misbehave in this scenario, it's not a good system. Okay, so the objective of maximizing over scenario is also aligned with safety goals. Okay, so now the question is how, how to optimize a scenario boosting objective. So I put up on the right again the equation. So what we have in the equation we have maximum over scenarios. So the first thing is we need a way to represent semantically scenarios. That's the first problem that we need to solve. Secondly, we have expectation of examples according to a generator G of X g Of S. Sorry. So we have a generator of examples from the scenario. So this is the second thing that we need to solve how we generate example from the scenario. And finally we need to automate the process of solving the maximization problem. So we have a definition of scenarios. We have a way to generate examples from a scenario. But how do we solve the maximization problem of finding a scenario on which we fail? So these are the three things that we need in order to apply scenario boosting. So the way we tackle it is maybe similar to how modern view of boosting. So the perspective is to think about boosting as a two player game. We have two players, one of them is a learner. We call it the theta player. This is a player that tries to learn the model and the parameters of the model is theta. So this player tries to minimize the objective. Then we have a second maximization player that tries to maximize over scenarios. So the second player plays on the scenarios and the objective is expected loss over generated examples from the scenario. So the model wants to have a good model such that there are no failed scenarios and and the other players try to find failed scenarios for the model. So one of the players will be a vanilla SGD just with distributions that are changing over time. The real novelty is about the second player. We call it Meteor and it's basically an agent that acts like a data scientist player. So this process of solving this min max problem is actually what happens in many, many companies that try to solve a real problem. There is a data analyst. The data analyst finds failures of the model and modifies the distribution in order to bring more examples from the failed scenario. So what we aim to do is to automate the the data scientist player. Okay. And what we took is a visual language reasoning model as an agent that plays a data scientist player. So it analyzed failures of the model, it defines scenario s and we use text, free text to describe scenarios, which is convenient for language models. And then we also use again Genai tools or rag tools in order to create a distribution for the scenario. Okay, so let's see how it works in practice. The first phase is to build a data set to ingest a data set for the data scientist for the meteor agent. So we start from millions hours of diverse driving data. These are from different countries and vehicle installations and all road types, et cetera. We embed them both the video, the full image as well as individual examples. So we pass it through a pipeline that embeds both full videos or short video sequences, but also single images and also individual Objects in in the images. And then when we create it, we have elaborated prompt that asks the data ingestion model to pay attention to how rare is this scenario, et cetera, et cetera, et cetera. So we have filters on different approaches that will be used later. The end result is that we convert video data into into a representation, a very short representation composed of free text plus structured properties plus some embedding vectors in order to use RAG later. Okay. The second tool that we build is semantic search. And this is the classical approach of fast approximate nearest neighbor in order to search over the embedding vectors, search over the text embedding and also sliders to control search results based on structured properties like is it nighttime or daytime? How rare is this scenario? Et cetera, et cetera, et cetera. And then there are two APIs, one for humans and one for agents. For the agent is MCP. Okay, so let's see how it works. So what we see on your left, we see a failed example. So we mis detect a pedestrian here there is a pedestrian. Let's see. Yeah, it works. There is a pedestrian hidden behind this glare of the camera. Now when we look at the complete video on the right, we see that this is not really something that you want to solve in single frame because in a bit the pedestrian will be detected and everything is fine. So there is no failure really here in terms of the entire system. So the agent here determines that the model is actually fine. There is nothing wrong, even though there is a training example on which it misses a detection in single frame, but the actual system is fine. On the other hand, here you can see a different failure. There is a pedestrian standing very close to a car with an open door and the model doesn't see it. And then the agent figures out that when that the current model requires economical head, torso, leg evidence in order to classify a pedestrian as a pedestrian. And then it will look for other examples on which we miss. It's a scenario. And we look for other example on which we miss the detection because it's not canonical. So the agent proposes search words for scenarios and several proposals and search into the database. So we search queries in the database, we re rank the results, then we choose the most helpful results and we evaluate the model on the newly generated distribution. And then we accept the hypothesis the data scientist accept of the data analyst, except the hypothesis if it is indeed a reproducible failure of a scenario. So what we did here is automating what usually humans are doing as data scientists. But we can automate it. And once we automate it, it can enter into the training loop. So we can make all the flywheel of finding evidence of problems, fixing them, finding more evidence of problems, fixing them fully automatic and make progress much faster for the long term. So these are some samples from this hypothesis and sometimes the failures are more rare and we don't find sufficient examples in the data set. So what we do, we use genai in order to generate more examples. So here you have four videos. Only one of them is real, the other three are not real. If you can tell which one is real and the others are not real, good for you. I cannot, maybe, I don't know, it's double blind. And also you can have augmentations and then get more from each example, the different weather conditions, et cetera. Okay, what else can be found in the data set? It's very interesting to look at the rarity signal and find edge cases. So here you can see. Let's see if you can see. Yeah, you can see. So you can see two people that are taking a ride on a bus, but not inside the bus, but, but outside the bus. By the way, where can you find rare examples? We have data from all over the world. So rare examples are coming mainly from India, China and Israel. So this one is from Israel. You can see it's a garbage truck that had an accident. It's a rare combination of both an accident and a garbage truck. This one is from I think China, I'm not sure. We talked about hay in a haystack. So here we see a haystack as a vehicle. Here you can see an airplane that looks like it's going to land on the road. And here is an example of a very, very novel way how to park your car. So there are many goodies in the data, you just need to find the right way to find them. Okay, so to wrap up what I argued is that heavy tail distribution plus uncertainty leads to to poor signal to noise ratio. And this is a root cause of very, very slow scaling in foundation models and also in end to end self driving models. And in order to scale efficiently, we need to separate failure discovery from failure resolution. Scenario boosting is an adaptation of a very, very old idea of adaboost into the modern world where instead of maximizing over individual examples, we maximize over scenario. And basically what we are aiming to do is to automate the data analyst job that looks at a model, looks at failure classes, tries to figure out what's going on and find more examples. We want to automate it in order to make all the training look much more automated and efficient. So the meteor agent automatically discovers, validates and generates rare safety scenarios. There are links here for a blog post that we wrote where we have the formal theorem and proofs about how heavy tail distribution plus uncertainty leads to poor snr. Thank you very much.