Waymo leverages Genie 3 to create a world model for self-driving cars
Waymo's World Model Revolutionizes Self-Driving Car Simulations
Google-spinoff Waymo is expanding its self-driving car fleet into new regions, with over 200 million miles of driving experience informing how the vehicles navigate roads. However, the company's AI has also driven billions of miles virtually, and there's a lot more to come with the new Waymo World Model. Based on Google DeepMind's Genie 3, Waymo says the model can create "hyper-realistic" simulated environments that train the AI on situations that are rarely (or never) encountered in real life—like snow on the Golden Gate Bridge.
The Limitations of Real-World Training Data
Until recently, the autonomous driving industry relied entirely on training data collected from real cars and real situations. That means rare, potentially dangerous events are not well represented in training data. For instance, a self-driving car may never encounter a pedestrian crossing the road at night, or a vehicle driving on a snowy road. This lack of diversity in training data can lead to poor performance in real-world scenarios.
The Power of Genie 3
Google revealed Genie 3 last year, positioning it as a significant upgrade over other world models by virtue of its long-horizon memory. In Google's world model, you can wander away from a given object, and when you look back, the model will still "remember" how that object is supposed to look. In earlier attempts at world models, the simulation would lose that context almost immediately. With Genie 3, the model can remember details for several minutes.
Simulating the Real World with Genie 3
Autoregressive world models like Genie don't actually create 3D spaces, but instead render video quickly enough that it feels like an explorable world. Naturally, video games are cited as a prime application for world models, so much so that gaming company stocks dropped when Google recently expanded access to the technology as Project Genie. However, the latency and still rather short memory of Genie make gaming uses far from a certainty. Nevertheless, Waymo says Genie 3 is actually ideal for simulating the kind of data it needs to train self-driving cars.
The Waymo World Model
The Waymo World Model is not just a straight port of Genie 3 with dashcam videos stuffed inside. Waymo and DeepMind used a specialized post-training process to make the new model generate both 2D video and 3D lidar outputs of the same scene. While cameras are great for visualizing fine details, Waymo says lidar is necessary to add critical depth information to what a self-driving car "sees" on the road—maybe someone should tell Tesla about that.
Driving Action Control
Using a world model allows Waymo to take video from its vehicles and use prompts to change the route the vehicle takes, which it calls driving action control. These simulations, which come with lidar maps, reportedly offer greater realism and consistency than older reconstructive simulation methods.
Synthetic Scenes and Mutating Real Videos
The Waymo World Model can also help improve the self-driving AI even without adding or removing everything. There are plenty of dashcam videos available for training self-driving vehicles, but they lack the multimodal sensor data of Waymo's vehicles. Dropping such a video into the Waymo World Model generates matching sensor data, showing how the driving AI would have seen that situation.
Real-World Applications
Waymo's early test cities were consistently sunny (like Phoenix) with little inclement weather. These kinds of simulations could help the cars adapt to the more varied conditions. The new markets include places with more difficult conditions, including Boston and Washington, D.C.
The Future of Self-Driving Cars
The benefit of the new AI model will depend on how accurately Genie 3 can simulate the real world. The test videos we've seen of Genie 3 run the gamut from pretty believable to uncanny valley territory, but Waymo believes the technology has improved to the point that it can teach self-driving cars a thing or two.
Conclusion
The Waymo World Model is a significant step forward in the development of self-driving cars. By leveraging the power of Genie 3, Waymo can create hyper-realistic simulated environments that train the AI on situations that are rarely (or never) encountered in real life. This technology has the potential to improve the safety and efficiency of self-driving cars, and could lead to a future where autonomous vehicles are a common sight on our roads.
However, there are still many challenges to overcome before self-driving cars become a reality. The development of this technology requires significant investment and expertise, and there are many technical and regulatory hurdles to navigate. Nevertheless, the potential benefits of self-driving cars make them an exciting and promising area of research and development.
As the world continues to evolve and become increasingly complex, the need for autonomous vehicles will only continue to grow. The Waymo World Model is an important step towards making this vision a reality, and we can expect to see significant advancements in this area in the years to come.
