Authors: Katie Washabaugh, Product Marketing Manager of AV Simulation at NVIDIA and Ohad Schwarzberg VP Product and AI at Foretellix
Foretellix is expanding its integration with the NVIDIA Omniverse Blueprint for Autonomous Vehicle (AV) Simulation to include NVIDIA Cosmos Transfer World Foundation Model and Video Data Search. These newly announced capabilities enhance Foretellix Foretify’s behavioral scenario generation with greater diversity in weather, lighting, and geolocation. Additionally, Foretify’s advanced behavioral search is enhanced leveraging Cosmos’s sensor domain search capabilities. These enhancements accelerate AI-powered AV training and validation for faster development and deployment. Foretellix, one of the first AV toolchain providers to integrate NVIDIA Omniverse, is now adopting Cosmos Transfer for enhanced AV simulation.
There is a change happening in the AV industry. It’s in a state of transition. Moving from rule-based architectures to hybrid AI-powered AV stacks that blend AI’s capabilities with rule-based safety measures. This is being driven by advances in deep learning and computational power, along with the cost and time demands of traditional software-based AV development.
End-to-end models integrate AV functions using deep neural networks, mapping sensor inputs (e.g., cameras, lidar, radar) directly to driving actions. This approach improves adaptability, reduces latency, and enhances generalization to novel scenarios by leveraging large-scale driving data.
While real-world data collection provides a foundation for AV development, it is extremely difficult to cover the full variation of physical aspects of every scenario (weather, lighting, speed, distance, etc.), rare edge cases, and the scenarios in the full operational design domain (ODD). However, this type of data is essential for accelerating safe AV deployment and can be generated with cost-effective data automation of massive amounts of varied and realistic data.
Foretify + Cosmos Transfer
By leveraging NVIDIA Cosmos Transfer as part of Foretellix’s integration of the NVIDIA Omniverse Blueprint for AV Simulation, Foretellix’s data-automation development toolchain is delivering even greater scalability and diversity in data for effective AV training and validation for AI-powered stacks. This integrated solution provides users with powerful capabilities, including:
- Foretify’s behavior layer for scenario evaluation and generation
- NVIDIA’s Omniverse and Cosmos Transfer for high-fidelity sensor simulation
NVIDIA Cosmos Transfer is part of the Cosmos platform of state-of-the-art generative world foundation models (WFM), advanced tokenizers, guardrails, and an accelerated data processing and curation pipeline built to accelerate the development of physical AI systems such as autonomous vehicles and robots. Cosmos Transfer connects to Omniverse to generate new data within physically based virtual environments, turning any driving scenario generated by Foretellix Foretify into hundreds of realistic variations.
Foretellix Foretify is a data automation toolchain that accelerates the development of AV systems by combining real-world driving data with advanced virtual simulations and using it for both training and validation. Foretellix’s “Evaluate >> Generate” approach to AV development can be broken down into four stages:
- Foretify unifies, curates, and cleanses existing real-world drive log and sim data
- Foretify evaluates the data to validate the performance and quality in specific drive logs according to defined KPIs
- Foretify generates variations of real-world scenarios and uses the physics-based rendering capabilities of Omniverse and Cosmos Transfer to create hyper-realistic sensor simulation data to train the AI-powered AV stack
- Foretify evaluates the complete drive data to measure ODD coverage, use as data driven evidence for safety reports, identify gaps, and generate abstract scenarios to close the gaps for system validation
The NVIDIA Omniverse Blueprint for AV Simulation is a reference workflow to create rich 3D worlds for training, testing, and validation. The blueprint contains APIs and services, such as Sensor RTX APIs for sensor simulation, to build and enhance digital twins from real-world sensor data, model physics and behavior of dynamic objects in a scene and generate physically accurate and diverse sensor data.
With this API-based architecture, the blueprint seamlessly integrates into existing workflows and tools like Foretify. This enables developers to replay driving data, generate new ground truth data, and perform closed-loop testing.
With Cosmos Transfer, Foretify can amplify the diversity of sensor data rendered by Sensor RTX, varying the visual appearance of the scene, from weather to graffiti to dirty lenses—even to new ODDs. Cosmos Transfer Video Data Search includes a robust capability for presentation level video search and high-fidelity rendering. Foretify’s native behavioral search capability at the logic level enhances scenario evaluation and generation, seamlessly complemented by Cosmos Transfer. The result is seamless scalability in data for training and validation, easing costly bottlenecks in the AV development pipeline.
Looking Ahead
As AVs become increasingly autonomous and rely more on AI to power the AV stack, the need for a controllable, predictable and repeatable data-automation toolchain to validate and train the AV stack will only grow. Foretellix’s integration of Cosmos Transfer via the NVIDIA Omniverse Blueprint for AV Simulation represents a major step forward in empowering developers to create the meaningful and realistic data required to rapidly develop, train and validate robust and safe AI-powered AV solutions at scale. By bridging advanced scenario-based training with high-fidelity simulation, this partnership is helping pave the way for the future of autonomous mobility.
See Innovation in Action
Are you ready to see the Foretellix Foretify development toolchain with high-fidelity sensor simulation in action? Visit Foretellix at NVIDIA GTC in San Jose at booth 1543 from March 18-21 or visit the Foretellix website to discover how this groundbreaking integration can supercharge your autonomous vehicle development process and help you launch with confidence.
