Unlocking Scalable Neural Reconstruction for AV Development

Short on time? Below is a podcast summary of this blog, generated by Google NotebookLM.

Neural reconstruction has emerged as a promising technology, enabling the creation of realistic 3D simulation from real-world drive data, to validate increasingly complex systems across a vast range of scenarios, including rare edge cases, without compromising safety while keeping to development schedules.

This blog highlights a powerful solution from Foretellix, leveraging Foretify’s built-in open architecture foundational approach, with integrations for AV developers to adopt neural reconstruction technologies, including NVIDIA NuRec and Parallel Domain Replica. By combining Foretellix’s scenario-based validation and automation toolchain, in particular the Foretellix Smart Replay technology, with neural reconstruction technology, developers can seamlessly generate, replay, and vary reconstructed scenes at scale.

Why Neural Reconstruction Is Becoming Mission-Critical

AV development has entered an era where end-to-end models and AI-powered autonomy demand richer, more varied data than ever before. Yet real-world testing alone can’t meet these needs. It’s too slow, too expensive, and too dangerous for rare or critical situations. Neural reconstruction offers a compelling alternative by transforming raw sensor logs into fully rendered, simulation-ready environments with unmatched realism, flexibility to vary conditions, and reusability across development and testing pipelines.

Common Challenges in Neural Reconstruction Workflows

Before diving into how the solution addresses industry needs, it’s important to highlight the common challenges we see AV teams facing in neural reconstruction workflows:

• Snippet discovery: Sifting through massive log datasets to find specific events (e.g. ego left turn with an oncoming agent at 30–40 mph) remains a slow, manual task.
• Test case definition & maintenance: It’s difficult to define, validate, and maintain robust tests based on reconstructed scenes.
• Dynamic scenario variation: Changing actor behaviors or environmental parameters across reconstructions is often tedious and error prone.
• Hybrid scenario creation: Combining real-world snippets with inserted synthetic actors (e.g. inserting a cut-in vehicle) demands fine-grained control.
• Coverage visibility: Teams lack insight into whether their reconstructed scenarios meet coverage goals or if they leave safety gaps that may limit the scalability and reusability of neural reconstruction in production AV workflows.

An Integrated Workflow from Raw Logs to Safety-Driven Simulation

To address these real-world challenges, Foretellix’s Foretify data-automation platform offers a cohesive, closed-loop solution enabling teams to transition from event detection to varied simulation quickly, bringing the promise of neural reconstruction into production-grade validation and training environments. The workflow is designed to move effortlessly from raw driving logs to simulation-ready, safety-focused scenarios, enabling teams to automate and scale what was once a manual, fragmented process:

At a high level, here’s how the workflow works:

1. Snippet Search and Selection: Using Foretify Evaluate, engineers query large scenario datasets to identify relevant real-world driving events based on scenario intent (e.g., occluded pedestrian crossing from the right side at a given speed) and extract snippets to be reconstructed, modified, and replayed.

2. Scene Reconstruction: The selected snippets are passed to NVIDIA NuRec or Parallel Domain Replica for neural reconstruction. These tools convert raw logs into 3D digital twins with sensor-level fidelity. As outlined in the following steps, engineers have the flexibility to modify and render novel scenes from the reconstructed environments.

A reconstructed scene with Parallel Domain

3. Scene Variation and Scenario Creation: The Foretellix toolchain allows users to modify reconstructed scenes, adjusting timing, agent behavior, or environmental conditions. For example:

Replay original or modified trajectories:
By leveraging Foretellix’s Smart Replay technology, users can configure the behavior of each actor in the scene with fine-grained control. Actors can be set to:

• Follow the exact trajectories and behaviors recorded in the original real-world event
• Follow a slightly modified trajectory to explore variations around the original behavior
• Be controlled by a reactive behavioral model for dynamic response within the scenario

Additionally, individual actors can be selectively removed from the scene to simplify the environment or isolate specific interactions.

In the video below, a side-by-side comparison of two scenes rendered with NuRec is shown – the scene on the bottom half has been modified with Foretify Smart Replay

Modify trajectories, insert new actors:
Leveraging Foretify Smart Replay, in combination with the power of Foretellix’s abstract V-Suite libraries, users can also:

• Insert new actors (either harvested from other scenes or synthetic)
• Create novel scenarios with inserted actors, leveraging the Foretellix V-Suite libraries

Actors can also significantly deviate from original paths to explore what-if conditions, alternative maneuvers, or edge-case variations. In the video below, the rendering of a novel scene with PD Replica is shown; the inserted actors are synthetic:

4. Validation and Coverage Feedback: Foretify Evaluate ingests both reconstructed scenarios and existing driving logs,  connecting them both to unified safety KPIs and ODD coverage metrics via built-in dashboards, allowing teams to close gaps and ensure validation completeness.

Use Cases and Applications

The integrated workflow unlocks a range of use cases across AV development:

• Training AI Models: ML engineers can generate diverse training datasets by reconstructing rare scenarios and creating safe variations
• Closed-Loop Simulation: V&V teams can replay real-world edge cases and simulate novel actor behaviors to assess AV safety responses
• Hybrid Scenario Creation: Developers can insert new agents into existing scenes to test system robustness under complex interactions
• Safety Evidence Generation: Teams can automatically link each test to scenario coverage dashboards and safety KPIs

For example, a developer might identify a left-turn event from a fleet log, reconstruct it using NuRec or PD Replica, insert a synthetic speeding vehicle using Foretify Generate, and assess how the AV planner responds, all within a single workflow.

What AV Teams Gain from This Integration

By closing the loop from real-world event to simulation-ready variation to KPI tracking, the combined solution from Foretellix with NVIDIA and Parallel Domain offers several benefits:

• Dramatically faster scenario creation from real-world logs
• High reusability of reconstructed data, reducing dependence on costly physical testing
• Scalable variation of edge cases, enabling robust AI training and safety validation
• Improved coverage tracking, ensuring confidence in both rare and routine scenarios

Laying the Groundwork for Safer AI-Driven Autonomy

As AV stacks become increasingly AI-driven, teams need simulation workflows that match the pace and complexity of development. These integrated solutions powered by Foretellix’s data automation platform and leading neural reconstruction tools from NVIDIA and Parallel Domain make it possible to go from real world drive logs to validated scenarios with unprecedented speed and control.

It’s a major step toward safer, more efficient AI AV development.

Want to see this integration in action? Contact us to learn more.