Greetings and welcome to a brand new segment on our blog: Under the Hood featuring industry experts. In this series, we’ll be peeling back the layers and diving deep into the exciting new features that we’re developing here at Foretellix.

To kick things off, we’re thrilled to have Bindesh Patel, Foretellix’s Senior Product Manager for Foretify Manager join us. Bindesh has been instrumental in spearheading the development of the new triage features in Foretify Manager, a powerful new addition to our product line.

Today’s interview takes place in a Q&A format, giving you the chance to gain firsthand insights from Bindesh on the inspiration behind triage, its technical aspects, and how it will benefit  customers. So, put on your thinking caps and get ready to explore the exciting world of V&V triage.

Welcome Bindesh.

Thanks for having me. Foretellix offers a powerful platform called Foretify that lets you scale verification and validation (V&V) testing to handle a massive number of test scenarios. Alongside this, we introduced the concept of coverage metrics to objectively measure how complete your V&V process is. This was implemented with Foretify Manager acting as the central hub, reporting all these metrics throughout the V&V process.

Now, tracking completeness is important, but V&V projects also need to address the issues and bugs that get uncovered. Foretify expands the scope and depth of tests, along with the scalability, which naturally exposes more bugs. This is a good thing – we want to find bugs! But to handle this effectively, we need a whole new set of features. That’s why we recently added triage functionality to Foretify Manager. It automates the identification and clustering of all these issues into more manageable groups that are actionable.

“Actionable” means they can be easily assigned to team members for further debugging and analysis, and can also connect to other tools like ticketing systems. The benefit of all this is a boost in engineering productivity. Tasks that were previously done manually outside the workflow, in a fragmented and tedious way, can now be unified under Foretify Manager. Additional features have also been added to automate many of these triage-related tasks, leading to even greater efficiency.

Please explain the triage features of Foretify Manager

When you run scenario-based testing at scale, you typically get slammed with a huge number of errors, especially early on in the process. Foretellix technology lets us scale up the number of tests we run, and that naturally exposes more bugs – which is a good thing, we want to find them!

The challenge is figuring out which errors are the most common, and more importantly, which ones actually affect the system under test (SUT) and which don’t. Maybe they’re caused by the testing environment or other factors. Manually sorting through this at scale is nearly impossible, so it needed automation.

Foretify Manager tackles this by offering guided automation and data views that can be shared among users. This whole package minimizes the need for all those external tools like TDS scripts and spreadsheets.

Why is triage a significant pain point?

Triage in Foretify Manager is basically like giving you a toolbox to sort and group your issues. This is crucial. You can categorize them based on severity, the team or person responsible, priority, or even whether it’s a real bug or not. There are many factors at play, and Foretify Manager lets you tackle them all with this categorization system.

Now, this toolbox is built around the idea of “guided automation” through rules. You basically code these rules, and once they’re set up, they can automatically be applied to future tests and regression runs, saving you a lot of time.

How does the failure triage feature assist engineers in prioritizing and managing the large volume of issues reported from testing at scale?

Typically, users upload a test suite run to Foretify Manager for analysis. They’ve always done this to check completeness (coverage analysis) and analyze issues. The team would first use our features, which I mentioned before, to cluster and categorize issues into unique sets of similar problems that need further investigation.

This clustering is automated using pre-defined rules you create in Foretify Manager. It’s important because it saves time. Plus, the triage process is iterative, meaning the status of these clusters is constantly monitored and refined for quicker resolution – that’s the ultimate goal!

Once you have these automated clusters, they can be assigned to specific owners and even linked to other systems like JIRA, a bug tracking tool. After assignment, owners can easily use Foretify Manager’s existing debug features with just one click. This helps them get right to the root cause of the issue and ultimately fix the system under test (SUT).

Here’s another benefit: If a new regression test appears, say tomorrow or next week, Foretify Manager lets you compare it with previous runs using the same rules, clustering, and views. This helps you ensure your overall V&V process is improving over time.

How does the triage feature work with Foretify Manager?

We’ve mentioned a few features here and there, but let’s dive deeper into some key ones that make this all possible. One of the big things is the concept of attributes. These can be system attributes, like whether a test run passed or failed according to Foretellix. They basically act like key-value pairs.

Now, with triage, we let you create your own custom attributes. These can be set anywhere, from the runs table to the triage views we’ve discussed. But the main way to set them is through rules. You can define these rules either through the user interface or our Python SDK API.

These rules basically tell the system how to set attributes based on other data. The whole point of attributes and rules is to allow you to group similar test runs together.

Once you have these attributes set and the runs are clustered, you can use the customizable triage views inside Foretify Manager. These are basically hierarchical views of the clusters you’ve created based on all those attribute settings. They’re basically clusters that make sense for you – that’s the essence of clustering and categorization. And from there, you can take action on each cluster, as we discussed earlier.

On top of that, you can also set up user-defined aggregations. These let you easily answer complex questions, like “show me the collision rate in cutting scenarios where the speed was above 90 kilometers per hour.” You can easily set up these “aggregates” to display and analyze this kind of data. We also have powerful built-in filtering capabilities, which let you focus only on the relevant information.

All of this boils down to being able to compare test results from different runs, say from different points in time. This allows you to track the trends you’re interested in.

How does the triage feature boost efficiency and scalability in development?

Traditionally, test suite results are uploaded to Foretify Manager for analyzing coverage and key performance indicators (KPIs). Now, with this new built-in triage component, you can seamlessly perform triage tasks for your V&V projects within the same platform. It’s important to remember that without this, most of this work would be done manually and outside the system, maybe using spreadsheets or custom scripts.

Foretify Manager brings everything together. The key benefit? It eliminates data fragmentation, where information gets scattered and lost.

As we’ve discussed, you can set attributes within triage views. These attributes can define things like owners and even link to other tools like JIRA, a popular ticket tracking system. This way, individual owners can focus on the key issues assigned to them. They’ll see the corresponding JIRA ticket right there in Foretify Manager. With a single click, they can open that ticket directly in JIRA.

How do you recommend training engineers to effectively use the triage feature?

To help you get started, we offer a dedicated training module on failure triage. This covers both the methodology itself and how to use our tool effectively. It’s a great resource!

On top of that, our Applications Engineering Team (AET) has a wealth of experience in advanced V&V. They’re also very familiar with this new failure triage feature. They work closely with our customers to help them adopt both the capabilities and the methodologies around it.

Thank you for joining us today, Bindesh, and explaining the new triage features of Foretify Manager. Please visit the Foretify Manager page for more information.

The new solution enables AV training and validation with high–fidelity sensor simulation.  

Ensuring the safety and reliability of autonomous vehicles (AVs) while accelerating time to market and controlling development costs is paramount. Addressing these challenges, Foretellix integrated the NVIDIA Omniverse Cloud APIs to enable high-fidelity sensor simulation on the Foretify™ platform. The new solution revolutionizes the way AVs are trained and validated by providing physically based, high-fidelity, end-to-end simulation at scale. The new solution will benefit both emerging AV stacks that rely on end-to-end AI training and stacks that comprise separate perception and planning subsystems. 

This week, NVIDIA and Foretellix are demonstrating the enhanced sensor-simulation workflow at GTC.

“With high-fidelity sensor simulation, the Foretify platform delivers a complete toolchain for training and validating autonomous vehicles,” said Ziv Binyamini, Foretellix CEO and co-founder. “By integrating the NVIDIA Omniverse Cloud APIs, we enable customers to streamline and automate the development process, reduce development costs, and accelerate the deployment of autonomous technologies on a global scale.” 

“The AV industry is expanding the use of end-to-end AI models, which require physically based sensor simulation,” said Zvi Greenstein, Vice President of Autonomous Vehicle Infrastructure at NVIDIA. “By leveraging NVIDIA Omniverse Cloud APIs, Foretellix is enhancing its Foretify toolchain with new sensor simulation capabilities and setting a new standard for AV development and validation.” 

The Need for High-Fidelity Sensor Simulation  

Modern AVs rely on multiple sensors, including video cameras, radars, and lidars, to generate an accurate worldview. The AV must accurately identify, track, and react to many actors, including other vehicles, pedestrians, cyclists, and more. It must also handle real-world aspects such as weather, light and road conditions, and sensor degradation. However, real-world data collection for training and validation is an extensive and costly effort, including test drives, data logging, and labeling.  

While object-level virtual simulation is broadly used to validate the planning and control system, sensor simulation is required to validate the perception system. Furthermore, emerging AV solutions rely on sensor data to train and validate the AI end-to-end.  However, achieving sensor simulation that accurately reflects the behavior and physics of the real world at scale is incredibly challenging. 

Performance and safety issues often result from the interaction between the perception and the planning and control subsystems. When these subsystems are validated separately, some issues can appear only during real-world tests. To ensure the safe operation of the AV, in addition to sub-system validation, the entire system needs to be validated from end to end under a large variability of driving scenarios, perception conditions, and locations. 

Virtual End-to-End V&V 

Foretellix and NVIDIA can deliver physically accurate, virtual, end-to-end validation at scale by enabling high-fidelity sensor simulation via the Omniverse Cloud APIs. The integration provides seamless access to NVIDIA’s breakthrough sensor simulation performance and Foretellix’s state-of-the-art automated test generation and monitoring technology. 

Foretellix customers can leverage abstract scenarios to automate the generation of end-to-end tests, perform large-scale triage, assess performance and KPIs, and measure ODD coverage. Figure 1 below shows a high-level diagram of the test development flow. 

The Foretify Core test generator uses abstract scenarios created by customers using Foretify Developer or ready-made scenarios imported from V‑Suites – the industry’s most comprehensive OpenSCENARIO® 2.0 (DSL) library – to generate and execute concrete tests comprised of multiple actors, including maneuvers and perception parameters. It provides mission control information to the AV stack, actor control commands to the simulator, and world-state and perception parameters to the NVIDIA Omniverse Cloud APIs. 

The Omniverse Cloud APIs render physically accurate sensor data based on the simulated world state and the provided perception parameters. It delivers the sensor data to the AV stack, which processes it and generates actuation commands for the simulator. 

Foretify Core executes the test by controlling the simulator at runtime. It uses advanced behavioral models to dynamically control and adjust each actor’s actions during simulation execution to achieve the scenario intent. Foretify Core is pre-integrated with most industry-leading simulators and can also integrate with OEM proprietary platforms, such as the Nuro simulator, as demonstrated at NVIDIA GTC.  

With these new capabilities, the Foretify Platform can measure ODD coverage across multiple perception parameters to analyze and report which conditions have been tested in various driving scenarios. This significantly improves the ability to assess the system’s safety and testing efficiency. 

High-Fidelity Reconstruction and Smart Replay of Drive Logs 

Companies developing AVs collect a huge amount of drive logs through physical drives. Storing all sensor data requires enormous storage space; thus, often, only object-level information is stored. Foretify LogIQ enables object-level drive log ingestion into the high-fidelity virtual simulation flow to reconstruct and replay a physically accurate sensor-level simulation of the drive log (Figure 2). This provides significant time and cost savings in developing, debugging, and issue resolution. In addition, the Foretify Smart Replay capability can generate variations of the original drive log, including perception parameter variations and maneuver modifications, to improve the validation of new AV software versions. 

Generate Synthetic Data Sets for Training 

Retraining AI systems is often required when the AV underperforms in certain conditions, such as unknown edge cases found during validation in physical driving or virtual simulation. Using Foretellix Foretify and NVIDIA Omniverse Cloud APIs, developers can now automatically generate a large number of sensor-accurate data sets to train the AV. This is especially useful when it is unsafe, costly, or difficult to recreate similar conditions in a test drive. 

Enhance Safety, Improve Time to Market, and Reduce Development Costs 

The Foretify platform with Omniverse Cloud APIs powers a major shift in AV testing, accelerates AI training, enables much higher utilization of drive logs, and improves the synergy between physical drives and virtual testing. It enables OEMs and AV developers to perform end-to-end, physically accurate simulations to help improve AV performance and safety, accelerate time to market, and reduce development costs. 

To learn more about the Foretify platform, visit www.foretellix.com/foretify-toolchain-overview/. To learn more about NVIDIA Omniverse Cloud, visit www.nvidia.com/en-us/omniverse/. 

Ensuring the safety of Autonomous Vehicles (AVs) requires testing millions of scenarios using virtual simulation to cover the nearly infinite number of scenarios that AVs might encounter. The goal is to ensure thorough ODD coverage and uncover edge cases that could be a significant safety risk. Doing this efficiently and economically is a considerable challenge for the AV industry. 

Maintaining our leadership in safety-driven V&V technology, we are thrilled to announce the Foretify 2024 release featuring Foretellix’s newest constrained-random test generation technology. The new test generator represents the culmination of tens of man-years of research and development, leveraging technology proven over decades of verification and validation of highly complex systems. The new Foretify release delivers 10X performance improvement compared to the previous generation. It is the automotive industry’s most efficient test generation, execution, and management solution.  

Foretify utilizes the latest ASAM OpenSCENARIO® 2.0 (DSL) scenario description language to deliver best-in-class V&V automation at scale. Foretify dramatically reduces the tedious, manual effort of creating tests by automating the generation of an unlimited number of concrete, meaningful, and valid test scenarios to cover the ODD optimally. It also stresses edge cases and exposes unknowns to identify safety and performance issues. It has driving domain semantic awareness, which enables it to choose parameter values consistent with each other, all system constraints, and the laws of physics.  

Early customers using the new Foretify release have reported over 10X improvement in the test generation and execution time of complex scenarios. This translates to higher efficiency, more productivity, shorter time to market, faster ODD expansion, and cost savings in developing and validating advanced ADAS and AVs. 

Figure 1 shows a high-level diagram of the Foretify Test Generator. The Test Generator receives an OpenScenario® 2.0 (DSL) scenario and an OpenDRIVE® map as inputs. The Planner automatically chooses a location on the map where the scenario can be executed as intended, and it calculates initial waypoints for each of the scenario actors while considering all global and scenario constraints. Changing the test seed automatically creates a new scenario variant at a different map location with random scenario parameters within the given constraints. The Driver executes the test by controlling the simulator and the AV stack at runtime. It uses advanced behavioral models to dynamically control and adjust each actor’s actions during simulation execution to achieve the scenario intent. The Completion Checker checks that the scenario intent captured in the OpenSCENARIO 2.0 code is achieved and automatically flags incomplete scenarios. 

Foretify is the leading Safety-Driven V&V platform. The platform is used by engineers at OEMs, Tier-1 suppliers, and AV stack providers to accelerate the development and deployment of safe Automated Driving Systems and ADAS.  

For more information about the Foretify platform, visit www.foretellix.com/foretify-platform.
To request a demo, please get in touch with us at info@foretellix.com.