Short on time? Below is a podcast summary of this blog, generated by Google NotebookLM.

The automotive industry is undergoing a transformative shift. For the first time, regulators are acknowledging virtual testing and simulation as valid inputs for certification and type approval in the upcoming UN regulation #152. This is not just a minor update—it’s a significant milestone that fundamentally changes the way safety and performance standards are approached, especially for advanced vehicle systems like Advanced Emergency Braking Systems (AEBS).

UN regulation #152 allows virtual testing to be recognized as part of the AEBS certification process and is expected to be finalized by Q2 of 2025. While AEBS is the initial focal point, this amendment opens the door for virtual testing to potentially become a standardized method across various vehicle safety systems. For the industry, this shift in regulatory acceptance is not only necessary; it’s inevitable. As advanced vehicle systems become increasingly complex, traditional testing methods alone are insufficient to address the full spectrum of scenarios these systems must handle.

For companies like Foretellix, this regulatory change is both validating and promising. At Foretellix, we have long advocated for scalable, virtual testing solutions to support the growing demands of Driver Control Assistance Systems (DCAS) and advanced driving assistance systems (ADAS). Foretellix’s data-driven autonomy development toolchain is purpose-built for these challenges, enabling a level of scenario testing and validation that traditional methods cannot match. This amendment represents a critical role of virtual testing in meeting regulatory standards efficiently and effectively.

The Regulatory Shift: Moving from Physical to Virtual Testing in Vehicle Certification

Historically, certification and type approval processes for vehicle safety systems have relied almost exclusively on physical testing. Physical tests provide direct evidence of system reliability under controlled conditions, but they come with limitations. They are costly, time-consuming, and lack the flexibility to cover the wide array of real-world scenarios that autonomous vehicles (AVs) might encounter. Physical testing alone cannot feasibly replicate the myriad situations that vehicles face, particularly as AV and ADAS technologies grow in sophistication.

The UNECE’s Working Group on Automated/Autonomous and Connected Vehicles (GRVA) is addressing this challenge head-on. In the upcoming amendment to UN regulation #152, virtual testing can be used to certify the system. This regulatory change is a direct response to the industry’s need for more comprehensive and scalable testing methods that can keep pace with rapid technological advancements.

This amendment to the AEBS certification process is a pivotal moment. Not only does it officially recognize virtual testing as a credible and practical alternative to physical testing, but it also reflects a growing understanding among regulators that the industry needs a broader, more adaptable approach to testing. The UNECE/WP.29’s adoption of virtual testing for AEBS approval is a first step, and its success could pave the way for expanded virtual testing across other vehicle safety systems.

Source: https://unece.org/sites/default/files/2024-01/GRVA-18-50e_1.pdf

The New Testing Framework: Balancing Physical and Virtual Testing

Under the proposed framework, vehicle manufacturers will be allowed to use virtual testing for specific certification requirements. To maintain robust standards, at least 30% of the required tests must still be conducted physically, with at least one test for each scenario variant relevant to approval. Importantly, the virtual tests must be validated in accordance with Annex 4 of the regulation, ensuring that the simulations meet stringent reliability and accuracy standards.

This 30% physical testing requirement represents a balanced approach that blends the strengths of both physical and virtual testing. By retaining a portion of physical tests, the framework safeguards the integrity of the certification process, while the inclusion of virtual testing allows for broader and more complex scenario coverage. The 30% requirement is viewed as an initial benchmark, and there is an expectation that this threshold will decrease as virtual testing technology advances and gains further acceptance in regulatory circles.

This gradual shift toward a greater reliance on virtual testing is essential. As vehicle systems become more intricate, so do the scenarios in which they must operate safely. Virtual testing allows manufacturers to simulate an extensive range of driving conditions—from rare weather events to unique traffic scenarios—offering unparalleled insight into system performance and safety.

Foretellix’s Role: Leading the Virtual Testing Revolution

At Foretellix, we’ve anticipated this regulatory shift and have designed our Data-Driven Autonomy Development Toolchain to meet precisely these types of challenges. Our toolchain is built to facilitate a new era of vehicle validation, offering advanced scenario generation and evaluation to support comprehensive, scalable testing solutions. Our approach goes beyond merely addressing regulatory requirements—it enables our clients to achieve a depth of validation that sets a new standard for safety and reliability in the industry.

With Foretellix’s toolchain, manufacturers can simulate virtually any scenario, from common driving situations to extreme edge cases. The platform’s data-driven approach allows for in-depth analysis and validation, which is essential for meeting both current and future regulatory demands. By enabling our customers to test and validate their systems at scale, Fortellix’s development toolchain is more than a response to regulatory changes—it’s a proactive solution designed to address the fundamental challenges of AV and ADAS development.

The Broader Implications: What This Means for the Industry

This amendment marks the beginning of a new era for the automotive industry. By acknowledging virtual testing as part of the certification process, regulators are embracing a more comprehensive approach to safety validation. This shift holds significant implications for the future of AV and ADAS technologies, including:

  1. Efficient AI Validation:
    Support the shift to AI-centric end-to-end stacks by evaluating the performance of the entire system from sensor inputs to decision-making outputs, to uncover risks, edge cases, and potential gaps.
  2. Realistic Scenarios at Scale:
    Generate any number of training and test scenarios under diverse environmental, weather, and traffic conditions, including hazardous situations too dangerous to recreate on actual roads.
  3. High-Fidelity Sensor Simulation:
    Reduce dependency on physical prototypes by utilizing virtual testing to significantly lower development expenses and make cutting-edge safety systems more accessible to all.
  4. Continuous Improvement:
    Rendering physically based sensor data for camera, radar, and lidar, enabling the creation of training and testing datasets and supporting closed-loop testing to enhance development efficiency.

While AEBS is the initial focus of this amendment, the potential for virtual testing to expand across other vehicle systems is substantial. The automotive industry is increasingly moving toward a holistic, system-level approach to vehicle safety, and virtual testing is a natural fit for this evolution.

Looking Ahead: The Future of Vehicle Certification and Virtual Testing

The UN Regulation #152 amendment brings the industry closer to a future where virtual testing is an integral part of safety validation. As this approach gains acceptance, we can expect to see a gradual reduction in the required percentage of physical tests, further reinforcing the role of virtual testing as a core component of certification.

As the industry continues to evolve, Foretellix remains dedicated to pioneering new standards for vehicle validation, ensuring that advanced vehicle systems meet the highest safety and performance standards. With the power of virtual testing and the capabilities of Foretellix’s platform, the future of vehicle certification is not only achievable—it’s here.

 

‘A method of solution is perfect if we can foresee from the start, and even prove, that following that method we shall attain our aim.’ — Gottfried Wilhelm Leibniz

Introduction

Automated Driving Systems (ADS) force a paradigm shift in safety. ADS technology is expected to assume total ownership of driving safety despite the elements of surprise, chaos, and unpredictability inherent in the driving environment. This imposes unprecedented safety expectations on engineered artifacts and the organizations that create them. Fulfilling these expectations requires substantial innovation in Safety Verification and Validation (V&V), especially for ADS functionality that understands the driving environment, predicts how it may evolve, and plans safe maneuvers. This blog post describes critical elements of Foretellix’s innovative Safety-Driven V&V (SDV) approach and how SDV provides the paradigm shift for ensuring the safety of ADS and ADAS at scale.

SDV: The methodology

The SDV methodology is established to ensure that the Safety V&V effort contributes to and enables all the safety assurance needs for ADS and ADAS. It follows the current safety standards and upcoming regulations and can be tailored to the needs of specific companies, engineering cultures, and validation philosophies. The application of the methodology forces deep thinking about the safety claims to be made, supporting evidence, testing strategies and modalities, evaluating residual risk, etc. This leads to a comprehensive and cohesive safety validation program that provides a clear and actionable guide to crossing the safety finish line. In particular, SDV includes support for the following:

  • Safety case: The SDV tools and methodology generate artifacts that serve as evidence for safety case claims related to test results, coverage, etc. With the proper tooling and processes, the continuous and iterative generation of evidence artifacts serves as a basis for the so-called “continuous safety case” approach, in which safety considerations are defined and tracked continuously throughout the lifecycle.
  • Residual risk estimation: All approaches to residual risk, regardless of specifics, need to account for whether sufficient testing has occurred, whether the results are consistent with the organization’s views on risk acceptability, and the likelihood of undiscovered safety issues. In turn, Foretellix provides the foundational tools for supporting this and is working with leading customers on developing more abstract methodological guidance on how test spaces could be structured and discretized.
  • The evolution of Safety/V&V programs across the development lifecycle: The Safety and V&V activities in the early stages of feature development are very different compared to those done during pre-launch validation of a feature-complete ADS/ADAS stack. SDV has the depth and breadth to “tune the focus and intensity” of V&V to immediately obtain valuable results for the development stage. As the system matures, the same tools and processes can ramp up and build off the previous results for wider and/or more focused V&V explorations.
  • Bringing together the teams engaged in core autonomy development, V&V, and Safety: SDV methodology requires the teams to come together and define which requirements and goals must be explicit, how they should be articulated and tested, how the results are evaluated, etc. The SDV tools generate artifacts (e.g., test results, coverage reports) that need to be jointly viewed and discussed by the teams, typically as part of explicitly defined processes for this purpose.
  • Post-deployment monitoring: Harvesting field data from deployed fleets and using that to continuously increase confidence in the Safety Case, validate safety assumptions, and proactively identify any needed changes (and subsequently validate them) are all activities that are an integral part of SDV. Upcoming regulations will also require this type of post-deployment analysis and strengthening of the safety validation.
  • ODD expansion: SDV also increases the efficiency of V&V when expanding to new ODDs… be that about expanding geo-fences or expanding to more operational and weather situations within the same geo-fence.
  • Standards and regulations: SDV supports relevant parts of all modern ADS/ADAS safety standards and regulatory requirements, including but not limited to ISO 26262, ISO 21448, ISO 34502, UL 4600, IEEE 2846, and UNECE regulation 157. The workflow supported by the SDV methodology is aligned with the regulatory direction developed by UNECE.

SDV: The technology

SDV’s key technical offerings are:

  • The ASAM OpenSCENARIO® 2.0 language to elegantly describe abstract scenarios in a concise, declarative, arbitrarily combinable, reusable, and formal-yet-intuitive manner. Scenario descriptions include scenario parameters, their ranges and distributions, constraints, and other metrics (KPIs/SPIs, coverage, etc.) to be computed and checked as the scenario executes. These abstract descriptions are reusable across maps and various ODDs.
  • Constrained-random generation of a large number of valid concrete scenarios from the abstract scenario descriptions. The generator doesn’t just randomly choose scenario parameter values within their ranges. (Doing that generates many silly/impossible scenarios requiring subsequent pruning.) Instead, it understands the semantics of the driving domain and picks scenario parameter values consistent with each other and the laws of physics. Appropriate map locations are also automatically selected. For example, if an actor is behind the Ego vehicle at the start of the scenario and ahead of it at the end of the scenario, the generator automatically infers that a map with a minimum of two lanes would be needed and lane changes, overtaking, and a subsequent cut-in by actor ahead of Ego would all be involved.
  • Seamless use of real-world driving logs in Safety validation. The tools convert driving logs into a timeline of scenarios and automatically compute scenario parameters, their distributions, and scenario metrics. This enables checking AV performance, detecting anomalous behavior, assessing simulation accuracy, etc. This also allows assessments of test coverage across real-world and virtual testing.
  • Sophisticated analysis tools expressly designed to enable efficient examination and arbitrary exploration of millions of scenario simulations and real-world driving results. From debug exploration of single scenarios to triage of large batches of simulation runs to statistical queries across multiple runs.
  • Test suite management and optimization to guide iterative testing towards specific test objectives. Whether you want to run the most common scenarios, corner cases, or find situations where particular KPIs/SPIs are minimized, or increase coverage over input values, or efficiently seek out parameter combinations that cause failures, the test suite managers guide the selection of the next set of test cases by analyzing results of previous test runs.

Conclusion

At some point, OEMs will need to bite the bullet, put down the metaphorical pen, and say, “This is it! We are ready for launch.” Getting to that point requires a blueprint that shows how novel safety technologies and methodologies can collectively get you to the finish. SDV is exciting because it lights a path to that promised land.  It adds the missing ingredients and can carry Automated Driving Systems across the safety finish line. Leibniz would approve!

To learn more, download the Safety-Driven V&V Guide

More than 350 years ago, Sir Isaac Newton formulated the theory of gravity and laws of motion. These include the relationships between variables such as speed, time, distance, and acceleration that natural motions universally adhere to. Many years later, ASAM (OpenSCENARIO®2.0.0), VVM (Pegasus family), and ISO 34501 introduced a scale of distinct scenario description levels to differentiate between the available scenario creation styles and technology generations. The scale is called “scenario abstractions levels.” In our ongoing discussions with users, we realize there is still some confusion about the levels and what is required to make them useful for users. In this blog, we define the abstraction levels, explain how to attain the productivity and robustness promised by both logical and abstract scenarios, and even acknowledge Isaac Newton’s contribution to impactful ADAS and ADS V&V projects ?.

What are the four levels of abstraction?

The abstraction levels define a spectrum of scenario descriptions and their related capabilities. Each level adds more scenario control and automation over the previous. While the first three levels of abstraction are formal and intended to be compiled by tools, functional description is geared toward people reading and understanding and thus may include free English, supporting images, and more. The definition of the four abstraction levels is straightforward.

  • Concrete scenarios allow only attribute value assignments
  • Logical scenarios allow assignments or attribute value selection according to distribution from fully specified ranges
  • Abstract scenarios allow any possible constraints, including ranges and distributions, cross attributes, high-level location specification, and timing constraints.
  • Functional scenarios – Geared for humans. Not a formal description; thus, not machine-readable.

If you know the level of abstraction basics, feel free to jump to the next section. If not, here are more details.

The four levels of abstraction are illustrated in the figure below:

Figure 1: ASAM and Pegasus levels of abstraction

With concrete scenarios, all scenario attributes must be carefully calculated and manually assigned by the user. For example, set car1’s speed to 10kph and start the change lane maneuver in a specific XYZ location. (See Figure 2 for a visual image of the cut-in scenario.) All attributes, including maps, vehicle speeds, distances, accelerations, and maneuver locations, must be specified in a concrete scenario. The laws of physics impose multiple dependencies between the scenario set of attributes, for example, the distance is the product of speed and time, or the average acceleration equals speed difference divided by time. Users must also select the proper location according to the scenario’s nature and attributes. For example, at least two lanes are needed for the cut-in scenario, and various speed settings require road segments of various lengths. Failing to provide a consistent set of attributes or locations will likely result in a meaningless scenario with a loss of human and machine resources.

Figure 2: the cut-in scenario – start behind on a side lane and finish ahead in the same lane

The next level of abstraction is the

logical scenario. As with the previous abstraction level, the user must specify a set of locations and attributes, but there is one significant addition – individual attributes can be left unassigned within a fully specified range. For example, instead of hardcoding the speed to 45kph, a speed range between 10 to 90 kph can be provided. The value selection can originate from aggregated real-life data distributions or be biased to force challenging conditions.   The assumption is that the tedious calculation involved with every concrete scenario will be eliminated, and scale will be reached. (More about this assumption below. ?)

The automotive validation journey is a challenging process. Equipped with the requirements, test plan, and gut feeling, there is a need to span spaces and look between the cracks for expected and unexpected bugs.  This is the main motivation for abstract scenarios.

Abstract scenarios allow total freedom for the user to express any desired dependency using any form of constraint, including ranges. Constraints can be:

  • Cross attribute constraints – e.g., the NPC’s speed should be slower than the location’s legal speed.
  • Abstract location constraints – e.g., create a cut-in near a junction. Note that the scenario does not force a specific junction but requires a junction nearby.
  • Timing and execution constraint – g., the change lane should take place with half a second-time headway. There are multiple ways to implement such a request, but the user just wants to capture the timing requirement in a constraint.
  • Control flow and a composition constraint – e.g., the scenario should combine the cut-in and oncoming

Abstract scenarios enable a goal-based approach in which the user can ask for any desired scenario (set the goal for the tool), trusting the tool to deliver multiple consistent results.

Functional scenarios are non-formal descriptions of the desired scenarios, including the goal, motivation, and intent. For example, a non-formal description in English may stipulate that to validate the Ego in cut-in scenarios, we should both force a maneuver from the Ego to avoid danger and ensure that the Ego does not get confused by the cut-in maneuvers. It may include images (such as figure 2 above) to illustrate the scenario to the test writer.

Why are users disappointed by logical scenarios? And what is required to fix that?

Logical scenarios are a significant improvement over concrete scenarios as they capture a desired scenario parameter space, but they are misleading to both users and vendors. Letting the tool select a speed between 10 to 90 kph requires the tool also to consistently adjust starting locations, other vehicle speeds, acceleration, maneuvers timing, and more. This is where Isaac Newton’s discoveries cannot be ignored – tools must be smart enough to propagate the speed selection to other attributes.

This misunderstanding is not a minor one. A project team’s hope of reaching the necessary large-scale test suite and replacing tedious manual effort falls apart due to this. Without a tool capable of proper inferences and propagation of values, logical scenarios may result in a massive number of meaningless scenarios.

Teams that plan to adopt optimization workflow (applying mathematical algorithms to gravitate towards risk) also get their share of disappointment. No automation has been available to take the optimizer value recommendations and create a consistent scenario around them. While some attributes, like time-of-day or road friction, might be independent of others, most scenario attributes are tightly connected and often in surprising ways.

The smart technology that makes the needed inferences is called a constraint solver; the industry now realizes that constraint solvers are critical for getting consistent logical scenarios or enabling optimization workflows. Even if the user does not specify constraints in his scenarios, the system needs to consider implicit constraints (like the laws of physics), which the constraint solver solves. By the way, this understanding is already proven to be mandatory in other industries, something that many of us at Foretellix have done for the last ~20 years.

What is the added value of abstract scenarios?

ASAM and VVM have abstract scenario levels for all the right reasons – abstract scenarios are game changers for scenario-based testing productivity and robustness.

An efficient testing workflow is not about throwing an endless number of scenarios, hoping they will expose bugs; it is a large-scale guided hunt following the test plan and intelligently exploring the implementation cracks. You need complete control over the generated scenarios (as well as other capabilities) to follow the test plan, and you need to combine randomness and the unknown. In your journey to span scenario space and find these bugs, you will constantly refine the scenarios and further direct them toward different areas of concern – abstract scenario controls are a required tool for the job, but it will still be hard work!

One of the most significant value propositions of abstract scenarios is the ability to aggregate generic V&V expertise within executable packages and to enable its use on multiple ODDs. Assume that you have defined a valuable concrete or logical cut-in scenario that challenges the sensitivity of an autonomous driving function. Such a scenario cannot be migrated to various project ODDs without manual adjustments. If it was designed for a sedan Ego, it might not be applicable for a truck with totally different dynamics. You need to manually adjust it to another map or move it from an urban to a highway setting. It must be modified for left-hand street driving if your ODD includes London. For development productivity, you need to combine scenarios modularly and have a technology that can resolve the new dependencies of the combined scenarios. For example, you need to be able to execute the cut-in and oncoming scenarios at the same time and have a tool that can find locations appropriate for both these scenarios. Modeling for adaptability, extensibility, and modularity are some of the basic mandatory capabilities that SW projects cannot do without. These capabilities are finally available for scenario-based testing in the form of abstract scenarios.

At the risk of being repetitive, abstract scenarios also depend on the availability of a constraint solver. The need here is evident because the ODD and scenario constraints are explicit and user-defined, unlike the implicit law of physics and vehicle dynamics constraints.

So which abstraction layer should I use while coding my scenarios?

All abstractions levels might be appropriate based on the requirement and use case. Nevertheless, here are a few essential guidelines.

  • The OSC2 scenario abstraction should be equivalent to or above the requirement specification abstraction. For example, if the requirement asks for a cut-in and lists all scenario attributes, a concrete scenario might be the solution. However, a better choice might be to create an abstract generic cut-in scenario and assign concrete values on top of it. This will be the most effective and productive way to develop and maintain your scenario catalog.
  • Be strategic and see beyond a specific project or ODD’s needs. With the OSC2 language and solver technology, you can create or adopt generic V&V packages. For example, it is highly desired not to hardcode scenarios to a specific map or a location. This means that pure logical scenarios should be discouraged.
  • Scenario abstraction is typically modified as the project progresses, and new realizations are formed. You may initially assume that a logical scenario with specific ranges is needed (ignoring the guideline above?) and later discover additional ODD needs (e.g., vehicle dynamics constraints or location constraints). Enhancing the scenario with such considerations will turn the logical scenario into an abstract one. This means that your toolchain should support all abstractions to maximize control and development efficiently.

Most of the industry acknowledges the need for a massive amount of intelligent, meaningful scenarios. The way to replace the tedious manual work is by defining scenarios in high-level terms and leveraging the constraint solver’s automation to produce known and unknown edge cases. Logical scenarios combined with a constraint solver provide a first level of abstraction. One source of users’ frustration is that tool vendors claim to support logical scenarios but do not provide the necessary solver technology. That being said, only abstract scenarios allow the refined control and reuse required for real-life project productivity and safety. If you wish to know more about abstraction or explore Foretellix’s unique scenario-creation technology, give us a shout at info@foretellix.com.

As usual, drive safe,

Sharon

Safety, together with its Verification and Validation, is the largest unsolved problem in autonomous driving. Safety is also simultaneously a purpose, blocker, enabler, and expected consequence of autonomous driving. Solving it requires radical innovations exceeding those enabling core autonomy capabilities like Perception, Prediction, and Planning. This is because truly effective Safety solutions address not only technology but also mindset, development methodology, and culture. Foretellix is leading the charge, which makes it tremendously exciting to be here.

I have been building autonomous driving systems for over ten years. During that period, I have transitioned from an academic and technical expert to leading AV Architecture, Integration, and Systems/Safety Engineering and V&V organizations for some of the biggest and most advanced players in the game. I am intimately familiar with the Safety/V&V challenges and shapes of good solutions.

Safety Verification & Validation (V&V) probably has the greatest, critically unaddressed scope for technical, methodological, analytical, and tool innovation. These innovations need to go beyond addressing mere technical challenges. They should also naturally break down silos and strengthen the collaboration among the Autonomy development, Verification & Validation, and Systems/Safety activities of the organization through shared artifacts, targets, and workflows. Indeed, with the right enabling tools and workflows, the TICK of development can be matched with the TOCK of V&V. With each TICK-TOCK, autonomy development is harmoniously accompanied by credible maturity in Safety. Edge cases get elicited and understood sooner, there is continuous feedback on where the autonomy is improving and regressing, critical performance and safety indicators get tracked and translated into measures for residual risk, and the safety-case frameworks get updated with numbers and evidence artifacts. All on a per-release, or even a per-sprint cadence.

Making impactful innovations and synthesizing them together into an end-to-end Safety V&V narrative requires a few vital ingredients: A determined focus (including adequate, undiluted resources), continuously accumulating exposure to nominal and edge cases across a variety of Operational Design Domains (ODDs) and autonomy levels, and the raw technical prowess and experience needed to design and implement analysis, optimization, test case generation, and big data tools.

Those vital ingredients are exactly where Foretellix shines. Since 2018, the company has devoted absolute focus to safety and coverage-driven verification and validation of autonomous driving. We have ongoing engagements with some of the most respected OEMs and Tier1s as well as emerging technology providers and AV companies in on-road and off-road ODDs spanning AV & ADAS use cases. These engagements produce a wealth of insights and experiences in characterizing frequent as well as rare situations. The company is also a powerhouse of talent in the verification and tooling space, having delivered proven and enduring success in hardware verification. Also important for me, personally, are Foretellix’s contributions to, promotion, and adoption of open industry standards like ASAM OpenSCENARIO® 2.0.0.

Come and partner with Foretellix to gain unprecedented insights into Safety V&V while you engineer the best AV stack for your mission.

About Sagar Behere

Mr. Behere has over a decade of experience developing autonomous driving systems as an Architect, Integrator, and leader in Systems/Safety Engineering and Validation. Prior to Foretellix, Sagar held critical leadership positions at Aurora, Toyota Research Institute, and Zoox. He has also worked on Safety and Automated Driving research together with OEMs like Volvo Car Corporation and Scania. Sagar holds a Ph.D. in architecture for autonomous driving from KTH, The Royal Institute of Technology in Stockholm, Sweden.