‘A method of solution is perfect if we can foresee from the start, and even prove, that following that method we shall attain our aim.’ — Gottfried Wilhelm Leibniz
Introduction
Automated Driving Systems (ADS) force a paradigm shift in safety. ADS technology is expected to assume total ownership of driving safety despite the elements of surprise, chaos, and unpredictability inherent in the driving environment. This imposes unprecedented safety expectations on engineered artifacts and the organizations that create them. Fulfilling these expectations requires substantial innovation in Safety Verification and Validation (V&V), especially for ADS functionality that understands the driving environment, predicts how it may evolve, and plans safe maneuvers. This blog post describes critical elements of Foretellix’s innovative Safety-Driven V&V (SDV) approach and how SDV provides the paradigm shift for ensuring the safety of ADS and ADAS at scale.
SDV: The methodology
The SDV methodology is established to ensure that the Safety V&V effort contributes to and enables all the safety assurance needs for ADS and ADAS. It follows the current safety standards and upcoming regulations and can be tailored to the needs of specific companies, engineering cultures, and validation philosophies. The application of the methodology forces deep thinking about the safety claims to be made, supporting evidence, testing strategies and modalities, evaluating residual risk, etc. This leads to a comprehensive and cohesive safety validation program that provides a clear and actionable guide to crossing the safety finish line. In particular, SDV includes support for the following:
- Safety case: The SDV tools and methodology generate artifacts that serve as evidence for safety case claims related to test results, coverage, etc. With the proper tooling and processes, the continuous and iterative generation of evidence artifacts serves as a basis for the so-called “continuous safety case” approach, in which safety considerations are defined and tracked continuously throughout the lifecycle.
- Residual risk estimation: All approaches to residual risk, regardless of specifics, need to account for whether sufficient testing has occurred, whether the results are consistent with the organization’s views on risk acceptability, and the likelihood of undiscovered safety issues. In turn, Foretellix provides the foundational tools for supporting this and is working with leading customers on developing more abstract methodological guidance on how test spaces could be structured and discretized.
- The evolution of Safety/V&V programs across the development lifecycle: The Safety and V&V activities in the early stages of feature development are very different compared to those done during pre-launch validation of a feature-complete ADS/ADAS stack. SDV has the depth and breadth to “tune the focus and intensity” of V&V to immediately obtain valuable results for the development stage. As the system matures, the same tools and processes can ramp up and build off the previous results for wider and/or more focused V&V explorations.
- Bringing together the teams engaged in core autonomy development, V&V, and Safety: SDV methodology requires the teams to come together and define which requirements and goals must be explicit, how they should be articulated and tested, how the results are evaluated, etc. The SDV tools generate artifacts (e.g., test results, coverage reports) that need to be jointly viewed and discussed by the teams, typically as part of explicitly defined processes for this purpose.
- Post-deployment monitoring: Harvesting field data from deployed fleets and using that to continuously increase confidence in the Safety Case, validate safety assumptions, and proactively identify any needed changes (and subsequently validate them) are all activities that are an integral part of SDV. Upcoming regulations will also require this type of post-deployment analysis and strengthening of the safety validation.
- ODD expansion: SDV also increases the efficiency of V&V when expanding to new ODDs… be that about expanding geo-fences or expanding to more operational and weather situations within the same geo-fence.
- Standards and regulations: SDV supports relevant parts of all modern ADS/ADAS safety standards and regulatory requirements, including but not limited to ISO 26262, ISO 21448, ISO 34502, UL 4600, IEEE 2846, and UNECE regulation 157. The workflow supported by the SDV methodology is aligned with the regulatory direction developed by UNECE.
SDV: The technology
SDV’s key technical offerings are:
- The ASAM OpenSCENARIO® 2.0 language to elegantly describe abstract scenarios in a concise, declarative, arbitrarily combinable, reusable, and formal-yet-intuitive manner. Scenario descriptions include scenario parameters, their ranges and distributions, constraints, and other metrics (KPIs/SPIs, coverage, etc.) to be computed and checked as the scenario executes. These abstract descriptions are reusable across maps and various ODDs.
- Constrained-random generation of a large number of valid concrete scenarios from the abstract scenario descriptions. The generator doesn’t just randomly choose scenario parameter values within their ranges. (Doing that generates many silly/impossible scenarios requiring subsequent pruning.) Instead, it understands the semantics of the driving domain and picks scenario parameter values consistent with each other and the laws of physics. Appropriate map locations are also automatically selected. For example, if an actor is behind the Ego vehicle at the start of the scenario and ahead of it at the end of the scenario, the generator automatically infers that a map with a minimum of two lanes would be needed and lane changes, overtaking, and a subsequent cut-in by actor ahead of Ego would all be involved.
- Seamless use of real-world driving logs in Safety validation. The tools convert driving logs into a timeline of scenarios and automatically compute scenario parameters, their distributions, and scenario metrics. This enables checking AV performance, detecting anomalous behavior, assessing simulation accuracy, etc. This also allows assessments of test coverage across real-world and virtual testing.
- Sophisticated analysis tools expressly designed to enable efficient examination and arbitrary exploration of millions of scenario simulations and real-world driving results. From debug exploration of single scenarios to triage of large batches of simulation runs to statistical queries across multiple runs.
- Test suite management and optimization to guide iterative testing towards specific test objectives. Whether you want to run the most common scenarios, corner cases, or find situations where particular KPIs/SPIs are minimized, or increase coverage over input values, or efficiently seek out parameter combinations that cause failures, the test suite managers guide the selection of the next set of test cases by analyzing results of previous test runs.
Conclusion
At some point, OEMs will need to bite the bullet, put down the metaphorical pen, and say, “This is it! We are ready for launch.” Getting to that point requires a blueprint that shows how novel safety technologies and methodologies can collectively get you to the finish. SDV is exciting because it lights a path to that promised land. It adds the missing ingredients and can carry Automated Driving Systems across the safety finish line. Leibniz would approve!
To learn more, download the Safety-Driven V&V Guide
