Over the years, Foretellix has collaborated with many ADAS and AV developers to explore their requirements for a ‘dream solution’. In this post, I’ll summarize their answers and offer a solution to the V&V challenges.
As I’m writing this blog, AAA recommends limiting the use of partially automated driving systems (you can read more here, here and also here). Such safety concerns damage public trust and reduce the perceived value of automation functions. And OEMs are, indeed, devoting large (and growing) efforts to the V&V of these functions.
We at Foretellix, always try to improve the robustness of active safety and automated driving systems by evaluating, challenging, and improving existing V&V tools and workflows.
Users describe their ‘dream solution’
Over the past couple of years, we collaborated with multiple customers and partners, asking them to describe their ‘ideal solution’. Below are the most commonly received answers about this tool – let’s call it ‘Solution-x’. (BTW, this is quite a list. If you think that you know the requirements, feel free to jump to the Foretellix offered solution)
Produce a massive number of tests effortlessly and out-of-the-box
While ADAS and highway are simpler than say level 4 AV, an infinite number of things can go wrong. Solution-x should generate an unlimited number (as much as I want) of variations of each scenario-based test and uncover edge cases, unknowns (as per ISO 21448 SOTIF) and efficiently find many bugs
The result tests should be of high quality and diversified; unique in locations, trajectories, circumstances, and participants that are beyond attribute variations on a hardcoded scenario.
My compute is a valuable resource and my engineering resources even more. I want to make sure that the generated scenarios and tests are both feasible and meaningful with minimal redundancy to ensure efficient execution, and to ensure that my team does not spend time debugging invalid scenarios.
Efficiently meet test intent
Solution-x should plan the required scenarios upfront, but also adjust the plan in real-time, according to unpredictable decisions of my SW stack.
Provide goals and report completeness
With an infinite scenario space, I need a measurable process to know when I have fulfilled my engineering task
Solution-x should provide an extensive verification plan combining industry knowledge of ADAS and Highway, past collisions, and standardization requirements
The provided plan should be easily extendible to include project-specific goals
If an execution fails to meet the test intent, solution-x should report the failure and suggest a different way to achieve the intent
Throughout the V&V process, solution-x should clearly indicate the current status, allowing users to focus on unverified areas and eliminate redundant tests
Combine out-of-the-box and user-defined tests
Beyond delivering the generic known prone to error ADAS edge cases, solution-x should make it easy for domain experts to leverage their knowledge and create multiple tests with low effort and minimal learning curve.
Be simulator and testing platform agnostic
Our company uses multiple simulators for various needs and may change or adopt another simulator in the future. Solution-x should be portable across all simulators and testing platforms we use (e.g. HIL and test-track)
Use a standard language with an industry commitment and roadmap
Standards foster a choice of tools and attract talented users searching for career growth.
Support regulation needs and future automation levels
For example, the new UNECE SAE level three ALKS regulation that lists specific needs
Current investment and methodology should be useful moving forward to higher SAE automation levels
The Foretellix Solution
While the Foretellix technology already solves many of these challenges, the message from users is to also provide an out-of-the-box solution on top of our powerful “do-it-yourself” tool. Working on specific ODD or automation functions they wish to get assistance with, the major questions of what to test? How to test? And when am I done? Considering all the above now is the perfect time to lift the curtain and introduce our solution-x: the Foretellix ADAS and Highway Solution.
The ADAS and Highway solution is the first of a family of verification packages that utilize the power of the Foretify platform and extend our multipurpose platform to address the unique challenges of different functions and ODDs. The ADAS and highway content is based on the conducted research of customers careabouts, past collisions, technology limitations, upcoming regulations, safety standards, and disengagement reports to deliver an effective solution for exposing bugs and promote safety.
Our ADAS and Highway solution was designed for ADAS and Highway V&V tasks, including:
A verification plan representing many scenario categories, variants, and overall coverage space that must be completed as part of the V&V process.
Extensive ready-made regression that activates and mixes 36 abstract M-SDL maneuver scenarios to generate 100,000s of meaningful concrete tests, stationary objects in random location and angles, and other risk dimensions.
Self-checking mechanism with pass-fail criteria
An intuitive dashboard displaying the executed and, as important, the non-executed scenarios
Simplified spreadsheet-based tabular interface to enhance and refine the tests to hit non-verified concerns and make sure we generate meaningful and feasible scenarios
The solution requires a simple initial integration step after which, both expert and naïve users can exercise the solution to achieve a thorough exploration of their devices.
The ADAS and Highway Solution pre-provided test regression is captured in a readable table format, that can be easily extended to meet any user needs. Users fill out a spreadsheet indicating the tests to run while either specifying concrete values for test attributes or asking the solver to complete the missing attributes with random values that adhere to abstract constraint requirements. Each table raw is equivalent to coding an M-SDL test.
To help users analyze regression results, Foretellix offers Foretify Manager, a highly configurable tool that quickly analyzes DUT errors as well as behaviors of the DUT that have not been well tested.
In addition to the pre-provided scenarios, the solution enables a powerful approach to explore unknowns and edge-cases using sub-scenario mixing. The ADAS and Highway solution leverages the Foretify solver to allow users to mix the pre-provided content to achieve new unexpected variations. To better understand the implication of this, please review one of the pre-provided mixes, the double-cut-in mix.
The purple area above is the first cut-in and the orange one constitutes the second. The Fortify solver can select a location that matches the two cut-ins needs and inserts implicit actions to ensure the proper vehicle coordination that is needed for the larger double cut-in scenario.
Another key benefit of the ADAS and Highway solution is the use of the open language M-SDL (the basis for OpenSCENARIO2.0 concept project and which is now in the process of becoming a standard). The language enables flexible test customization, extra tuning of the success and failure criteria, and powerful refinements of the built-in coverage model. Adopting and learning the M-SDL triggered interest for multiple individuals and teams that search for the next level of automation.
Note that I did not discuss user regulation needs. This is an exciting part of the ADAS and Highway solution which deserve its own dedicated blog 😊
As reported above, the industry still struggles with existing ADAS automation functions, which continue to grow in complexity. The ADAS and Highway Solution offers an extensive out-of-the-box solution tuned for finding bugs and promoting safety.
To learn more about the solution please follow this link, and, as usual, comments are welcome either below or via email
On July 20th, 2022, the Association for Standardization of Automation and Measuring Systems (ASAM) released its latest standard language and domain model for scenario-based testing – ASAM OpenSCENARIO® 2.0.0 (OSC2.0). OSC2.0 offers abstraction and power features to advance industry best practices.
The Foretellix V-Suites™ are the most comprehensive V&V packages available today, aggregating extensive industry knowledge gained from working with multiple users, regulation requirements, and from collision databases.
Earlier this month, I joined the Board of Foretellix, a company whose mission is to make ADAS and autonomous systems safe. It is no small undertaking, but I am convinced Foretellix is very well positioned to achieve this goal. I wanted to share some of my thoughts and tell you why I decided to join Foretellix on this journey.