Given the challenges of verifying and validating (V&V) ADAS and automated highway functions, how would you describe the ideal toolset and methodology?
Over the years, Foretellix has collaborated with many ADAS and AV developers to explore their requirements for a ‘dream solution’. In this post, I’ll summarize their answers and offer a solution to the V&V challenges.
As I’m writing this blog, AAA recommends limiting the use of partially automated driving systems (you can read more here, here and also here). Such safety concerns damage public trust and reduce the perceived value of automation functions. And OEMs are, indeed, devoting large (and growing) efforts to the V&V of these functions.
We at Foretellix, always try to improve the robustness of active safety and automated driving systems by evaluating, challenging, and improving existing V&V tools and workflows.
Users describe their ‘dream solution’
Over the past couple of years, we collaborated with multiple customers and partners, asking them to describe their ‘ideal solution’. Below are the most commonly received answers about this tool – let’s call it ‘Solution-x’. (BTW, this is quite a list. If you think that you know the requirements, feel free to jump to the Foretellix offered solution)
Produce a massive number of tests effortlessly and out-of-the-box
- While ADAS and highway are simpler than say level 4 AV, an infinite number of things can go wrong. Solution-x should generate an unlimited number (as much as I want) of variations of each scenario-based test and uncover edge cases, unknowns (as per ISO 21448 SOTIF) and efficiently find many bugs
- The result tests should be of high quality and diversified; unique in locations, trajectories, circumstances, and participants that are beyond attribute variations on a hardcoded scenario.
- My compute is a valuable resource and my engineering resources even more. I want to make sure that the generated scenarios and tests are both feasible and meaningful with minimal redundancy to ensure efficient execution, and to ensure that my team does not spend time debugging invalid scenarios.
Efficiently meet test intent
- Solution-x should plan the required scenarios upfront, but also adjust the plan in real-time, according to unpredictable decisions of my SW stack.
Provide goals and report completeness
- With an infinite scenario space, I need a measurable process to know when I have fulfilled my engineering task
- Solution-x should provide an extensive verification plan combining industry knowledge of ADAS and Highway, past collisions, and standardization requirements
- The provided plan should be easily extendible to include project-specific goals
- If an execution fails to meet the test intent, solution-x should report the failure and suggest a different way to achieve the intent
- Throughout the V&V process, solution-x should clearly indicate the current status, allowing users to focus on unverified areas and eliminate redundant tests
Combine out-of-the-box and user-defined tests
- Beyond delivering the generic known prone to error ADAS edge cases, solution-x should make it easy for domain experts to leverage their knowledge and create multiple tests with low effort and minimal learning curve.
Be simulator and testing platform agnostic
- Our company uses multiple simulators for various needs and may change or adopt another simulator in the future. Solution-x should be portable across all simulators and testing platforms we use (e.g. HIL and test-track)
Use a standard language with an industry commitment and roadmap
- Standards foster a choice of tools and attract talented users searching for career growth.
Support regulation needs and future automation levels
- For example, the new UNECE SAE level three ALKS regulation that lists specific needs
- Current investment and methodology should be useful moving forward to higher SAE automation levels
The Foretellix Solution
While the Foretellix technology already solves many of these challenges, the message from users is to also provide an out-of-the-box solution on top of our powerful “do-it-yourself” tool. Working on specific ODD or automation functions they wish to get assistance with, the major questions of what to test? How to test? And when am I done? Considering all the above now is the perfect time to lift the curtain and introduce our solution-x: the Foretellix ADAS and Highway Solution.
The ADAS and Highway solution is the first of a family of verification packages that utilize the power of the Foretify platform and extend our multipurpose platform to address the unique challenges of different functions and ODDs. The ADAS and highway content is based on the conducted research of customers careabouts, past collisions, technology limitations, upcoming regulations, safety standards, and disengagement reports to deliver an effective solution for exposing bugs and promote safety.
Our ADAS and Highway solution was designed for ADAS and Highway V&V tasks, including:
- A verification plan representing many scenario categories, variants, and overall coverage space that must be completed as part of the V&V process.
- Extensive ready-made regression that activates and mixes 36 abstract M-SDL maneuver scenarios to generate 100,000s of meaningful concrete tests, stationary objects in random location and angles, and other risk dimensions.
- Self-checking mechanism with pass-fail criteria
- An intuitive dashboard displaying the executed and, as important, the non-executed scenarios
- Simplified spreadsheet-based tabular interface to enhance and refine the tests to hit non-verified concerns and make sure we generate meaningful and feasible scenarios
The solution requires a simple initial integration step after which, both expert and naïve users can exercise the solution to achieve a thorough exploration of their devices.
The ADAS and Highway Solution pre-provided test regression is captured in a readable table format, that can be easily extended to meet any user needs. Users fill out a spreadsheet indicating the tests to run while either specifying concrete values for test attributes or asking the solver to complete the missing attributes with random values that adhere to abstract constraint requirements. Each table raw is equivalent to coding an M-SDL test.
To help users analyze regression results, Foretellix offers Foretify Manager, a highly configurable tool that quickly analyzes DUT errors as well as behaviors of the DUT that have not been well tested.
In addition to the pre-provided scenarios, the solution enables a powerful approach to explore unknowns and edge-cases using sub-scenario mixing. The ADAS and Highway solution leverages the Foretify solver to allow users to mix the pre-provided content to achieve new unexpected variations. To better understand the implication of this, please review one of the pre-provided mixes, the double-cut-in mix.
The purple area above is the first cut-in and the orange one constitutes the second. The Fortify solver can select a location that matches the two cut-ins needs and inserts implicit actions to ensure the proper vehicle coordination that is needed for the larger double cut-in scenario.
Another key benefit of the ADAS and Highway solution is the use of the open language M-SDL (the basis for OpenSCENARIO2.0 concept project and which is now in the process of becoming a standard). The language enables flexible test customization, extra tuning of the success and failure criteria, and powerful refinements of the built-in coverage model. Adopting and learning the M-SDL triggered interest for multiple individuals and teams that search for the next level of automation.
Note that I did not discuss user regulation needs. This is an exciting part of the ADAS and Highway solution which deserve its own dedicated blog ?
Conclusion
As reported above, the industry still struggles with existing ADAS automation functions, which continue to grow in complexity. The ADAS and Highway Solution offers an extensive out-of-the-box solution tuned for finding bugs and promoting safety.
To learn more about the solution please follow this link, and, as usual, comments are welcome either below or via email
Travel safely,
Sharon
