On July 20th, 2022, the Association for Standardization of Automation and Measuring Systems (ASAM) released its latest standard language and domain model for scenario-based testing – ASAM OpenSCENARIO® 2.0.0 (OSC2.0). OSC2.0 offers abstraction and power features to advance industry best practices. Following users’ requests, ASAM also published its updated roadmap on the converged OSC (see Figure 1 below and this document ). This information is vital for teams that are in the process of setting up their V&V toolchain and methodology for the years to come. If you are in the process of planning your short- or long-term AV or ADAS development project, you might find this blog helpful, and we encourage you to keep reading.
Short- and long-term planning of V&V projects
Have you ever tried to estimate the number of tests that should be accomplished with your Software-In-the-Loop (SIL) infrastructure to make autonomous driving systems safe*? The answer primarily depends on your ODD, system engineering requirements, the automated function SAE level, and several other factors. But how about a million tests per day?
If you think it is way too high, consider the following:
- ASIC companies (e.g. Intel, Apple, Qualcomm) are using significantly more simulations a day to validate our smartphone devices. To put things in perspective, our smartphones have about twenty times less HW, significantly reduced scenario space, and lower cost of failure. Why should we assume lower test numbers in our industry?
- Many leading automotive companies have already announced that they execute more than a million tests per day. Regardless of the number, you must execute many high-quality scenarios to be competitive in this industry.
Achieving a massive number of quality scenarios is a multi-year marathon. It requires strategized development, and proper planning is a prerequisite to the correct workflow and toolchain. Even if you initially need a smaller number of scenarios, this number multiplies, and development takes time. Developing or adopting scenarios that will remain reusable, applicable to multiple project ODDs, and valid in the future is vital. Let’s start with the implications of scale on your choice of language and why a standard language is the right decision for you.
* Foretellix has developed a calculator that allows users to get a data-driven answer for this crucial question, and we can advise you on how to set up your CICD flow accordingly.
Why Adopt a Standard Language?
A standard language is your safest path because multiple tool vendors will support your scenario investment. In the history of languages, a by-committee or de-facto standard language always emerges, and an entire eco-system appears around it. In our automotive industry, ASAM is the organization that carries the standardization torch for the benefit of all stakeholders. OSC1.x and now OSC2.0 are the highlights of ASAM’s essential deliverables.
A standard solution is good for users. Unlike proprietary languages that lock you into a specific solution, a standard language provides the freedom to select the best-in-class. Standards also fuel collaboration at the team and company level and allow sharing scenarios, requirements, and expertise with existing or future partners.
While some tool vendors may make their product sticky using proprietary languages, standards also deliver lots of value for vendors. Knowing your customers’ language facilitates building solutions that will serve significant industry segments and enable collaboration with other vendors. With that in mind, Foretellix opened its language – Measurable Scenario Description Language (M-SDL) – and eventually offered it to ASAM as a basis for OSC2.0.
The good news is that many companies – both tool providers and AV/ADAS developers- are already adopting a standard or searching their way out of proprietary languages.
Why Choose the OSC2.0 Standard Language?
As industries and our thinking evolve, programming languages also need to evolve. The autonomous driving industry is one of the fastest changing industries, and OSC2 capabilities and syntax were designed to enable this rapid change.
Here is a quick overview of the OSC2 technical advantages over OSC1.x:
- A modern human-readable Domain Specific Language (DSL) that is concise and intuitive for those who are domain experts but not necessarily programmers or V&V experts.
- Extra expressiveness with constraints that allows for defining abstract scenarios and enables the automation and scale needed for complex V&V tasks.
- Features for setting coverage goals and measuring the testing progress. Managing millions of scenarios is a challenging task. Coverage helps you intelligently assign compute and human resources and prove to yourself and others (in a data-driven safety case) that the SUT is safe.
- Ability to define success criteria and KPIs and capture requirements definition in an executable format. It minimizes the need for manual verdict analysis and debugging and is critical for reaching thoroughness and scale.
- Advanced reuse capabilities such as object-oriented and aspect-oriented features to define modular, composable, reusable, and extendable V&V packages. It is critical for reducing development and maintenance efforts and enabling collaboration among a distributed development team.
- Ability to define scenarios that are adjustable to any location or map.
- Extendible domain model to make the solution applicable to any original feature your company develops.
- Increased support for binding to external code/functions/methods (e.g. distribution functions, statistics)
ASAM’s recent roadmap re-enforces the previously published plan of making OSC2.x a superset of OSC1.x and converging on a single standard version based on the OSC2.0 DSL. Additionally, as part of the ASAM implementers forum, multiple vendors have already announced various levels of OSC2 support. It means that sooner or later, the existing OSC2.x will be the only viable standard option and the best one moving forward.
Personally, collaborating with many bright and passionate individuals as part of the ASAM OSC2 project, I can testify that we have tried to maintain consistency with OSC1. We also created an entire chapter about migrating OSC1 to OSC2. Nevertheless, there was no way to deliver the required capabilities on top of OSC1 in a backward compatible fashion, and while both languages are called OpenScenario and offer overlapping capabilities, these are indeed two different languages.
If you wish to learn more and further understand each item above, you can find more information here.
What methodology should I use with OSC2?
If you give a V&V person a whiteboard and ask him to draw his V&V workflow, the result will be something along the lines of figure 3.
The same figure will likely be created whether the V&V person struggles with dozens of scenarios or successfully orchestrates millions of them. Selecting a methodology that scales appropriately requires a more refined analysis of both the automation enablers and methodology parts. Coupled with the right enabling technologiesOSC2 can boost the scalability of every step of the workflow. At ASAM, we used this guiding principle of not forcing one specific methodology on OSC2. As a result, you can adopt OSC2, enjoy the readability features, but still use it with low abstraction scenarios that will inherently limit your V&V scalability. On the flip side, OSC2 enables a powerful workflow that makes it possible to scale up your project as needed. Foretellix’s OSC2-based methodology answers questions such as:
- How to build generic scenarios that are modular and reusable? When is the right time to move from concrete vs. abstract?
- How to create ODD agnostic V&V packages that can be used in all phases of your automotive project?
- How to structure the V&V plan and correlate it to system engineering requirements and other V&V risk dimensions?
- How to layer constraints effectively to steer scenario creation and not be restricted to (minor modifications)?
- How to create a tailored test suite optimized for a specific stack release or your recent stack changes?
In summary, selecting the correct methodology is a critical ingredient of any successful automotive project, and it needs to cover high-level concepts as well as practical operation guidance with careful attention to detail.
When should I move to the new standard?
Once you decide to move to OSC2, do so without delay as it prevents producing more legacy code that further complicates the migration while enabling you to enjoy the automation and robustness sooner.
As mentioned above, multiple tool vendors offer different levels of OSC2 support. Please check out our state-of-the-art solution?. Foretellix has been developing native engines for the OSC2 paradigm for the past few years, successfully applying them to projects and shaping the methodology to maximize productivity and safety. We also provide a unique value proposition. Foretellix has decided not to develop test execution platforms (i.e., we do not provide HIL or simulator products). We add an OSC2 support layer to any commercial or homegrown execution platform so that our users can leverage any simulator for its strengths and maintain the freedom to choose their test execution platforms in the future.
What’s the safest and most efficient way to move to the OSC2 standard?
The answer depends on the source scenarios’ starting condition and the migration motivation. This topic is too wide to be covered in what was initially planned to be a short blog, but here are a few high-level steps:
- Step 1: Estimate the value of existing scenarios. This step’s most challenging part is to assess your homegrown scenarios objectively. Time invested does not necessarily equal a high-value scenario, nor is it about talent – it is mostly about tools; you may be a talented runner and still lose the race competing against a couch potato driving a car. Conversely, a concrete scenario may capture a rare edge case that you wish to maintain in your test-suite execution moving forward. There are ways to apply an OSC2 coverage model to non-OSC2 scenarios and objectively assess their aggregated contribution.
- Step 2: Decide which scenarios to convert. If the motivation for moving to OSC2 was automation and robustness rather than simulator replacement (most of our users are in that category), you may decide to initially keep executing the legacy scenarios and, over time, migrate these to the standard input format.
- Step 3: Convert scenarios. There are two options here: convert line-by-line or refactor the code.
- With line-by-line migration, you take every statement in the source language and migrate it to OSC2. If your source language is OSC1.x, the migration process may include an automatic conversion solution that can do much of the heavy lifting. The problem with this conversion is that the resulting abstraction level mirrors the abstraction of the source, meaning that much of the OSC2 potential is not utilized. Logical and concrete scenarios contain both intent and specific test implementation. For example, a concrete location may be selected by the test writer but only characterized at a high level in the requirement. The resulting scenario will be limited to the test writer’s hard-coded implementation and cannot be composed or modified in other ways.
- Code refactoring – At this level, you analyze the source scenarios and create reusable abstract OSC2 scenarios, thus identifying the hidden intent and stripping away implementation details. We have seen hundreds of OSC1 scenarios easily refactored to a handful of OSC2 abstract scenarios that cover much less space. In some of the migrations, scenario input formats also contained tool settings. It is a bad practice that should be eliminated to keep the scenario generic.
Summary
The recent OSC2 is a considerable step forward, providing the level of abstraction and the capabilities needed to address the industry’s growing complexity. Along with the release, ASAM clarifies the standard future direction of making OSC2.x the future solution. If you wish to learn more about how these can be used today or want to explore migration alternatives, please contact us at info@www.foretellix.com.
Drive safe,
Sharon
