On July 20th, 2022, the Association for Standardization of Automation and Measuring Systems (ASAM) released its latest standard language and domain model for scenario-based testing – ASAM OpenSCENARIO® 2.0.0 (OSC2.0). OSC2.0 offers abstraction and power features to advance industry best practices. Following users’ requests, ASAM also published its updated roadmap on the converged OSC (see Figure 1 below and this document ). This information is vital for teams that are in the process of setting up their V&V toolchain and methodology for the years to come. If you are in the process of planning your short- or long-term AV or ADAS development project, you might find this blog helpful, and we encourage you to keep reading.

Short- and long-term planning of V&V projects

Have you ever tried to estimate the number of tests that should be accomplished with your Software-In-the-Loop (SIL) infrastructure to make autonomous driving systems safe*? The answer primarily depends on your ODD, system engineering requirements, the automated function SAE level, and several other factors. But how about a million tests per day?

If you think it is way too high, consider the following:

• ASIC companies (e.g. Intel, Apple, Qualcomm) are using significantly more simulations a day to validate our smartphone devices. To put things in perspective, our smartphones have about twenty times less HW, significantly reduced scenario space, and lower cost of failure. Why should we assume lower test numbers in our industry?
• Many leading automotive companies have already announced that they execute more than a million tests per day. Regardless of the number, you must execute many high-quality scenarios to be competitive in this industry.

Achieving a massive number of quality scenarios is a multi-year marathon. It requires strategized development, and proper planning is a prerequisite to the correct workflow and toolchain. Even if you initially need a smaller number of scenarios, this number multiplies, and development takes time. Developing or adopting scenarios that will remain reusable, applicable to multiple project ODDs, and valid in the future is vital. Let’s start with the implications of scale on your choice of language and why a standard language is the right decision for you.

* Foretellix has developed a calculator that allows users to get a data-driven answer for this crucial question, and we can advise you on how to set up your CICD flow accordingly.

Why Adopt a Standard Language?

A standard language is your safest path because multiple tool vendors will support your scenario investment. In the history of languages, a by-committee or de-facto standard language always emerges, and an entire eco-system appears around it. In our automotive industry, ASAM is the organization that carries the standardization torch for the benefit of all stakeholders. OSC1.x and now OSC2.0 are the highlights of ASAM’s essential deliverables.

A standard solution is good for users. Unlike proprietary languages that lock you into a specific solution, a standard language provides the freedom to select the best-in-class. Standards also fuel collaboration at the team and company level and allow sharing scenarios, requirements, and expertise with existing or future partners.

While some tool vendors may make their product sticky using proprietary languages, standards also deliver lots of value for vendors. Knowing your customers’ language facilitates building solutions that will serve significant industry segments and enable collaboration with other vendors. With that in mind, Foretellix opened its language –  Measurable Scenario Description Language (M-SDL) – and eventually offered it to ASAM as a basis for OSC2.0.

The good news is that many companies – both tool providers and  AV/ADAS developers- are already adopting a standard or searching their way out of proprietary languages.

Why Choose the OSC2.0 Standard Language?

As industries and our thinking evolve, programming languages also need to evolve. The autonomous driving industry is one of the fastest changing industries, and OSC2 capabilities and syntax were designed to enable this rapid change.

Here is a quick overview of the OSC2 technical advantages over OSC1.x:

• A modern human-readable Domain Specific Language (DSL) that is concise and intuitive for those who are domain experts but not necessarily programmers or V&V experts.
• Extra expressiveness with constraints that allows for defining abstract scenarios and enables the automation and scale needed for complex V&V tasks.
• Features for setting coverage goals and measuring the testing progress. Managing millions of scenarios is a challenging task. Coverage helps you intelligently assign compute and human resources and prove to yourself and others (in a data-driven safety case) that the SUT is safe.
• Ability to define success criteria and KPIs and capture requirements definition in an executable format. It minimizes the need for manual verdict analysis and debugging and is critical for reaching thoroughness and scale.
• Advanced reuse capabilities such as object-oriented and aspect-oriented features to define modular, composable, reusable, and extendable V&V packages. It is critical for reducing development and maintenance efforts and enabling collaboration among a distributed development team.
• Ability to define scenarios that are adjustable to any location or map.
• Extendible domain model to make the solution applicable to any original feature your company develops.
• Increased support for binding to external code/functions/methods (e.g. distribution functions, statistics)

ASAM’s recent roadmap re-enforces the previously published plan of making OSC2.x a superset of OSC1.x and converging on a single standard version based on the OSC2.0 DSL. Additionally, as part of the ASAM implementers forum, multiple vendors have already announced various levels of OSC2 support. It means that sooner or later, the existing OSC2.x will be the only viable standard option and the best one moving forward.

Personally, collaborating with many bright and passionate individuals as part of the ASAM OSC2 project, I can testify that we have tried to maintain consistency with OSC1. We also created an entire chapter about migrating OSC1 to OSC2. Nevertheless, there was no way to deliver the required capabilities on top of OSC1 in a backward compatible fashion, and while both languages are called OpenScenario and offer overlapping capabilities, these are indeed two different languages.

If you wish to learn more and further understand each item above, you can find more information here.

What methodology should I use with OSC2?

If you give a V&V person a whiteboard and ask him to draw his V&V workflow, the result will be something along the lines of figure 3.

The same figure will likely be created whether the V&V person struggles with dozens of scenarios or successfully orchestrates millions of them. Selecting a methodology that scales appropriately requires a more refined analysis of both the automation enablers and methodology parts. Coupled with the right enabling technologiesOSC2 can boost the scalability of every step of the workflow. At ASAM, we used this guiding principle of not forcing one specific methodology on OSC2. As a result, you can adopt OSC2, enjoy the readability features, but still use it with low abstraction scenarios that will inherently limit your V&V scalability. On the flip side, OSC2 enables a powerful workflow that makes it possible to scale up your project as needed. Foretellix’s OSC2-based methodology answers questions such as:

• How to build generic scenarios that are modular and reusable? When is the right time to move from concrete vs. abstract?
• How to create ODD agnostic V&V packages that can be used in all phases of your automotive project?
• How to structure the V&V plan and correlate it to system engineering requirements and other V&V risk dimensions?
• How to layer constraints effectively to steer scenario creation and not be restricted to (minor modifications)?
• How to create a tailored test suite optimized for a specific stack release or your recent stack changes?

In summary, selecting the correct methodology is a critical ingredient of any successful automotive project, and it needs to cover high-level concepts as well as practical operation guidance with careful attention to detail.

When should I move to the new standard?

Once you decide to move to OSC2, do so without delay as it prevents producing more legacy code that further complicates the migration while enabling you to enjoy the automation and robustness sooner.

As mentioned above, multiple tool vendors offer different levels of OSC2 support. Please check out our state-of-the-art solution?. Foretellix has been developing native engines for the OSC2 paradigm for the past few years, successfully applying them to projects and shaping the methodology to maximize productivity and safety. We also provide a unique value proposition. Foretellix has decided not to develop test execution platforms (i.e., we do not provide HIL or simulator products). We add an OSC2 support layer to any commercial or homegrown execution platform so that our users can leverage any simulator for its strengths and maintain the freedom to choose their test execution platforms in the future.

What’s the safest and most efficient way to move to the OSC2 standard?

The answer depends on the source scenarios’ starting condition and the migration motivation. This topic is too wide to be covered in what was initially planned to be a short blog, but here are a few high-level steps:

• Step 1: Estimate the value of existing scenarios. This step’s most challenging part is to assess your homegrown scenarios objectively. Time invested does not necessarily equal a high-value scenario, nor is it about talent – it is mostly about tools; you may be a talented runner and still lose the race competing against a couch potato driving a car. Conversely, a concrete scenario may capture a rare edge case that you wish to maintain in your test-suite execution moving forward. There are ways to apply an OSC2 coverage model to non-OSC2 scenarios and objectively assess their aggregated contribution.
• Step 2: Decide which scenarios to convert. If the motivation for moving to OSC2 was automation and robustness rather than simulator replacement (most of our users are in that category), you may decide to initially keep executing the legacy scenarios and, over time, migrate these to the standard input format.
• Step 3: Convert scenarios. There are two options here: convert line-by-line or refactor the code.
  • With line-by-line migration, you take every statement in the source language and migrate it to OSC2. If your source language is OSC1.x, the migration process may include an automatic conversion solution that can do much of the heavy lifting. The problem with this conversion is that the resulting abstraction level mirrors the abstraction of the source, meaning that much of the OSC2 potential is not utilized. Logical and concrete scenarios contain both intent and specific test implementation. For example, a concrete location may be selected by the test writer but only characterized at a high level in the requirement. The resulting scenario will be limited to the test writer’s hard-coded implementation and cannot be composed or modified in other ways.
  • Code refactoring – At this level, you analyze the source scenarios and create reusable abstract OSC2 scenarios, thus identifying the hidden intent and stripping away implementation details. We have seen hundreds of OSC1 scenarios easily refactored to a handful of OSC2 abstract scenarios that cover much less space. In some of the migrations, scenario input formats also contained tool settings. It is a bad practice that should be eliminated to keep the scenario generic.

Summary

The recent OSC2  is a considerable step forward, providing the level of abstraction and the capabilities needed to address the industry’s growing complexity. Along with the release, ASAM clarifies the standard future direction of making OSC2.x the future solution. If you wish to learn more about how these can be used today or want to explore migration alternatives, please contact us at info@foretellix.com.

Drive safe,
Sharon

Our first two OSC2 automation expert seminars were extremely successful, with hundreds of attendees and a lively exchange of information both in the Q&A sessions and on chat. We would like to thank all attendees. We hope that this was informative and valuable for you and appreciate the positive feedback that we have received so far. Our goal with these webinars is to visit key topics and jointly find solutions for the benefit of the industry.

Many users requested a transcript of the webinar Q&A segment, and this blog post is dedicated to that. Note that it is more informative than our typical blogs. It ranges from conceptual, methodology, and standard implementation questions, but you can choose to read all of it or jump to the specific answers.

Feel free to bring up other topics of interest in the comments below or send an email to info@foretellix.com. We will do our best to respond to any requests within the next couple of sessions and maybe even hold full sessions around the deeper or more interesting topics.

We have already published the date of our next webinar on the much-requested topic of requirements-based testing and how to efficiently deploy this with an automated V&V process. The webinar will take place on the 24^th of March 5pm CET. You can register to join and ask questions.

Questions:

• Is there also the intent to be used to create or validate datasets for machine learning technologies?
• How can a random scenario be generated without defined constraints
• Are the road and intersection an “actor”?
• Are there standards for reporting logs of each run to understand failure states and pre and post-actor states?
• “Lane change” is a good concept in environments that provide lanes. But what about environments with mere “free space traffic” (think of Indian roads with lots of single-track vehicles)?
• How to know the CDV does not have a major hole?  How do you know you have done enough to “verify”?
• How do you calculate the coverage metrics?
• How do you handle the issue “curse of dimensionality”. If I have 100 parameters, will the platform test all combinations?
• In your demo showing the test scores, ranks, etc., could you explain again what you are optimizing in terms of “ranks?”
• Can your tool be also used for parameter identification through its optimization capabilities? Is there a predefined workflow for that?
• Do you consider the probability of scenarios when selecting them or when evaluating the performance of the AV in the report?
• How do you define “undesirable” results such as a collision?
• Are these sequences of behaviors open loop? e.g. in the lane overtaking sequence, could you randomly generate a sequence where the actor sideswipes the AV? How would you isolate those scenarios where a collision is due to an overly aggressive actor rather than the AV’s failure? Since you’re generating a huge volume of scenarios, manual triage isn’t an option.
• If the randomly picked parameters generate a combination that is not physically possible for a vehicle at runtime, how is it handled?
• Is there an open parser/interpreter to validate a scenario?
• How should we set the scenario parameter range?
• Are the trajectories generated kinematically feasible by default? If not, is there a way to encode such constraints?
• How do you create edge case scenarios?
• Is there a reference implementation for tools implementing the format?

Is there also the intent to be used to create or validate datasets for machine learning technologies?

Before addressing the question, it is important to clarify that the OpenSCENARIO 2.0 is a standard input format, and the ability to capture scenario intent can fuel multiple tools and purposes. The standard itself neither limits nor dictates a specific purpose, flow, or tool. In regard to training, we agree with the question’s suggested direction. The steerable random flow allows users to create multiple surprising scenarios that can be used for both training and validation purposes. There is much more to be said on the training flow and the mix of surprises with real-life probabilities.

How can a random scenario be generated without defined constraints?

As the question suggests, for a lot of practical applications you need to use constraints. Logical scenarios allow random selection (aka Monte-Carlo) within ranges. These ranges define a space of scenarios to be tested but this isn’t enough for most practical purposes. Attributes such as speed, distance, time, acceleration, and even the road, are tightly correlated with implied physical constraints. For example, driving at a certain speed for a certain amount of time determines the distance traveled and requires a road of sufficient length. Constraints also provide abstraction. They allow capturing a high-level intent in a formal description. This is necessary for technology to replace the tedious cognitive work by humans that is currently the norm. For example, the tool will select a proper segment for a cut-in scenario, adjusting the scenario’s physical and intent considerations accordingly.

Are the road and intersection an “actor”?

Roads and intersections are structs composed of multiple segments. See the domain model subsection in the LRM.

Are there standards for reporting logs of each run to understand failure states and pre and post-actor states?

We agree that this is needed. In general, the more you can standardize the better, but the creation of 2.0 was a huge cooperative effort. One reason this version of OpenSCENARIO is called 2.0, is that we will eventually have more than just two versions. There was a lot of excellent work done to standardize things, yet some solid ideas just couldn’t fit into our schedule for 2.0. For instance, there are basic methods of reporting errors and warnings as well as more sophisticated controls that were postponed to a future release. Foretellix has standards that we will be happy to contribute to subsequent versions, but we’ll see how that pans out.

“Lane change” is a good concept in environments that provide lanes. But what about environments with mere “free space traffic” (think of Indian roads with lots of single-track vehicles)?

The OSC2.0 domain model includes many actions and modifiers in addition to lanes, such as movable objects that can move in free space, and a car that can drive along paths. See the LRM for more details.

How to know the CDV does not have a major hole?  How do you know you have done enough to “verify”?

There is no single solution that ensures 100% full coverage of the entire scenario space. Coverage Driven Verification allows you to simply and effectively define and achieve goals. The use of automation enables scale and measurability and reduces room for human errors. As such, it is more effective than the manual approach which often suffers from major holes.  At the same time, there are important points to make about CDV and the possibility of missing holes:

• The use of random generation may expose unknown, overlooked conditions. The combination of this and generic KPIs identifies holes in your original plan.
• The abstraction and ability to create reusable V&V packages allow aggregating expertise and interesting scenarios. Over time, your reusable scenarios become a thorough asset to rely on.
• The coverage goals can be automatically or manually extracted from multiple complementary resources and execution platforms. Optimizers can take these as starting points and steer the automatic test creation to visit risky areas.

CDV enables an automated flow full of checks and balances to identify coverage holes or missing requirements.

How do you calculate the coverage metrics?

OSC2.0 coverage features facilitate organizing infinity with a gradable coverage model. As a basis, each coverage item is calculated according to its collected type expressions:

• For Boolean expressions (for example, was the siren on) – both true and false need to be observed for 100% coverage, so observing just one of the values is 50% coverage (one out of two). No observed value means 0% coverage.
• For enumerated types (for example, was the vehicle category truck, sedan, or van), the number of observed values divided by the number of possible values is the verification grade. For example, if an enum type has three possible values and two of these were observed in a test suite, the verification grade is 2/3, or 66 percent.
• For continuous values such as speed, which is an unsigned integer, the user can create buckets of values and count the hits in each bucket. For example, a speed of 0 to 120kph can be split into four buckets: 0..30, 31..60, 61..90, 91..120. By default, one hit in a bucket is sufficient to declare the bucket to be covered.
• The overall average of all the calculated item grades is the overall coverage grade.

OSC2 lets you further shape the grades by ignoring values that are not interesting (for example, I do not care about trucks in my project), setting a different target for some values (for example, I want to try at least 10 cut-ins from the right and 5 are enough from the left), and more. Users may build different dashboards and analytics tools around these language concepts.

How do you handle the issue “curse of dimensionality”. If I have 100 parameters, will the platform test all combinations?

This is the core issue of taming infinity. Scenarios have parameters, each with a certain range of values. Also, because you can always mix scenarios, each with multiple parameters, the possibilities will never end, right? Even thinking about that makes your head spin. And one of the things we try and show — and I think you got at least a first glimpse of it — is that there are ways to use coverage definitions and related technology to make this a kind of optimization problem in the following sense: if you just have one night and, say, a hundred machines, what is the best way to use those resources to collect a first pass coverage?

If you have a weekend and a thousand machines, then here is your next step, but you’re going to cover all your scenarios, breadth-first at least to some degree. Then if you have more time, you can collect more and more coverage. And obviously, you will never do all the combinations. In fact, even with just a single speed parameter, you will never get to all the combinations. So technology and methodology are required to make a practical solution.

So as I think the writer of this question already implied, when you define your coverage, you should be thinking through what are the dangerous things and what are the various interactions. In a sense, a coverage model should be a map of your fears. And so if you are assuming that in this ODD there will be multiple car types, and there will be trucks, busses, and so on, then you had better define a coverage model that goes through them all. And if you think there will be snow and rain as well as nice days and fog, you had better go through all of these and then make sure that you cross them all and combine them all. And if you don’t have the assets, or if there’s a specific simulator you’re connected to that does not support snow, for instance, then you should do something about it, such as using a better simulator.

In your demo showing the test scores, ranks, etc., could you explain again what you are optimizing in terms of “ranks?”

Ranking is one of the optimization capabilities that Foretellix built on top of the OSC2 standard input format. Specifically, ranking optimization helps you create an optimized test suite for the entire V&V effort or for selected V&V needs.

The fact is that executed scenarios cover their own specified intent but might also cover much more in a by-the-way manner. For example, while doing a cut-in in a random location I may also drive under a bridge. Ranking can take a test suite with 500 tests and identify 50 tests that produce the same coverage result.  In subsequent test regressions, I may run 50 scenarios that have minimal redundancy, shorten the test suite execution time, and save the expensive simulation cycles for other goals.

Can your tool be also used for parameter identification through its optimization capabilities? Is there a predefined workflow for that?

The answer is yes. Please note that these webinars are really about the approach.  CDV efficiently spans the desired space and identifies unknown scenarios that can later be sent to a KPI optimizer. Specifically, Foretellix technology uses machine learning algorithms to identify feature importance, which can help both debug and efficient scenario exploration.

Do you consider the probability of scenarios when selecting them or when evaluating the performance of the AV in the report?

Yes. By default, unified distribution is applied for value selections – this means that all legal values have the same unbiased chance of being selected. You can use OSC2 default constraints to capture the desired probabilities for value selection. As the name suggests, default constraints apply default values. If a test constraint further steers the needed distribution (for example, for an edge case scenario), these constraints are ignored. Note that in edge case scenarios, even though one behavior was pushed to the extreme, it is important that other behaviors are normal to avoid a scenario that is just too crazy. Note that in some cases where statistical reports are needed, a joint distribution needs to be applied.

How do you define “undesirable” results such as a collision?

OSC2 provides success criteria mechanisms to enable self-checking tests. Such mechanisms are critical to enabling scale. Two main checker categories are:

• Generic checkers – Applied for every execution regardless of the executed scenarios (for example, don’t collide with other actors)
• Scenario specific checks – Applied for a specific narrow scenario context (for example, in the cut-in scenario end of change lane phase, allow a maximum of 1s TTC).

Are these sequences of behaviors open loop? e.g. in the lane overtaking sequence, could you randomly generate a sequence where the actor sideswipes the AV? How would you isolate those scenarios where a collision is due to an overly aggressive actor rather than the AV’s failure? Since you’re generating a huge volume of scenarios, manual triage isn’t an option.

This is exactly true. When you go from the thousands of runs per night to the hundred thousand runs per night, you can no longer think about manual checking of things. Not every accident and not every collision is necessarily a SUT error. If you create a challenging scenario where, for example, 50% of the instances of that scenario are supposed to end in a collision, then such collisions are not obviously all errors of the SUT. You need a pretty sophisticated way to define what is clearly an error,  what is clearly okay, and what are the gray areas that you may want to check later? And it is a combination of technology, methodology, and even art.

You also need a standard format for errors, and you need tools that help you cluster the errors, help you analyze them. We didn’t get to show that, but I think that’s an interesting thing to see.

If the randomly picked parameters generate a combination that is not physically possible for a vehicle at runtime, how is it handled?

The constraints include both scenario constraints, such as vehicle category, and implied physical constraints, such as those that connect distance to speed, to acceleration, and so on. An OSC2 tool should adjust the scenario to meet both your scenario and implied physical constraints, or it should report an error if no such scenario can be achieved. Many times, the constraint solver may report constraint contradictions even before simulation starts. In the case of partially controlled actors (for example, smart actors driven by behavioral models) a runtime contradiction may be reported. As was said previously, Foretellix tools load the physical constraints by default, so users can focus on the abstract scenario constraints.

Is there an open parser/interpreter to validate a scenario?

One of the advantages of a standard is that the multiple vendors involved will push the industry forward. PMSF suggests a parser and CLI for OSC2.0 syntax checking. For more information, please click here.

How should we set the scenario parameter range?

Scenario parameter ranges are just another type of constraint. As was discussed, abstract scenarios can include any constraint kind to capture dependencies. A parameter range can be between any two expressions: not just a literally specified range such as keep(speed in [10.100]KPH)  but also keep(speed in [attr_a..attr_b]). Note also that the final calculated values will be a resolution of the range and the context scenario. For example, if a truck is used as part of the cut-in scenario, the eventually randomized vehicle speeds will conform to both the range requirement and the truck speed capabilities.

Are the trajectories generated kinematically feasible by default? If not, is there a way to encode such constraints?

By default, generated scenarios and dynamics are both feasible and physically possible. Note that virtual platforms may benefit from running scenarios that are not physically possible. (For example, you can save simulation time by allowing infinite acceleration.) The OSC2 LRM and domain model allow executing actions that are not physically possible, as well as those that are. There are methodology guidelines on how to write reusable scenarios that are portable between virtual and physical platforms.

How do you create edge-case scenarios?

The definition of edge-case scenarios can be wide. It could be non-typical parameter values, the desired traffic density, error injections, road conditions, a specific mix of scenarios, and more. There are efficient ways to come up with the right set of edge-cases in the planning process. Once the plan is set, OSC2 facilitates translating the functional goals into abstract scenarios and a coverage model to efficiently meet these goals.

Is there a reference implementation for tools implementing the format?

Not at this point. In terms of effort, it would be extremely challenging to create a reference implementation. Also, past experience has shown that a reference implementation prevents industry progress by forcing a specific implementation scheme on vendors so that, for example, constraint solvers did not progress.

As usual, drive safe,
Sharon