Below is our latest incident report examining the Cruise incident in Austin, Texas, from a few weeks ago, where a driverless Cruise vehicle turned left into a bike lane. Fortunately, no one was hurt, but it raises questions about the safety of autonomous vehicles in edge cases. Matt Fournier, one of our V&V experts, discusses the importance of testing autonomous vehicles under a wide range of conditions to ensure their safety. Using Foretellix’s Foretify™ platform, he created an abstract scenario that generates over 10,000 concrete variations of the incident. Read more to learn about the challenges autonomous vehicles face and how Foretellix’s Foretify™ platform can help ensure their safety.
A chilly night at the edge in Texas
Last month, in Austin, Texas, cyclist Robert Foster caught on camera a driverless Cruise vehicle turning left in an intersection and driving directly into the bike lane. While biking on Guadalupe Street near the University of Texas, Foster spotted the Cruise vehicle making a wide left turn straight into the bike lane and continuing on that path. A few minutes later, another Cruise car made the same turn and went straight into the bike lane again. Luckily, other than a couple of terrified bikers, no one was hurt in this incident, and no damage was done. Video 1 below shows the incident as broadcasted by KVUE, courtesy of Robert Foster.
Video 1 – Capture from the Austin incident
Root cause analysis of this particular incident is outside the scope of this blog, and we trust that Cruise will get to the bottom of this issue and provide a software update that overcomes this situation. But due to the complexity of the AV, you probably won’t be surprised if the next Cruise software update will not perform correctly in other circumstances. Even best-in-class AVs like Cruise’s still struggle in too many edge cases, and every month a new surprising failing scenario seems to be discovered.
Unleashing your constraints
Matt Fournier is a V&V expert in Foretellix’s Application Engineering team in Detroit.
“When most people look at the Austin incident, they see a vehicle turning left at night into an empty bike lane where a construction barrier is placed across the sidewalk extending partially into the bike lane,” said Matt. “When I’m thinking about this incident, I imagine all the possibilities that could have contributed to this behavior, as well the potential unknown causes of similar incidents. For example, a vehicle crosses a junction in any legal direction, starting from any possible lanes and ending at any of several possible lanes on the other side. I imagine night-time, daytime, rain, or fog. I also imagine black and green bike lanes, one lane mark on the left side, or two-lane marks on each side. I imagine multiple possible barriers like traffic cones, jersey barriers, and traffic barrels. I imagine scenarios where the bike lane is empty and scenarios where it’s occupied by one or more cyclists biking at various speeds. And I absolutely imagine all these scenarios happening in many other junctions in Austin. And then what about San Francisco, Boston, or Florida? Oh, and what about London, where cars drive on the left side?”
“I think you get my point,” says Matt. “A scenario like this can happen in so many places with so many different variations. You can never know which one will push the autonomous vehicle into an edge case that was not considered or tested before. Unless… you can actually test enough of them.”
Abstracting the Austin Scenario
The Cruise Austin incident can be considered as one concrete example of an infinitely large space of scenario variations that can be automatically generated by Verification and Validation (V&V) tools like the Foretellix Foretify™ platform and tested by AV providers to ensure acceptable safety under all conditions. Extending and abstracting the Austin incident using ASAM OpenSCENARIO 2.0® (OSC2) code is a powerful tool for validating the safety of the AV and helping to ensure it will operate appropriately under any similar or related conditions.
You can make such an abstract scenario part of a more comprehensive V&V library of abstract scenarios, such as the Foretellix V-Suite™, to cover many scenarios spanning the entire space of circumstances an AV can face within its Operational Design Domain (ODD).
Foretify™ generates large-scale scenario variations.
Matt developed the OSC2 example code in Figure 1 using Foretify Developer. In just 25 lines of code, it describes an abstract scenario of a vehicle crossing a junction in any direction and driving into a road with a bike lane under varying conditions.
The code points to an ASAM OpenDrive® map of downtown Austin. It specifies several variables: time of day, barrier type, position, vehicle speed, bike speed, and location.
The Foretify constrained-random test generator can automatically choose random junctions on the Austin map with crossing bike lanes; it can randomize all the specified parameters while ensuring that only valid combinations are generated.
Using Foretify Manager, Matt ran a weekend test suite using the OSC2 code that yielded over 10,000 concrete and meaningful variations in downtown Austin alone. The generated scenarios were orchestrated by Foretify and executed using the CARLA virtual simulator. Foretify Manager then analyzed results from the 10,000 test runs to ensure enough testing coverage was achieved across all variables.
Figure 2 shows a part of the Austin west campus area map, with several junctions that were selected by Foretify.
Big data analytics
“Generating and executing 10,000 tests is fairly easy using OSC2 and the Foretify constrained-random test generator,” said Matt. “Analyzing 10,000 test runs can be challenging if you try to do it manually. Remember that you probably want to run this test suite repeatedly to verify new software releases, making it even harder. That’s why Fortify Manager is such a powerful tool. It provides a dashboard showing your test coverage, which helps you ensure all variations and combinations of variations are tested. After you add KPIs and checks to your OSC2 code, Foretify Manager helps you analyze the KPIs across multiple conditions, and it uses the checks to highlight the conditions where the AV fails.” Figure 3 is a screenshot from the Foretify Manager coverage analysis dashboard of the weekend test suite Matt did.
Foretify generated examples
Matt captured 4 example scenarios generated by Foretify in Video 2. The top left scenario is a close duplication of the original incident. The top right scenario includes a biker crossing the junction riding on the bike lane on a single-lane road, and it has a smaller barrier size. The bottom left scenario consists of a leading blue vehicle on a two-lane road and no barrier. The bottom right scenario is in daylight.
“Imagine that these four example scenarios are just a tiny sample of so many possible variations generated inside my 10,000 test runs,” stresses Matt. “It can really blow your mind how many combinations you need to test to ensure your AV behaves well in all possible combinations. No one really knows the unknown corner scenario that will surprise the AI driver until you systematically test enough of them throughout the entire scenario space. With Foretify – you can actually do that”.
Video 2: simulated scenario examples
