CN116490858A - Adaptive Generation and Evaluation of Critical Scenarios for Autonomous Vehicles - Google Patents

Abstract

Aspects of methods, systems, and use cases for adaptive generation and modification of autonomous driving scenarios for testing in autonomous driving simulation platforms are described. In an example, a method for autonomous driving scenario assessment includes: identifying a first scenario for use in a simulation, the first scenario being defined by a first set of scenario parameters; identifying a first risk assessment metric for the first scenario based on safety operations used by an autonomous driving model in the first scenario; determining a second (changed) set of scenario parameters from changes in the first set of scenario parameters based on the second risk assessment metric; and generating a second scenario based on the second (changed) set of scenario parameters. For example, the second autonomous driving scenario may be more difficult for safety operations than the first risk assessment metric, thereby enabling increased testing complexity.

Description

Adaptive generation and evaluation of autonomous vehicle key scenarios

Priority claim

The present application claims priority from international application number PCT/CN2020/137372 filed on month 17 of 2020 and international application number PCT/CN2020/137373 filed on month 12 of 2020, each of which is incorporated herein by reference in its entirety.

Technical Field

Embodiments described herein relate generally to automated driving or driver assisted driving techniques, and more particularly, to methods for evaluation and generation of autonomous driving scenarios in simulation and test environments.

Background

The large-scale use of autonomous driving (autonomous driving, AD) is becoming more and more recent, but the safety of autonomous vehicles (autonomous vehicle, AV) remains a significant challenge. In the foreseeable future, AV will coexist on the road for a long time with human-driven vehicles. However, human-driven vehicles and pedestrians' behavior often have considerable uncertainty due to reckless driving, intentional violation of traffic rules, inattention, vehicle loss of control, inappetence, and the like. These uncertainties will lead to a number of unpredictable and dangerous situations for AV, which are referred to herein as "critical scenarios".

These real life dangerous situations are very valuable for the development and safety assessment of AV. With improvements, AV can be trained and learn how to predict and evaluate potential risks on roads, and how to interact with human drivers and pedestrians. However, existing methods for testing the performance and response of AV to various critical scenarios suffer from various limitations. Such test limitations include the amount of time required to consider a large number of critical scenarios applicable to AD settings, as well as the time required to identify, verify, and apply critical scenarios during testing of AV software.

Drawings

In the drawings (which are not necessarily drawn to scale), like numerals may describe similar components throughout the different views. The same numbers with different letter suffixes may represent different instances of similar components. Some embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:

FIG. 1 illustrates an autonomous vehicle system implementing autonomous driving software according to an example.

FIG. 2 illustrates an example data flow for adaptive generation and testing of critical scenarios according to an example.

Fig. 3 illustrates collision time calculation for autonomous driving cases for use with adaptive generation and testing of critical scenarios, according to an example.

FIG. 4 illustrates a collision time distribution with a search space defined using a collision time input, according to an example.

Fig. 5 illustrates a flow chart of a method for adaptive generation and use of key scenarios according to an example.

Fig. 6 illustrates an example of a mobile vehicle scenario featuring an autonomous vehicle operational safety model according to an example.

FIG. 7 illustrates an example workflow for operating a safety complexity assessment for an autonomous driving scenario, according to an example.

FIG. 8 illustrates vehicle locations in a test scene map according to an example.

Fig. 9A and 9B illustrate a test scene map providing link range adjustment according to an example.

FIG. 10 illustrates a test scene map that provides scene normalization according to an example.

Fig. 11A, 11B, and 11C illustrate examples of test scenario evaluation in a straight-path environment according to examples.

Fig. 12A and 12B illustrate examples of test scenario evaluation in an intersection environment according to examples.

Fig. 13 illustrates an example of a process flow for scene complexity assessment based on vehicle operational safety rules according to an example.

FIG. 14 illustrates a flow chart of an example method for generating and evaluating a scenario for testing autonomous driving operations, according to an example.

Fig. 15 illustrates a machine in the example form of a computer system for performing any of the methodologies discussed herein, according to an example.

Detailed Description

The following includes mechanisms and techniques that generally relate to identifying and testing scenarios for verifying AD operations performed by AD software and logic. In particular, the following describes an adaptive scenario generation system and method for use in a robust test procedure that can iteratively generate more critical (and more dangerous) test cases than a previous round of testing by adjusting certain parameters.

Additionally, the following provides a canonical and uniform assessment method for autonomous driving scenarios to improve test efficiency and help discover key cases. Such methods may be used not only for simulation scenarios, but also for real world field/road test scenarios. Thus, the following method may assist in the application and use of AV security models, as well as the resulting security improvements provided by such models.

Real world testing of AD software for vehicles is extremely expensive, risky and time consuming. To address these limitations, many aspects of AV operation and AD performance have been developed to test based on simulation testing. However, simulation-based testing typically involves manually designed scenes based on expert knowledge about traffic scene layout applicable to AD. For example, the scenes used in the test are typically selected from predefined scene data sets that are pre-collected, fixed, and cannot be dynamically changed during the test procedure. However, since most traffic scenarios are redundant or not challenging, even simulations are often too time consuming to fully test all possible scenarios on versions of AD software.

Such limitations of simulation-based testing are provided below as a solution. In the following examples, an adaptive scenario generation process is used to speed up the test time required for AD testing by removing or avoiding the use of redundant or low complexity test samples. Further, the scene generation process may generate and explore samples that have critical scenes but are not fully covered in the pre-collected scene dataset.

In an example, the adaptive scene generation process provides a closed loop process to progressively increase the complexity of the test scene and adaptively generate samples and scenes. The adjustment of various parameters of the scene may be identified based on critical measurements useful for the test, such as time to collision (time to collision, TTC), and the adjustment of these parameters may be divided into different risk levels. Other examples of using a particular search algorithm to modify the state, and identifying different conditions with statistical models are also disclosed.

Additionally, this adaptive scene generation method can be easily used as a plug-in or extension to any scene-based AD test framework, thereby enabling even existing test systems to provide more versatile and challenging test cases. Other uses and applications of the present technology for AD testing and AD performance will also be apparent.

While both AD and advanced driver assistance systems (advanced driver assistance system, ADAS) systems incorporate various safety features, verifiable safety models are still being created for vehicle operation. These models tend to formalize parameters of motion and interaction between vehicles, use these parameters to model vehicle presence in the world, and define acceptable interactions between vehicles based on vehicle presence. One such vehicle operational security model (vehicle operation Safety model, VOSM) is responsibilities-Sensitive security (RSS).

Many of the scenarios and examples discussed below include parameters or formulas from RSS, although most examples are equally applicable to parameters and formulas from other VOSMs. The VOSM defines several parameters for each vehicle and uses these parameters to model a set of distances that determine whether the vehicle is safe, and so on. Generally, the safety distance relates to a sufficient longitudinal distance and a sufficient transverse distance. This will be discussed below with reference to fig. 6, which illustrates the relationship of these distances in the security model.

VOSM provides a mathematical model for safety assurance during autonomous driving. VOSM in particular formulates a set of safety criteria (such as minimum distance d between vehicles to avoid collisions _min ). Multiple parameters (such as response time p of vehicle, minimum brake a _min,brake And maximum acceleration a _max,brake ) Is used to calculate the formula. If all requirements are met, the vehicle passes the VOSM standard and is believed to be safe, otherwise the vehicle is unsafe.

Various forms of security requirements are being developed to standardize such methods. For example, the chinese ITS standard "technical requirements for security for AV decision making" (T/ITS 0116-2019) established an RSS-based security standard for use in china. To advance the use of this standard, test standards such as the chinese ITS standard "test procedure and evaluation rules for security for AV decision making" (T/ITS 0116-2019) are also under development. It is expected that the test standard will require the execution of three parts: simulation testing, field testing, and road testing.

Currently, it is very difficult to measure or quantify the difficulty of autonomous driving scenarios, especially for on-site and road testing scenarios. However, when a large number of scenes need to be evaluated, accurate scoring or measurement of each scene is necessary. Using current simulation and testing methods (such as evaluation methods that identify the density of objects in traffic, and these methods are simply labeled with basic tags such as "crowded" or "few vehicles"), there is only a basic form of identifying the type of scene. Such differentiation is very ambiguous and, therefore, test results from different scenarios are difficult to compare to each other under the same criteria.

Accordingly, a more detailed scenario complexity evaluation method based on VOSM rules for autonomous driving is also provided below. In the following, an evaluation method for evaluating speed variation in a virtual self-path and time cost generated with a fastest route is provided. The route may be dynamically calculated by VOSM rules in combination with a range of road segments defining a drivable area. In the event that all actors are cleared, the evaluation will be normalized by the same scenario. Based on the results of the complexity assessment method, the scene may be further adjusted or modified (e.g., to increase the complexity of the test scene and identify new scene changes, as described above).

As an overview of AD systems, various AD and ADAS models, logic, and algorithms use vehicle sensor data to control or assist in controlling (e.g., via driver prompts, partial steering inputs, emergency braking, etc.) a vehicle. AD systems may fully control a vehicle without driver assistance, while ADAS systems enhance control of the driver of the vehicle, or automate, adjust, or enhance vehicle systems to improve safety and provide better driving. In ADAS systems, safety features are designed to avoid collisions and accidents by providing a technique to alert the driver to potential problems, or to avoid collisions by implementing security measures and taking over control of the vehicle. The following provides an overview of the testing and evaluation of scenarios related to the testing of AD models, but it will be appreciated that such testing and evaluation operations are also related to ADAS models.

Fig. 1 is a schematic diagram illustrating an example AD system 100 implemented within an autonomous vehicle. The system 100 and the data processing platform 102 that it includes are depicted in a simplified manner because a number of features are not shown or described for purposes of simplicity. As will be appreciated, the particular software, logic, or functional AD operations performed by the data processing platform 102 may depend on a version of the software, logic, or data that may be programmed, reprogrammed, or updated at regular intervals by an authorized entity (e.g., a vehicle manufacturer or service provider). The present techniques for scene verification may be used to perform tests on such software, logic, or data to ensure that AD features operate correctly and safely.

Fig. 1 includes a data processing platform 102 incorporated into a vehicle 104. The data processing platform 102 includes a sensor array interface 106, processing circuitry 108, data identification circuitry 110, and a vehicle interface 112. In various aspects, the operation of the processing circuitry 108 or the data identification circuitry 110 may be adjusted using programmed software, logic, or data that is tested and verified using the techniques discussed herein.

The vehicle 104 (which may also be referred to as a "self-vehicle," "host vehicle," or "host vehicle") may be any type of vehicle, such as a commercial vehicle, consumer vehicle, recreational vehicle, automobile, truck, motorcycle, boat, unmanned aerial vehicle, robot, aircraft, hovercraft, or any mobile aircraft capable of operating at least in part in an autonomous mode. The vehicle 104 may sometimes be operated in a manual mode, wherein a driver conventionally uses pedals, steering wheels, and other controls to operate the vehicle 104. At other times, the vehicle 104 may operate in a fully autonomous mode, wherein the vehicle 104 operates without user intervention. Further, the vehicle 104 may operate in a semi-autonomous mode, wherein the vehicle 104 controls many aspects of driving, but the driver may use conventional inputs (e.g., steering wheel) and non-conventional inputs (e.g., voice control) to intervene in or affect operation. In this way, the vehicle may operate simultaneously or at different times in any number of driving automation levels defined from level 1 to level 5 (e.g., as defined by SAE international J3016: level 1, driver assistance, level 2, partial driving automation, level 3, conditional driving automation, level 4, high driving automation, level 5, full driving automation). The vehicle may also operate using combinations or variations of these levels. For example, the vehicle may operate according to the new concept l2+ (2-plus), which describes a hybrid ADAS/AV for facilitating an enhanced driving experience and improving safety without providing a fully autonomous control system.

The sensor array interface 106 may be used to provide input or output signaling to the data processing platform 102 from one or more sensors in the sensor array mounted on the vehicle 104. Examples of sensors include, but are not limited to: a forward camera, a side camera, or a backward camera; a radio detection and ranging sensor; liDAR; an ultrasonic distance measurement sensor; or other sensor. Forward or forward direction is used in this document to refer to the main direction of travel, the direction in which the seat is arranged to face, the direction of travel when the transmission is set to drive, etc. Conventionally, then, backward or rearward is used to describe sensors pointing in directions generally opposite those of forward or forward. It should be appreciated that some forward facing cameras may have a relatively wide (even up to 180 degrees) field of view. Similarly, a rearward camera pointing at an angle (possibly 60 degrees off center) for detecting traffic in an adjacent traffic lane may also have a relatively wide field of view, which may overlap with the field of view of the forward camera. Lateral sensors are those that are directed outward from the sides of the vehicle 104. The cameras in the sensor array may include infrared or visible light cameras capable of focusing at long distances with a narrow field of view or short distances with a large field of view. The vehicle 104 may also include various other sensors, such as driver identification sensors (e.g., seat sensors, eye tracking and identification sensors, fingerprint scanners, voice recognition modules, etc.), occupant sensors, or various environmental sensors for detecting wind speed, outdoor temperature, barometric pressure, rain/humidity, etc.

The sensor data is used to determine the operational context of the vehicle, environmental information, road conditions, driving conditions, etc. The sensor array interface 106 may communicate with another interface of the vehicle 104, such as an on-board navigation system, to provide or obtain sensor data. The components of the data processing platform 102 may use a network to communicate with components internal to the data processing platform 102 or components external to the data processing platform 102, which may include a Local Area Network (LAN), a Wide Area Network (WAN), a wireless network (e.g., IEEE 802.11 (Wi-Fi) or cellular (e.g., 4G/5G) network), an ad hoc network, a personal area network (e.g., bluetooth), a vehicle-based network (e.g., controller area network (Controller Area Network, CAN) bus), or other combinations or permutations of network protocols and network types. The network may comprise a single Local Area Network (LAN) or Wide Area Network (WAN), or a combination of LANs or WANs, such as the internet. Various devices coupled to the network may be coupled to the network via one or more wired or wireless connections.

The data processing platform 102 may communicate with a vehicle control platform 118 using the vehicle data interface 112 to receive and obtain vehicle data. The vehicle control platform 118 may be a component of a larger architecture that controls aspects of the operation of the vehicle. The vehicle control platform 118 may have interfaces to autonomous driving control systems (e.g., steering, braking, acceleration, etc.), comfort systems (e.g., heating, air conditioning, seat positioning, etc.), navigation interfaces (e.g., map and routing systems, positioning systems, etc.), collision avoidance systems, communication systems, safety systems, vehicle status monitors (e.g., tire pressure monitors, fuel level sensors, battery level sensors, speedometers, etc.), and the like. The vehicle control platform 118 may control or monitor one or more subsystems and communicate data from such subsystems to the data processing platform 102. In some examples, features of the sensor array interface 106 and the vehicle data interface 112 are integrated into the same or coordinated data collection interface to receive data from at least one sensing component of the vehicle. Such data may be provided via the interface(s) during autonomous operation of the vehicle, and such data may be automatically recorded or monitored.

In an example, sensor data (such as brake, throttle, speed data signals, and other data signal types) may be provided to the data identification circuitry 110, which data identification circuitry 110 may pre-process the input signal. The data identification circuitry 110 may implement various rules, algorithms, or logic, including one of several types of machine learning, such as artificial neural networks (artificial neural network, ANN), support vector machines (support vector machine, SVM), gaussian mixture models (Gaussian mixture model, GMM), deep learning, and the like. Based on the possible classifications or data value identifications, the processing circuitry 108 may initiate one or more responsive data processing, logging, or communication activities. Other autonomous vehicles and data processing actions may be monitored, coordinated, or initiated depending on the type, severity, location, or other aspects of the event detected using the data processing platform 102. It will be appreciated that any number of models, logic, and algorithms for AV and data processing operations may be simulated, tested, and validated using the techniques discussed herein.

In order to improve the test efficiency of software for operating such AV, the following proposes to generate more key scenarios for input to the AD test system, thereby greatly reducing test time and cost. In more detail, the downsampling process below identifies the process of generating a higher risk (and more complex) scene by taking into account feedback measured from previous test results. Further, to measure the risk level, different metrics may be used, such as TTC (time to collision). Given the risk metrics, a search-based approach and an improved approach to conditional probability sampling for generating more challenging cases and inputs for AD software testing is discussed below.

Fig. 2 depicts a system architecture for AD testing. In particular, fig. 2 depicts a scenario-based AD test framework involving sampling operations 202, a test engine 206, evaluation decisions 208, and outputs 210. The AD test framework is extended below using an adaptive scenario/sample generation system 212, which system 212 improves test speed by generating (than the previous few rounds) more challenging/risky scenarios for the test engine 206.

In conventional test setup, at each test step, the test engine 206 will extract a scene from the pre-collected dataset (e.g., extract a sample from the scene database 204 using the sampling operation 202) and provide this input to the test engine 206 until enough steps or until some criteria are met by the evaluation decision 208 (such as enough failure cases, etc.). In such conventional test settings, the extracted inputs are only a subset of the pre-collected scene data sets from database 204. The scene dataset collected in advance must be large enough to cover all cases. However, this may result in testing using redundant samples, further slowing down the time required for testing, as the test samples of the next round do not take into account the previous test results.

The test framework is extended by an adaptive scene/sample generation system 212, which system 212 uses further methods for test case sampling. For example, the sampling method may apply different sampling mechanisms, such as random sampling, monte carlo sampling, importance based sampling, etc., to reduce test time. Further, the adaptive scenario/sample generation system 212 considers the last round of test results and generates more challenging/risky cases for the test engine.

As depicted, the result collection process 214 of the system 212 operates to collect current test results and calculate a risk metric (e.g., TTC) related to identifying further critical scenarios. The adaptive sample generation process of system 216 operates to adaptively generate more challenging scenarios based on results from one or more rounds of previous tests. These generated scenarios are then provided to the test engine 206 for further testing and evaluation.

As will be appreciated, the presently described systems and techniques may be used as modules or plug-ins to presently available test systems. Thus, existing systems may also implicitly or explicitly generate more key scenes (e.g., new scenes) based on testing of even one input scene.

Additional details regarding the processes used in adaptive scene generation are provided below. In particular, such adaptive generation may generate more critical/dangerous cases than the previous round of testing by adjusting certain parameters. To achieve this, three problems are solved: 1) How to parameterize and represent a scene; 2) How to measure the key level/degree of the test scene; 3) How to generate more key scenes by adjusting scene parameters.

1) Scene parameterization and representation. Typically, a scene is composed of all traffic participants (vehicle _i (vehicle i)) and road layoutThe subscript i indicates the ith traffic participant. For each participant (e.g. a vehicleWith) assuming parameterized trajectories ∈ ->Where the parameter vector θ may be the velocity, acceleration or distance of each time step. Road->May be static or time-varying +.>But static roads may be considered for simplicity. This method also takes into account the initial state x ⁰ ＝x(t ₀ The method comprises the steps of carrying out a first treatment on the surface of the θ), initial time t ₀ And a horizontal time T. In this example, the scene is represented by a tuple:

2) Critical measurement. In an example, metrics: TTC (time to collision) is a popular safety indicator that is defined as the time required for two vehicles to collide while maintaining the current heading path and motion model. The basic assumption is constant velocity or acceleration. The instantaneous TTC of VUT (vehicle under test ) and VT (vehicle target) can be defined as:

Wherein D is _rel And V _rel The relative distance and velocity between VUT and VT are shown, respectively. In TTC, high risk events tend to be associated with small D _rel Corresponding to TTC.

In particular the number of the elements to be processed,

this is illustrated with reference to fig. 3, which provides TTC calculations for side case 302, back-end case 304, and head-on case 306.

In an example, because TTCs may be used as risk assessment metrics, four risk levels may be defined by predefining several thresholds (e.g., given statistical analysis methods) on the TTCs, as shown in table 1.

TABLE 1

Definition of risk level for given TTC value

Given the above-described scene parameterization and TTC as a risk metric, the scene generation problem becomes to generate a more critical scene (with smaller TTC values and in the range of risk level II or III) than the last round of input scene samples. In particular, the parameter p is expected to be adjusted to generate different elements in the scene tuple, including initial state, dynamic trajectory, and road layout. Hereinafter, examples are provided as to how to adaptively generate scenes having different initial states and dynamic trajectories. It will be appreciated that other methods may be provided.

In a further example, grid search based scene generation may be performed, such as by generating a grid search based scene by generating a scene based on a scene of an initial state (x ⁰ ) Is used for generating the adaptive scene. Letting car-1 represent VUT, the problem becomes to adjust the parameters of car-1 and car-2, and to generate more dangerous or critical scenarios for testing. Given the TTC calculation described above, the parameters include θ= (d..v.) in the side case, or θ= (x..v.) in the back-end or face case. Under such settings, the following grid search algorithm may be used to generate more critical scenarios for testing.

TABLE 2

Adaptive scene generation algorithm based on grid search for initial state (side case)

Similar search algorithms can be easily deployed on back-end and head-on cases using p= (x., v.). Furthermore, the algorithm may use other search methods with user-defined search spaces or parameters instead of grid searching.

In a further example, scene generation based on grid search may be performed. Such examples may include scene generation by adapting the trajectory (x. (t; θ)). In particular, the following relates to a change x (t; θ) of a new scene. This scene generation method first samples the time step, holds the state before this moment, and regenerates a new scene after the point in time of the sampling.

TABLE 3 Table 3

Adaptive scene generation algorithm (side case) based on grid search for trajectories

In a further example, efficient scene generation may be performed by probabilistic modeling. To further improve the efficiency of sample generation, TTC probability distributions given any NDS (Naturalistic Driving Study, natural driving study) dataset can be represented. A representation of a dataset is shown in fig. 4, which depicts an example TTC distribution 400 in which TTC inputs are used as conditions for identifying a search space. Within this method, the evaluation becomes: given any input TTCin value (calculated from one scenario), a new sample with a smaller TTC value is generated. Here, any conditional monte carlo based generation method (such as using a naive monte carlo or importance based sampling method) can be used to sample the search space using < TTCin.

FIG. 5 depicts a flowchart 500 of a method for adaptive generation and use of critical scenarios for implementing the present technology.

Operation 502 involves identifying an initial scene sample (such as from a scene dataset) for testing. This initial scene sample may involve testing some key scene condition for AD operation based on known conditions or modeling.

Operation 504 involves testing an initial scenario with AD operation according to the various AD test methods discussed above. Data from the initial scenario test is collected and used for further analysis in the following operations. Such data may be parameterized and measured as discussed above.

As discussed above, operation 506 includes identifying a risk assessment metric from the test of the initial scenario. For example, this risk assessment metric is provided by measuring the time to collision value from the initial scenario test.

Operation 508 includes adjusting parameters of the initial scene based on the risk assessment metrics, as discussed above. From these adjustments, subsequent test scenarios with new parameters (e.g., with different initial states and dynamic trajectories) may be generated.

Operation 510 comprises testing for subsequent scenarios. Operations 506-510 may be repeated, using the results of the subsequent scenes for further risk assessment, adjustment, and generation of additional subsequent scenes.

FIG. 6 illustrates an example of a mobile vehicle scenario, demonstrating the application of an autonomous vehicle operational safety model. As illustrated, the self-vehicle 605 shares the road with two target vehicles, a car 620 and a truck 625. Herein, "self-vehicle" refers to a vehicle from which a certain perspective is obtained, and the target vehicle is the subject within that perspective.

As described above, the VOSM may provide a mathematical model for safety assurance during autonomous driving. The VOSM will typically formulate a set of safety criteria (such as a minimum distance d between vehicles to avoid collisions _min ). Multiple parameters (such as response time ρ of vehicle, minimum brake a _min，brake And maximum acceleration a _max，brake ) Is used to calculate the formula. If all of the requirements are met,the vehicle passes the standard and is believed to be safe, otherwise the vehicle is unsafe.

In an example, the VOSM (or "security model" as used herein) defines a safe longitudinal distance 610 and a safe lateral distance 615 for the self-vehicle 605. These distances create areas, enclosures, bubbles, or shields around the self-vehicle 605, also illustrated around the car 620 and truck 625. In general, violating these safety distances (e.g., intersection or overlap 630) indicates that the self-vehicle 605 is unsafe and corrective action should be taken. Note that the intersection 630 does not necessarily result in a collision, but an unsafe condition occurs according to the safety model.

The security model may provide the following representation of the safe longitudinal distance (minimum longitudinal safe distance):

The symbol [ x ] +: = max { x,0}, here. Wherein:

v _f longitudinal speed of the front vehicle;

maximum braking acceleration of the front vehicle;

v _r longitudinal speed of the rear vehicle;

ρ is the response time of the rear vehicles;

maximum forward acceleration of the rear vehicle;

minimum braking acceleration of the rear vehicle.

The model also provides the following representation of the safe lateral distance (minimum lateral safe distance):

wherein:

v ₁ lateral speed of self-vehicle;

v ₂ the transverse speed of the target vehicle;

ρ ₁ response time of self-vehicle;

ρ ₂ response time of the target vehicle;

μ, transverse disturbance range;

maximum lateral acceleration of the self-vehicle;

maximum lateral acceleration of the target vehicle;

minimum lateral braking of the self-vehicle;

minimum lateral braking of the target vehicle;

the lateral speed of the self-vehicle at the end of the response time;

lateral speed of target vehicle at end of response time

Note that: v _1，ρ ≥0；v _2，ρ ≤0。.

Thus, relative to the safe longitudinal distance of equation (4), a _max，accel And a _min，brake Is self-vehicle 605 (c) _r ) And ρ is the response time of the self-vehicle 605. ρ is relative to the safe lateral distance of equation (2) ₁ Is self-vehicle 605 (c) ₁ ) Response time and p ₂ Is the target vehicle (c) ₂ ) Such as truck 625, response time. In addition, in the case of the optical fiber,and->Respectively c ₁ Maximum acceleration rate and minimum brake rate, < ->And->Respectively c ₂ A maximum acceleration rate and a minimum brake rate.

The self-vehicle 605 is expected to perform a corrective action when it detects that it is closer than either the minimum safe longitudinal distance or the minimum safe lateral distance. Such corrective action may include braking or becoming increasing the distance between the self-vehicle 605 and the target vehicle 625 or other object until the minimum safe longitudinal distance and the minimum safe lateral distance are restored.

Equations (4) and (5) above illustrate the parameterization of the safety model's response time to the self-vehicle 605 and the target vehicle 625, the maximum lateral acceleration or maximum longitudinal acceleration of the target vehicle, and the minimum braking (e.g., deceleration) of the self-vehicle. Here, the maximum acceleration is the maximum acceleration that can be achieved by the vehicle, and the minimum braking is the deceleration that can be ensured by the vehicle when performing a maneuver. Thus, if the vehicle is in peak operating conditions, the maximum braking and minimum braking may be the same. However, if, for example, the self-vehicle 605 has worn the brakes, the minimum braking of the self-vehicle 605 is reduced from the maximum braking based on the brake wear.

The actual values for either the maximum and minimum of braking or acceleration are generally defined by the manufacturer of the self-vehicle 605, or by the VOSM or the like. These values are defined to provide an actual safety margin given equations (4) and (5). Note that equations (4) and (5) assume generally a worst case scenario, where even though it is more likely that the performance of the self-vehicle 605 will exceed its minimum brake and the performance of the target vehicle 625 will be below its maximum acceleration, the self-vehicle 605 performs poorly (thereby using minimum brake for the self-vehicle) and the target vehicle 625 is at peak performance (thereby using maximum acceleration for the target vehicle 625).

By using equations (4) and (5), a hazard zone is defined around the self-vehicle 605. As described above, when another object interferes with the area, or is projected to interfere with the area, then the self-vehicle 605 is expected to take action. Because the speeds of both the self-vehicle 605 and the target vehicle 625 are parameters of equations (1) and (2), the hazard zone continuously changes based on the detected movement of the self-vehicle 605 and the target vehicle 625.

Other VOSMs will typically follow the security model templates and requirements described above by defining relevant parameters and providing acceptable vehicle interactions based on these parameters.

FIG. 7 illustrates an example workflow for operating a safety complexity assessment for an autonomous driving scenario. As discussed in the examples below, the safety zone is dynamically calculated from the speed of the surrounding vehicle and the speed of Pn using the safety model rules of longitudinal and lateral distances. Thus, at operation 702, the workflow includes adjusting a position along the trajectory based on the security model rules; the security area and location along the track are then identified at operation 704. Additional adjustments and positioning may be made after further iterations.

FIG. 8 provides an illustration of vehicle position in a test scene map 802, illustrating movement of a test vehicle along a designated point (P0, P1, P2, P3). The starting point is P0 and the target point is the destination. The vehicles a/B/C are other actors in the scene and will follow actions and routes predefined by the scene.

In fig. 8, a safety zone 804 (at the current point in time) is calculated according to safety model rules related to the speed of the surrounding vehicle. In an example, R806 is system defined and represents a road segment range. R806 may be a constant or a piecewise variable.

As shown in fig. 8 and subsequent figures, a number of waypoints (e.g., points P1, P2, P3) are selected for evaluation. In an example, each waypoint Pn added to the waypoint list meets the following requirements: (a) All candidate points are located on the boundary of a security circle centered on Pn-1 and in the security model rule security region (security region 804); (b) Pn should be a point having the shortest distance from the target point; (c) At certain critical moments, there is no safety zone for travel. The waypoint should remain the same as the previous waypoint.

In an example, the waypoint selection may be provided by the following waypoint selection algorithm.

Table 4 waypoint selection algorithm

In various examples, a road segment range (e.g., R806) may be selected, defined, or adjusted. For example, the road segment range R may be defined by the system as required. It may be constant for all segments or may vary from segment to segment throughout the route. The road segment range R should be small enough to accommodate complex road environments. The value can be initialized by expert knowledge and then adjusted by the security area.

In an example, the road segment range selection may be provided by the following road segment range adjustment algorithm.

Table 5 dynamic road segment Range adjustment Algorithm

Fig. 9A and 9B provide additional illustrations of examples of maps 902A, 902B for adjusting road segment ranges. At t=t1, a security area 904A is defined; at t=t1, pl will be the next waypoint according to the security area 906 b. However, the trajectories of P0 to Pl will be at timeVehicle C moving to the right during this period is occluded. Thus, the road segment range needs to be adjusted or reduced to ensure that the trajectory is not occluded.

In a further example, a cost function may also be considered in the method. The total cost includes time costs and maneuver costs. The time cost is the integral of the route selection time for each road segment. For each road segment, the same value is assigned for speed and set as the maximum speed limit for the fastest route selection.

Special cases occur when there is no security area for the next waypoint selection; this means that the routing needs to be stopped. In this case, the waiting time will be collected as follows.

Cost＝Cost _Time +Cost _Maneuver (cost = cost _Time Cost + cost _{Motorized drive} )

Maneuver costs are designed to evaluate maneuver changes that may reflect the complexity of the maneuver. Similar to the time cost, in some cases there is no next waypoint, so the speed will be zero.

In a further example, normalization may also be considered. Since each scene has various road networks that are difficult to compare (e.g., intersections versus curved roads), normalization steps will be applied in the pipeline. All actors will be removed from the scene and the total cost will be recalculated as a benchmark for normalization.

Fig. 10 provides an example of an intersection in a map 1002. Dotted lines 1004, 1006 refer to the trajectories of actors a and C. The dashed line 1008 with nodes refers to the trajectory and waypoints with individual actors A, B, C in the scene. The dashed line 1010 with nodes refers to the trajectory and waypoint without an actor A, B, C. Here, the dashed line 1010 is smooth and reflects the base forward direction. By introducing normalization steps, the impact of road networks (intersections) can be reduced.

The cost function can be rewritten as:

Cost _Time ' and Cost _Maneuver ' is the corresponding time cost and maneuver cost in road clear situations.

The operations may be applicable to other case examples, such as those shown in map 1102A, 1102B, 1102C in fig. 11A, 11B, and 11C, which illustrate times T0, T1, T2, respectively, for straight-road scene assessment. Here, vehicles A, B and C are actors in the scene, and they will change lanes and accelerate forward to the right.

P0 is the starting point and R is assigned a constant value. At t=t0, we need to find the next waypoint Pl. At this time, P0 is the center of the safety circle having the radius R. In the green region, it has the point with the shortest distance from the target point. Subsequently, it will be assigned as Pl. Repeating these steps for P1/P2/P3, and so on.

Another example of case is shown in maps 1202A, 1202B in fig. 12A and 12B, respectively, illustrating times TO, tl for an intersection scene. In an intersection scenario, actor A, C will follow tracks 1204, 1206, while actor B will remain in its position.

As depicted in fig. 12A, the virtual self-path begins at P0 at time TO with an initial road segment range r=r1 (in range 1210) from the security zone 1208, where Pl is the next waypoint Pl. In time Tl of the scene, as depicted in fig. 12B, the next waypoint is P2' within the same road segment range r=r1 (range 1212). However, the trajectory from Pl to P2 will be occluded by the moving actor C. As a result, the range R should be adjusted to r=r2 (range 1214), R2 being smaller than Rl. P2 may then serve as the next waypoint.

Fig. 13 depicts a flowchart 1300 of a method of scene complexity assessment based on vehicle operational safety rules in accordance with the present technique. As discussed herein, the method may be used to evaluate a scenario (e.g., a scenario for testing or validating vehicle control system software) with a VOSM in a test setup. In the following flowcharts, RSS or other security models may be tested and applied as relevant VOSMs.

Operation 1302 includes calculating a safety zone based on VOSM rules, such as for evaluating longitudinal safety distance and lateral safety distance based on other vehicle speeds.

Operation 1304 includes selecting waypoints for security evaluation based on the security area. As described above, these waypoints may be selected at the edges of a known security zone. The algorithm for waypoint selection is discussed above.

Operation 1306 includes adjusting a drivable path range of the test scenario for evaluation with the safety rules so that the critical scenario may be invoked. Algorithms for road segment range adjustment are discussed above.

Operation 1308 involves evaluating a cost function of the adjusted drivable path range using a cost function comprising a time cost and a maneuver cost. Details of such cost functions are discussed above.

Operation 1310 involves generating an assessment of a scene based on the security area, the adjusted road segment range, and the cost function. Thus, the difficulty of the test scenario may be quantified or represented.

Fig. 14 illustrates a flowchart 1400 of a method for generating and evaluating a scenario for testing autonomous driving operations. The operations of the method may be performed in a test simulator or platform, in a device integrated within a simulation computing engine, service, machine, apparatus, or as part of instructions from a computer-readable storage medium or apparatus-readable storage medium executed by circuitry of a computer machine or apparatus.

At 1402, operations are performed to obtain and/or identify autonomous driving scenarios for evaluation, such as selecting a particular scenario for use in a test simulation. The autonomous driving scenario may be a first (initial) autonomous driving scenario in an iterative stream of multiple analyzed scenarios. An autonomous driving scene is defined by a first plurality of scene parameters. In a particular example, the scene parameters specify characteristics of the road, participants on the road, and trajectories of each of the participants, or other scene characteristics related to testing and simulation as described above.

At 1404, operations are optionally performed to cause testing of an autonomous driving scenario using safety operations of the autonomous driving model. For example, since the testing of the autonomous driving scenario applies the safety operation to the autonomous driving scenario, at least a portion of the first autonomous driving scenario may be tested on the autonomous driving simulation platform. In this scenario, a (first) risk assessment metric is generated based on results in a test from the autonomous driving scenario.

At 1406, an operation is performed to identify or determine a risk assessment metric for the autonomous driving scenario. The risk assessment metric is based on one or more safety operations used by the autonomous driving model in the driving scenario. One or more values of the one or more risk assessment metrics (e.g., the first risk assessment metric) may be generated based on the results from the security operation of the test (at operation 1404) or other evaluative processes, consistent with the examples discussed above.

At 1408, an operation is performed to change or otherwise determine new parameters that result in greater difficulty of the security operation and increased risk assessment metrics. In an example, a first plurality of scene parameters from a first scene is used as a starting point, and such parameters are modified to produce a second plurality of scene parameters. In certain examples, the second plurality of scene parameters is determined by performing a grid search on the scene parameters, or by performing probabilistic modeling on the scene parameters to identify a search space, as discussed above.

The second risk assessment metric may be used to indicate or identify greater difficulty for the security operation than the first risk assessment metric. In a particular example, the risk assessment metric is associated with a Time To Collision (TTC) calculation. In a further example, such TTC calculations result in a value within one of a plurality of risk levels. Such risk levels may be used to identify more difficult scenes and scene parameters, such as when the second risk assessment metric is associated with (or provides) a TTC calculation that produces a value within a risk level equal to or greater than the first risk assessment metric.

At 1410, an operation is optionally performed to generate an assessment (e.g., score, rating, metric) of the relevant scenario based on a test or evaluative result of the safety operation in the autonomous driving scenario. For example, the evaluation may be based on results of test, simulation, or risk assessment measurements related to a first autonomous driving scenario (with a first set of scenario parameters) and a second autonomous driving scenario (with a second/changed set of scenario parameters). Results or evaluations of other scenes or scene sets may also be considered. In particular examples, the difficulty assessment may be based on a safety area, road segment range, or cost function for a maneuver (or maneuvers) associated with a safety operation performed with the test autonomous driving scenario. As discussed above, the cost function may be related to a time cost or a complexity cost for the maneuver. Likewise, the road segment range may be related to waypoints used in the simulation for the maneuver.

At 1412, the flowchart ends with generating a second (new) autonomous driving scene from the second (new) scene parameter set. This second autonomous driving scenario may be used for further simulation, testing or evaluation. In further examples, the operations of testing and generating subsequent autonomous driving scenarios (including the second autonomous driving scenario) may be used during the multiple rounds of testing (e.g., returning to operation 1402 for the second scenario, the third scenario, or the nth scenario). Such iterative operations may be used to progressively increase the respective risk assessment metrics generated for the scenarios, which indicate greater difficulty for safety operations in each of the subsequent autonomous driving scenarios. Although not shown, the iterative operation may continue until a certain condition is identified or met. Other uses for scenario evaluation for testing, simulation, and evaluation purposes may be used as discussed herein.

Embodiments may be implemented in one or a combination of hardware, firmware, and software. Embodiments may also be implemented as instructions stored on a machine-readable storage device, which may be read and executed by at least one processor to perform the operations described herein. A machine-readable storage device may include any non-transitory mechanism or medium for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable storage device may include read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, and other storage devices and media. The processor subsystem may be used to execute instructions on a machine-readable medium. The processor subsystem may include one or more processors, each having one or more cores. Additionally, the processor subsystem may be provided on one or more physical devices. The processor subsystem may include one or more special purpose processors, such as a graphics processing unit (graphics processing unit, GPU), digital signal processor (digital signal processor, DSP), field programmable gate array (field programmable gate array, FPGA), or fixed function processor.

Examples as described herein may include or operate on logic or several components, modules, mechanisms, or units (collectively referred to as components). Such components may be hardware, software, or firmware communicatively coupled to one or more processors to perform the operations described herein. A component may be a hardware component and, as such, may be considered a tangible entity capable of performing the specified operations and may be configured or arranged in some manner. In an example, the circuits may be arranged as components in a specified manner (e.g., internally or with respect to external entities such as other circuits). In an example, all or part of one or more computer systems (e.g., a stand-alone client or server computer system) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) to operate components for performing specified operations. In an example, the software may reside on a machine readable medium. In an example, the software, when executed by the underlying hardware of the component, causes the hardware to perform the specified operations. Thus, the terms hardware component, module, mechanism, or unit are understood to encompass a tangible entity, be it an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a specified manner or to perform some or all of any of the operations described herein. Considering the example in which components are temporarily configured, each of these components need not be instantiated at any one time. For example, where the components include a general-purpose hardware processor configured using software, the general-purpose hardware processor may be configured as corresponding different components at different times. The software may configure the hardware processor accordingly, for example, to construct a particular component at one instance of time, and to construct a different component at a different instance of time. A component may also be a software or firmware module that operates to perform the methods described herein.

As used in this document, circuitry or circuitry may include, for example, alone or in any combination: hard-wired circuitry; programmable circuitry, such as a computer processor including one or more individual instruction processing cores; a state machine circuit system; and/or firmware storing instructions executed by the programmable circuitry. The circuitry, or modules may be embodied collectively or individually as circuitry forming part of a larger system, such as an integrated circuit (integrated circuit, IC), system on-chip (SoC), desktop computer, laptop computer, tablet computer, server, smart phone, or the like.

As used in any embodiment herein, the term "logic" may refer to firmware and/or circuitry configured to perform any of the foregoing operations. The firmware may be embodied as code, instructions, or a set of instructions and/or data that are hard-coded (e.g., non-volatile) in a memory device and/or circuitry. As used in any embodiment herein, "circuitry" may include, for example, singly or in any combination, hardwired circuitry, programmable circuitry, state machine circuitry, logic, and/or firmware that stores instructions executed by programmable circuitry. The circuitry may be embodied as an integrated circuit (such as an integrated circuit chip). In some embodiments, circuitry may be formed, at least in part, by processor circuitry that executes code and/or sets of instructions (e.g., software, firmware, etc.) corresponding to the functions described herein to thereby convert a general-purpose processor into a special-purpose processing environment to perform one or more of the operations described herein. In some embodiments, the processor circuitry may be embodied as a stand-alone integrated circuit or may be incorporated as one of several components. In some embodiments, various components and circuitry of a node or other system may be combined in a system-on-a-chip (SoC) architecture. In other examples, the processing circuitry may be embodied or provided by a combination of a data processing unit (data processing unit, DPU), an infrastructure processing unit (infrastructure processing unit, IPU), acceleration circuitry, or a graphics processing unit (graphical processing unit, GPU) or a programmed FPGA.

As discussed herein, various structural and functional components may be implemented by circuitry. Circuitry (e.g., processing circuitry) is a collection of circuits comprising hardware (e.g., simple circuits, gates, logic, etc.) implemented in a tangible entity of machine 1500. Circuitry membership may be flexible over time. Circuitry includes members that, when operated, can perform the specified operations, either alone or in combination. In an example, the hardware of the circuitry may be unalterably designed to perform a particular operation (e.g., hardwired). In an example, hardware of the circuitry may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) comprising a machine-readable medium physically modified (e.g., magnetically, electrically, movably arranged, etc. with respect to unchanged aggregated particles) to encode instructions of a particular operation. When connecting physical components, the underlying electrical properties of the hardware components change, for example, from an insulator to a conductor, or vice versa. These instructions enable embedded hardware (e.g., execution units or loading mechanisms) to create members of circuitry in the hardware via variable connections to perform portions of certain operations when operated on. Accordingly, in an example, the machine-readable medium element is part of circuitry or is communicatively coupled to other components of the circuitry when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuitry. For example, in operation, the execution unit may be used in a first circuit of a first circuitry at one time and reused by a second circuit in the first circuitry or by a third circuit in the second circuitry at a different time. The following are additional examples of these components of a computer system.

Fig. 15 is a block diagram illustrating a machine in the example form of a computer system 1500 in which a set or sequence of instructions may be executed to cause the machine to perform any of the methods discussed herein, according to an embodiment. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or it may act as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a vehicle subsystem, a personal computer (personal computer, PC), a tablet PC, a hybrid tablet, a mobile phone or smart phone, or any machine capable of executing instructions (sequentially or otherwise) specifying actions to be taken by the machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. Similarly, the term "processor-based system" shall be taken to include any collection of one or more machines controlled or operated by processor circuitry (e.g., a computer) or other circuitry to execute instructions, individually or jointly, to perform any one or more of the methodologies discussed herein.

Example computer system 1500 includes: at least one hardware processor 1502 (e.g., a central processing unit (central processing unit, CPU), a graphics processing unit (graphics processing unit, GPU), or both, a processor core, a compute node, etc.), a main memory 1504 and a static memory 1506 (e.g., memory or storage for firmware, microcode, basic input-output (BIOS), unified extensible firmware interface (unified extensible firmware interface, UEFI), etc.), which communicate via a link 1530 (e.g., bus or interconnect). The computer system 1500 may further include a display unit 1510, an alphanumeric input device 1512 (e.g., a keyboard), and a User Interface (UI) navigation device 1514 (e.g., a mouse). In one example, the display unit 1510, the input device 1512, and the UI navigation device 1514 are incorporated into a touch screen display. The computer system 1500 may additionally include a storage device 1516 (e.g., a drive unit), a signal generating device 1518 (e.g., a speaker), a network interface device 1520, and one or more sensors (not shown), such as a global positioning system (global positioning system, GPS) sensor, compass, accelerometer, gyroscope, magnetometer, or other sensor. The machine 1500 may include an output controller 1528 for interfacing with or controlling one or more peripheral devices (e.g., printer, card reader, etc.), such as a serial (e.g., universal serial bus (universal serial bus, USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (near field communication), etc.) connection.

The registers of the processor 1502, main memory 1504, static memory 1506, or mass storage 1508 may be or include a machine-readable medium 1522, on which is stored one or more data structures or sets of instructions 1524 (e.g., software) embodying or utilized by any one or more of the methods or functions described herein, or the set of instructions 1524. The instructions 1524 may also reside, completely or at least partially, within any of the registers of the main memory 1504, the static memory 1506, and/or within the processor 1502 during execution thereof by the computer system 1500. In an example, one or any combination of the hardware processor 1502, the main memory 1504, the static memory 1506, and the mass storage device 1508 also constitute a machine-readable medium.

While the machine-readable medium 1522 is illustrated in an example embodiment as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions 1524. The term "machine-readable medium" shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory including, by way of example and not limitation: semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (electrically erasable programmable read-only memory), and flash memory devices); magnetic disks such as internal hard disks and removable disks; magneto-optical disk; CD-ROM and DVD-ROM discs.

The instructions 1524 may be further transmitted or received over the communications network 1526 using a transmission medium via the network interface device 1520 using any one of a number of well-known transmission protocols (e.g., HTTP). Examples of communication networks include: local area networks (local area network, LANs), wide area networks (wide area network, WANs), the internet, mobile telephone networks, plain old telephone (plain old telephone, POTS) networks, and wireless data networks (e.g., bluetooth, wi-Fi, 3G, and 4G LTE/LTE-a, 5G, DSRC, or similar networks). The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

In an example, information stored on or otherwise provided on a machine-readable medium may represent instructions, such as the instructions themselves or the format from which the instructions may be derived. The format from which the instructions may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packages), and so forth. Information representing instructions in a machine-readable medium may be processed by processing circuitry into instructions to implement any of the operations discussed herein. For example, deriving instructions from information (e.g., processing by processing circuitry) may include: compile, interpret, load, organize (e.g., dynamically or statically link), encode, decode, encrypt, decrypt, package, unpack, or otherwise manipulate information into instructions (e.g., from source code, object code, etc.).

In an example, derivation of the instructions may include assembling, compiling, or otherwise interpreting the information (e.g., by processing circuitry) to create the instructions in some intermediate or pre-processed format provided from a machine-readable medium. When the information is provided in multiple parts, it may be combined, unpacked, and modified to create instructions. For example, the information may be in multiple compressed source code packages (or object code, or binary executable code, etc.) on one or several remote servers. The source code packages may be encrypted when transmitted over a network, and may be decrypted, decompressed, (if necessary) assembled (e.g., linked) and compiled or interpreted (e.g., compiled or interpreted as a library, a separate executable file, etc.) at the local machine and executed by the local machine.

It should be appreciated that the functional units or capabilities described in this specification may have been referred to or labeled as components or modules, in order to particularly emphasize their implementation independence. Such components may be embodied in any number of software or hardware forms. For example, a component or module may be implemented as a hardware circuit comprising custom Very Large Scale Integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. The components or modules may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like. The components or modules may also be implemented in software for execution by various types of processors. The identified components or modules may, for instance, comprise one or more physical or logical blocks of computer instructions which may, for instance, be organized as an object, procedure, or function. However, the executable objects of the identified components or modules need not be physically together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the components or modules and achieve the stated purpose for the components or modules.

Indeed, a component or module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices or processing systems. In particular, some aspects of the described processes (such as code rewriting and code analysis) may be performed on a different processing system (e.g., in a computer in a data center) than the processing system in which the code is deployed (e.g., in a computer embedded in a sensor or robot). Similarly, operational data may be identified and illustrated herein within components or modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices. The components or modules may be passive or active, including agents operable to perform desired functions.

In view of the above disclosure, various examples of embodiments are set forth below. It should be noted that one or more features of the examples taken alone or in combination should be considered to be within the scope of the disclosure of the present application.

Example 1 is a method, comprising: obtaining a first autonomous driving scenario for use in an autonomous driving simulation, the first autonomous driving scenario being defined by a first plurality of scenario parameters; identifying a first risk assessment metric for the first autonomous driving scenario based on a safety operation used by the autonomous driving model in the first autonomous driving scenario; determining a second plurality of scene parameters from the changes in the first plurality of scene parameters that produce a second risk assessment metric, wherein the second risk assessment metric indicates greater difficulty for the security operation than the first risk assessment metric; and generating a second autonomous driving scene from the second plurality of scene parameters for use in autonomous driving simulation.

In example 2, the subject matter of example 1 optionally includes testing at least a portion of the first autonomous driving scenario on the autonomous driving simulation platform, wherein the testing of the first autonomous driving scenario applies the security operation to the first autonomous driving scenario, and wherein the first risk assessment metric is based on a result of the testing of the first autonomous driving scenario; and repeating the operations of testing and generating the subsequent autonomous driving scenario during the multiple rounds of testing, thereby using operations that progressively increase respective risk assessment metrics that indicate greater difficulty for safety operations in the subsequent autonomous driving scenario.

In example 3, the subject matter of example 2 optionally includes generating an assessment of the autonomous driving model based on results of the safety operation used by the autonomous driving model in one or more of: the first autonomous driving scenario, the second autonomous driving scenario, or a subsequent autonomous driving scenario.

In example 4, the subject matter of any one or more of examples 2-3 optionally includes wherein the autonomous driving model is one of a plurality of autonomous driving models associated with an autonomous driving or advanced driver assistance system, and wherein the autonomous driving simulation platform is a test simulation computing system including hardware configured to evaluate the plurality of autonomous driving models.

In example 5, the subject matter of any one or more of examples 1-4 optionally includes wherein the first plurality of scene parameters and the second plurality of scene parameters specify characteristics of the road, participants on the road, and trajectories of each of the participants.

In example 6, the subject matter of any one or more of examples 1-5 optionally includes wherein the second plurality of scene parameters used in the second autonomous driving scene specify an initial state and trajectory that is different from an initial state and trajectory specified by the first plurality of scene parameters used in the first autonomous driving scene.

In example 7, the subject matter of any one or more of examples 1-6 optionally includes wherein the first risk assessment metric is associated with a time-to-collision (TTC) calculation, wherein the TTC calculation produces a value within one of a plurality of risk levels, and wherein the second risk assessment metric is associated with a TTC calculation producing a value within a risk level equal to or higher than the first risk assessment metric.

In example 8, the subject matter of any one or more of examples 1-7 optionally includes wherein determining the second plurality of scene parameters includes performing a grid search on scene parameters to be used by the second plurality of scene parameters.

In example 9, the subject matter of any one or more of examples 1-8 optionally includes wherein determining the second plurality of scene parameters includes performing probabilistic modeling on scene parameters to be used by the second plurality of scene parameters to identify the search space.

In example 10, the subject matter of any one or more of examples 1-9 optionally includes operations further comprising: generating a difficulty assessment of the first autonomous driving scenario for maneuvers associated with safety operations performed with testing the first autonomous driving scenario, the difficulty assessment based on a safety area, a road segment range, or a cost function; wherein generating the second autonomous driving scenario and determining the second plurality of scenario parameters are further based on the difficulty assessment.

In example 11, the subject matter of example 10 optionally includes wherein the cost function relates to a time cost and a complexity cost for the maneuver.

In example 12, the subject matter of any one or more of examples 10-11 optionally includes wherein the road segment range relates to waypoints used in the autonomous driving simulation for the maneuver.

Example 13 is at least one machine-readable storage medium comprising instructions that, when executed by processing circuitry of a machine, cause the processing circuitry to perform any of the methods of examples 1-12.

Example 14 is a computing system for autonomous driving test scenario evaluation, comprising: a memory configured to store data of an autonomous driving model; and processing circuitry configured to: identifying a first autonomous driving scenario for use in an autonomous driving simulation, the first autonomous driving scenario defined by a first plurality of scenario parameters; identifying a first risk assessment metric for the first autonomous driving scenario based on a safety operation used by the autonomous driving model in the first autonomous driving scenario; determining a second plurality of scene parameters from the changes in the first plurality of scene parameters that produce a second risk assessment metric, wherein the second risk assessment metric indicates greater difficulty for the security operation than the first risk assessment metric; and generating a second autonomous driving scene from the second plurality of scene parameters for use in autonomous driving simulation.

In example 15, the subject matter of example 14 optionally includes processing circuitry further configured to: testing at least a portion of a first autonomous driving scenario on an autonomous driving simulation platform, wherein the testing of the first autonomous driving scenario applies a safety operation to the first autonomous driving scenario, and wherein the first risk assessment metric is based on a test result of the first autonomous driving scenario; and repeating the operations of testing and generating subsequent scene parameters during the multiple rounds of testing, thereby using operations that progressively increase respective risk assessment metrics that indicate greater difficulty for safety operations in subsequent autonomous driving scenes.

In example 16, the subject matter of example 15 optionally includes processing circuitry further configured to: generating an assessment of the autonomous driving model based on results of safety operations used by the autonomous driving model in one or more of: the first autonomous driving scenario, the second autonomous driving scenario, or a subsequent autonomous driving scenario.

In example 17, the subject matter of any one or more of examples 15-16 optionally includes wherein the autonomous driving model is one of a plurality of autonomous driving models associated with an autonomous driving or advanced driver assistance system, and wherein the autonomous driving simulation platform is a test simulation computing system including hardware configured to evaluate the plurality of autonomous driving models.

In example 18, the subject matter of any one or more of examples 14-17 optionally includes the first plurality of scene parameters and the second plurality of scene parameters specifying characteristics of the road, participants on the road, and trajectories of each of the participants.

In example 19, the subject matter of any one or more of examples 14-18 optionally includes wherein the second plurality of scene parameters used in the second autonomous driving scene specify an initial state and trajectory that is different from an initial state and trajectory specified by the first plurality of scene parameters used in the first autonomous driving scene.

In example 20, the subject matter of any one or more of examples 14-19 optionally includes wherein the first risk assessment metric is associated with a time-to-collision (TTC) calculation, wherein the TTC calculation produces a value within one of a plurality of risk levels, and wherein the second risk assessment metric is associated with a TTC calculation producing a value within a risk level equal to or higher than the first risk assessment metric.

In example 21, the subject matter of any one or more of examples 14-20 optionally includes operations wherein determining the second plurality of scene parameters includes performing a grid search on scene parameters to be used by the second plurality of scene parameters.

In example 22, the subject matter of any one or more of examples 14-21 optionally includes wherein determining the second plurality of scene parameters includes performing probabilistic modeling on scene parameters to be used by the second plurality of scene parameters to identify operations of the search space.

In example 23, the subject matter of any one or more of examples 14-22 optionally includes processing circuitry further configured to: generating a difficulty assessment of the first autonomous driving scenario for maneuvers associated with safety operations performed with testing the first autonomous driving scenario, the difficulty assessment based on a safety area, a road segment range, or a cost function; wherein the operations of generating the second autonomous driving scenario and determining the second plurality of scenario parameters are further based on the difficulty assessment.

In example 24, the subject matter of example 23 optionally includes wherein the cost function relates to a time cost and a complexity cost for the maneuver.

In example 25, the subject matter of any one or more of examples 23-24 optionally includes wherein the road segment range relates to waypoints used in the autonomous driving simulation for the maneuver.

Example 26 is an apparatus, comprising: means for identifying a first autonomous driving scenario for use in a first autonomous driving simulation, the first autonomous driving scenario defined by a first plurality of scenario parameters; means for identifying a first risk assessment metric for the first autonomous driving scenario based on a safety operation used by the autonomous driving model in the first autonomous driving scenario; means for determining a second plurality of scene parameters from a change in the first plurality of scene parameters that produces a second risk assessment metric, wherein the second risk assessment metric indicates greater difficulty for a security operation than the first risk assessment metric; and generating a second autonomous driving scenario from the second plurality of scenario parameters, the second autonomous driving scenario for use in autonomous driving simulation.

In example 27, the subject matter of example 26 optionally includes means for testing at least a portion of the first autonomous driving scenario on the autonomous driving simulation platform, wherein the testing of the first autonomous driving scenario applies the security operation to the first autonomous driving scenario, and wherein the first risk assessment metric is based on a test result of the first autonomous driving scenario; and means for generating an assessment of the autonomous driving model based on a result of the safety operation used by the autonomous driving model.

In example 28, the subject matter of example 27 optionally includes means for repeating the operations of testing and generating the subsequent autonomous driving scenario during the multiple rounds of testing, thereby using operations that progressively increase respective risk assessment metrics that indicate greater difficulty for safety operations in the subsequent autonomous driving scenario.

In example 29, the subject matter of any one or more of examples 26-28 optionally includes means for defining a first plurality of scene parameters specifying characteristics of the road, participants on the road, and trajectories of each of the participants.

In example 30, the subject matter of any one or more of examples 26-29 optionally includes wherein the second plurality of scene parameters used in the second autonomous driving scene specify an initial state and trajectory that is different from an initial state and trajectory specified by the first plurality of scene parameters used in the first autonomous driving scene.

In example 31, the subject matter of any one or more of examples 26-30 optionally includes means for generating a time-to-collision (TTC) calculation, wherein the first risk assessment metric is associated with the calculation, wherein the TTC calculation produces a value within one of a plurality of risk levels, and wherein the second risk assessment metric is associated with the TTC calculation producing a value within a risk level equal to or higher than the first risk assessment metric.

In example 32, the subject matter of any one or more of examples 26-31 optionally includes means for performing a grid search for a scene parameter to be used by the second plurality of scene parameters.

In example 33, the subject matter of any one or more of examples 26-32 optionally includes means for performing probabilistic modeling to identify a search space for the scene parameters to be used by the second plurality of scene parameters.

In example 34, the subject matter of any one or more of examples 26-33 optionally includes means for generating a difficulty assessment of the first autonomous driving scenario for a maneuver associated with a safety operation performed with testing the first autonomous driving scenario, the difficulty assessment based on the safety area, the road segment range, or the cost function; wherein the second autonomous driving scenario and the second plurality of scenario parameters are further based on the difficulty assessment; wherein the cost function relates to time cost and complexity cost for the maneuver; and wherein the road segment range relates to waypoints used in simulation for the maneuver.

In example 35, the subject matter of any one or more of examples 26-34 optionally includes wherein the autonomous driving model is one of a plurality of autonomous driving models associated with the means for performing autonomous driving or the means for performing advanced driver assistance.

While these implementations have been described with reference to specific exemplary aspects, it will be evident that various modifications and changes may be made thereto without departing from the broader scope of the disclosure.

Claims (25)

1. At least one machine-readable storage medium comprising instructions that, when executed by processing circuitry of a machine, cause the processing circuitry to perform operations comprising:

obtaining a first autonomous driving scenario for use in autonomous driving simulation, the first autonomous driving scenario being defined by a first plurality of scenario parameters;

identifying a first risk assessment metric for the first autonomous driving scenario based on a safety operation used by an autonomous driving model in the first autonomous driving scenario;

determining a second plurality of scene parameters from the changes in the first plurality of scene parameters that result in a second risk assessment metric, wherein the second risk assessment metric indicates a greater difficulty for the security operation than the first risk assessment metric; and

A second autonomous driving scene is generated from the second plurality of scene parameters for use in the autonomous driving simulation.

2. The machine-readable storage medium of claim 1, the operations further comprising:

testing at least a portion of the first autonomous driving scenario on an autonomous driving simulation platform, wherein the testing of the first autonomous driving scenario applies the safety operation to the first autonomous driving scenario, and wherein the first risk assessment metric is based on a result of the testing of the first autonomous driving scenario; and

repeating the operations of testing and generating a subsequent autonomous driving scenario during multiple rounds of testing, thereby using operations that progressively increase respective risk assessment metrics that indicate greater difficulty for the safety operations in the subsequent autonomous driving scenario.

3. The machine-readable storage medium of claim 2, the operations further comprising:

generating an assessment of the autonomous driving model based on results of the safety operation used by the autonomous driving model in one or more of: the first autonomous driving scenario, the second autonomous driving scenario, or the subsequent autonomous driving scenario.

4. The machine-readable storage medium of claim 2, wherein the autonomous driving model is one of a plurality of autonomous driving models associated with an autonomous driving or advanced driver assistance system, and wherein the autonomous driving simulation platform is a test simulation computing system comprising hardware configured to evaluate the plurality of autonomous driving models.

5. The machine-readable storage medium of claim 1, wherein the first plurality of scene parameters and the second plurality of scene parameters specify characteristics of a road, participants on the road, and trajectories of each of the participants.

6. The machine-readable storage medium of claim 1, wherein the second plurality of scene parameters used in the second autonomous driving scene specify an initial state and trajectory that is different from an initial state and trajectory specified by the first plurality of scene parameters used in the first autonomous driving scene.

7. The machine-readable storage medium of claim 1, wherein the first risk assessment metric is associated with a time-to-collision (TTC) calculation, wherein the TTC calculation produces a value within one of a plurality of risk levels, and wherein the second risk assessment metric is associated with a TTC calculation producing a value within a risk level equal to or higher than the first risk assessment metric.

8. The machine-readable storage medium of claim 1, wherein determining the second plurality of scene parameters comprises performing a grid search for scene parameters to be used by the second plurality of scene parameters.

9. The machine-readable storage medium of claim 1, wherein determining the second plurality of scene parameters comprises performing probabilistic modeling on scene parameters to be used by the second plurality of scene parameters to identify a search space.

10. The machine-readable storage medium of claim 1, the operations further comprising:

generating a difficulty assessment of the first autonomous driving scenario for maneuvers associated with the safety operation performed with testing the first autonomous driving scenario, the difficulty assessment based on a safety area, a road segment range, or a cost function;

wherein generating the second autonomous driving scenario and determining the second plurality of scenario parameters are further based on the difficulty assessment.

11. The machine-readable storage medium of claim 10, wherein the cost function relates to a time cost and a complexity cost for the maneuver.

12. The machine-readable storage medium of claim 10, wherein the road segment range relates to waypoints used in the autonomous driving simulation for the maneuver.

13. A test simulation computing system comprising computing hardware configured by the machine-readable storage medium of any of claims 1-12.

14. A computing system for autonomous driving test scenario evaluation, comprising:

a memory configured to store data of an autonomous driving model; and

processing circuitry configured to:

identifying a first autonomous driving scenario for use in autonomous driving simulation, the first autonomous driving scenario defined by a first plurality of scenario parameters;

identify a first risk assessment metric for the first autonomous driving scenario based on a safety operation used by the autonomous driving model in the first autonomous driving scenario;

determining a second plurality of scene parameters from a change in the first plurality of scene parameters that produces a second risk assessment metric, wherein the second risk assessment metric indicates a greater difficulty for the security operation than the first risk assessment metric; and

a second autonomous driving scene is generated from the second plurality of scene parameters for use in the autonomous driving simulation.

15. The computing system of claim 14, the processing circuitry further configured to:

Testing at least a portion of the first autonomous driving scenario on an autonomous driving simulation platform, wherein the testing of the first autonomous driving scenario applies the safety operation to the first autonomous driving scenario, and wherein the first risk assessment metric is based on a result of the testing of the first autonomous driving scenario; and

repeating the operations of testing and generating a subsequent autonomous driving scenario during multiple rounds of testing, thereby using operations that progressively increase respective risk assessment metrics that indicate greater difficulty for the safety operations in the subsequent autonomous driving scenario.

16. The computing system of claim 15, the processing circuitry further configured to:

generating an assessment of the autonomous driving model based on results of the safety operation used by the autonomous driving model in one or more of: the first autonomous driving scenario, the second autonomous driving scenario, or the subsequent autonomous driving scenario.

17. The computing system of claim 15, wherein the autonomous driving model is one of a plurality of autonomous driving models associated with an autonomous driving or advanced driver assistance system, and wherein the autonomous driving simulation platform is a test simulation computing system including hardware configured to evaluate the plurality of autonomous driving models.

18. The computing system of claim 14, the first plurality of scene parameters and the second plurality of scene parameters specifying characteristics of a road, participants on the road, and trajectories of each of the participants.

19. The computing system of claim 14, wherein the second plurality of scene parameters used in the second autonomous driving scene specify an initial state and trajectory that is different from an initial state and trajectory specified by the first plurality of scene parameters used in the first autonomous driving scene.

20. The computing system of claim 14, wherein the first risk assessment metric is associated with a time-to-collision (TTC) calculation, wherein the TTC calculation produces a value within one of a plurality of risk levels, and wherein the second risk assessment metric is associated with a TTC calculation producing a value within a risk level equal to or higher than the first risk assessment metric.

21. The computing system of claim 14, wherein determining the second plurality of scene parameters comprises performing a grid search of scene parameters to be used by the second plurality of scene parameters.

22. The computing system of claim 14, wherein determining the second plurality of scene parameters comprises performing probabilistic modeling on scene parameters to be used by the second plurality of scene parameters to identify a search space.

23. The computing system of claim 14, the processing circuitry further configured to:

generating a difficulty assessment of the first autonomous driving scenario for maneuvers associated with the safety operation performed with testing the first autonomous driving scenario, the difficulty assessment based on a safety area, a road segment range, or a cost function;

wherein the operations of generating the second autonomous driving scenario and determining the second plurality of scenario parameters are further based on the difficulty assessment.

24. The computing system of claim 23, wherein the cost function relates to a time cost and a complexity cost for the maneuver.

25. The computing system of claim 23, wherein the road segment range relates to waypoints used in the autonomous driving simulation for the maneuver.