Virtual Risk Assessment for the Deployment of Autonomous Shuttles

In recent years, trials of autonomous shuttle vehicles have been conducted worldwide. Currently, there exists no generalized process model for deployment and continuous operation of shuttles. Shuttle suppliers use their own developed procedures, making it difficult for the relevant stakeholders (e.g., public authorities) to assess the risk of potential shuttle deployment. The Digibus® Austria flagship project, among other goals, develops an approach for the virtual risk assessment of identified critical spots along proposed shuttle paths. Embedded into the deployment process, this serves as a significant body of evidence for safety assurance in shuttle deployment. Conducted simulation studies optimizing the shuttle’s trajectory for concrete maneuvers, along with derived requirements for the associated virtual environment, are part of the first noteworthy outcomes. Concretely, the developed virtual environment is integrated in the framework used for virtual validation. The framework is then used for a detailed evaluation of a right-turn maneuver, analyzing possible shuttle trajectories. Considerable differences in sensor coverage at the shuttle’s stopping point can be shown. Conclusively, by utilizing the shuttle’s restricted operational domain, the proposed virtual risk assessment is considered the first step toward a general procedure for the safety assurance of automated vehicles.

Many different trials of autonomous shuttles have started in recent years (1,2). Although the first steps toward a standardized deployment process for shuttle operation have been taken, there is still the open question of how to accurately assess the potential risk for a shuttle deployment, including associated validation procedures (3). This is an issue since public authorities are lacking those procedures as a basis for their decisions on the allowance of shuttle deployments, followed by continuous operation. What is missing is a structured argument, supported by a body of evidence, providing a compelling and valid case that shuttle operation is safe for the given environment. Generally, this is referred to as a safety case (4). Virtual validation provides significant support in the needed body of evidence. Especially in recent years, the required virtual environments have made a giant leap, enabling the necessary level of detail of the general road network, including surrounding elements (e.g., buildings and vegetation) as well as realistic environmental conditions. Virtual validation is an approach taken by many research projects and initiatives in the industry, aiming for the development of verification and validation (V&V) processes for such automated driving systems (ADS). For instance, automotive industry leaders have published a white paper (5), presenting different test strategies in response to critical challenges for V&V, like a statistical demonstration of system safety. Among other things, they propose a scenario-based testing approach, which includes the use of different test techniques, ranging from virtual testing to proving ground tests. This is partly based on the work carried out by the German research project PEGASUS (6) (https:// www.pegasusprojekt.de), which developed criteria, tools, and methods for the release of highly automated driving functions based on a highway chauffeur use case. Another framework for ADS, referring to testable cases and scenarios, was published by the National Highway Traffic Safety Administration (7) (https://www.nhtsa.gov/), also proposing a mixture of test methods including virtual testing.
Although there is a consensus on the overall approach for the V&V challenges of ADS, the research work carried out so far focuses mostly on scenarios for highway-based use cases, resulting in limited applicability to the maneuver-based operation of autonomous shuttles. Additionally, the operational domain of such shuttles primarily consists of urban environments. The proposed approach in this paper to assess the risk of an autonomous shuttle deployment by adapting the methods developed in the mentioned research projects and initiatives for a maneuver-based urban use case and is, therefore, an effort toward the development of methodologies for one of the critical areas for autonomous vehicle deployment. This is also supported by one of the two successor projects from PEGASUS, VV-Methoden (https://www.vvm-projekt.de/en/), which is also using a junction-based urban environment use case. The project develops methods which should serve as a baseline for future homologation efforts (8). This paper is an effort to define a sophisticated approach for the virtual assessment of potential risks of a shuttle deployment to reduce the chances of hazardous events. The virtual risk assessment is embedded into a deployment process and supports the necessary safety case by providing a body of evidence that the proposed path can be successfully operated by the autonomous shuttle or recommend necessary adaptions to responsible stakeholders. Additionally, the shuttle's existing sensor setup can be evaluated for the planned track, leading to already validated suggestions for risk reduction in the deployment.

Methods
In this section the developed methods are presented. In the beginning, the deployment process, based on Rehrl et al. (3), is discussed in detail, explaining the main phases. This is followed by a detailed explanation of one of the main phases of the deployment process, the risk assessment. This assessment is specifically executed for potential deployments and operations of autonomous shuttles. It should be noted that the insights, not only into the deployment process but also into the risk asssessment, are based on the findings gained from the shuttle trial conducted in the suburban town of Koppl (near Salzburg, Austria) during the execution of the Digibus Ò Austria flagship project. The main part of the methods section deals with an optional part of the risk assessment, the virtual risk assessment, using simulation. This means that virtual testing is used to determine the safety of deployments and continuous operation of autonomous shuttles. In this context, saftey is not interpreted with a pure focus on the driving function integrated in the shuttle, but from a deployment-driven perspective. This means the planned shuttle track is analyzed, including sensor coverage. This could be further enhanced in the future toward a more comprehensive evaluation.

Deployment Process
Currently, there exists no standard process for deployment of autonomous vehicles. Recently, Singapore has announced the opening of more than 600 miles of public roads for conducting tests with autonomous vehicles and is, therefore, one of the public administrations aiming toward standardized trial procedures (9). Singapore's Land Transport Authority has already issued a four-part standard called TR68 (10), which covers key guidelines like basic behavior and safety principles for autonomous vehicles' ADS. As this standard aims at providing comprehensive guidance for deployment and testing of all types of autonomous vehicles on public roads, it can serve as a blueprint for future national guidelines. Although such instructions are a much-needed foundation for safe autonomous vehicle deployment, they typically lack a concrete process model including welldefined deployment and operation activities organized as consecutive steps with decisions and actors. The development of such a process model is one of the goals pursued in Digibus Ò Austria (www.digibus.at), the Austrian flagship project on integrating autonomous shuttles in public transport systems. A consortium of 13 partners, under the coordination of Salzburg Research, develops and evaluates methodologies and technologies for safe deployment and operation of autonomous L3-4 shuttles (11). The proposed process model recommends seven major activities with several sub-activities, being structured in four consecutive phases (3). The start phase includes a preliminary feasibility check as well as an assignment activity. The preparation phase continues with risk assessment and preparation followed by the deployment phase with deployment and validation and approval activities and finally, the operation phase defines the operation activity.

Risk Assessment
According to the previously introduced process model, the risk assessment is part of the preparation phase and one of the defined main activities in the process model for the deployment of autonomous shuttles. Currently, there is no widely accepted standardized approach for conducting such a risk assessment for the deployment of autonomous shuttles. For highly automated vehicles in general there are only developed methods and procedures for the highway (6), with ongoing developments for urban-based use cases (12). The main idea is to describe the open context of traffic as a finite set of scenarios, which are then allocated to methods which differ in the use of virtual elements and models. This ranges from the proving ground all the way to virtual testing. Certain key performance indicators need to be developed to assess the executed test cases and to perform a comprehensive risk assessment (13).
Part of the Digibus Ò Austria project is dedicated to the development of such methodologies for shuttles. It combines currently used approaches of shuttle suppliers as well as national risk assessment standards and is carried out by the project partner AIT (Austrian Institute of Technology). In the first step, the proposed uniform risk assessment methodology splits the intended path of the autonomous shuttle into different areas, based on predefined road network characteristics, as pictured in Figure 1. This figure shows a section of the actual track used for the shuttle trial conducted in the Digibus Ò Austria project. These characteristics distinguish between intersections and non-intersections, as well as pedestrian and bicycle crossings. Each of these categories presents different challenges for an autonomous shuttle and the split into different areas makes it easier to locate critical parts of the proposed shuttle path and identify necessary assessments for risk reduction. The categorization structure presented in Figure 1 makes no claim to be sufficient for analyzing all potential shuttle tracks, even in suburban areas. However, its elements are sufficient for assessing the track of the Digibus Ò Austria shuttle trial.
In the next step of the uniform risk assessment methodology, the risk potential of each identified area is calculated, based on an established method for determining the general risk of road sections (14). This national guideline from the Austrian government categorizes each area into seven different sub-categories, for which the maximum risk potential is calculated. This maximum is determined from the various risk potentials of the elements belonging to the same sub-category. For example, the sub-category ''Road layout and visibility'' includes features like road slope, lane width, and road curvature. Additional essential elements include properties of the proposed stopping points of the shuttle, like the general visibility of the vehicle for other traffic participants. The risk potential of each component is determined using a pre-defined evaluation table, based on the probability and the expected impact of an occurrence, leading to risk potentials between one (lowest risk) and five (highest risk). The highest risk value of all the different elements within a category is chosen as the maximum risk potential of this very same category. In the last step, the total risk is determined as the highest value from the different maximum risk potentials of each group. Categories that do not apply to the assessed area are excluded.
The highest possible risk for the given example area is reached for the zoomed out area in Figure 1, which indicates that additional investigations are necessary, exceeding general risk assessment approaches. The proposed deployment process of the Digibus Ò Austria project suggests conducting detailed simulation studies for the identified critical spots, based on the uniform risk assessment methodology. This should lead to a more sophisticated assessment of the potential risks of an autonomous shuttle deployment, as well as recommendations for the different stakeholders (e.g., shuttle supplier, public authorities) in the deployment process to lower the risks of a potential shuttle trial.

Virtual Risk Assessment Approach
The identified critical spots along the proposed shuttle path are associated with a significantly higher risk to the pursued safe operation of the vehicle and are, therefore, the key for successful deployment. Usually, the interaction with other traffic participants holds the most potential for unsafe behavior as it involves the correct interpretation of those objects. Therefore junctions, an important part of road networks, especially in urban environments, are among the most risk-prone places, not only for autonomous shuttles. Because of the complexity of such junction-based urban use cases, many hazardous events, either known or unknown beforehand, can occur and affect the safety of the vehicle. The goal, therefore, should be to strive for the absence of unreasonable risk from hazards which results from functional insufficiencies of the intended functionality. This is referred to as the safety of the intended functionality (SOTIF) and is defined in ISO/PAS 21448 (15). The proposed SOTIF workflow can be seen in Figure 2. In the first step, unknown hazardous events need to be revealed, moving them from the bottom left to the top left quarter of Figure 2. The second step is about making the correct adjustments so that hazardous events move to the topright quarter, leading to a significant risk reduction.
The operational design domain (ODD) is defined as the operating conditions under which ADS is specifically designed to function, therefore, also reflecting the technological limitations (16). To unveil critical situations in the ODD, scenario-based testing is one of the most promising approaches. A more detailed explanation of how scenario-based testing can be used to analyze the open context, presented by a specific ODD, is given in Neurohr et al. (12). In Ulbrich et al. (17) the term scenario is defined as a temporal sequence of scenes, which covers a specific time span and additionally defines all surrounding elements of the ego vehicle-along a multiple-layer model, initially defined by Schuldt et al. (18)-so that a systematic coverage of the ODD at hand can theoretically be achieved. Additionally, by using a scenario-based approach, the derivation of sophisticated pass/fail criteria for evaluation can be obtained by defining appropriate key performance indicators. Achieving the needed statistical coverage for ADS of SAE level 3 or higher, because of the enormous ODD, can only be done by an intelligent combination between different test methods. There is a consensus that simulation, next to the proving ground, is an essential component in reaching that goal. In the case of an autonomous shuttle, even though it needs to have SAE level 4 for its business model to work, the ODD is nonetheless rather small, as the shuttle drives along a pre-defined path and should fulfill pre-defined maneuvers. An autonomous shuttle use case serves as a perfect starting point for the development of methods for V&V. In the case of risk assessments for the proposed autonomous shuttle path, the specific surrounding environment along the route, especially for the critical spots, is crucial. Consequently, the use of proving ground testing is not the ideal approach, as it is challenging to create such a specific environment for efficient assessment. In a simulation, however, creating matching virtual environments is a difficult task, which needs to be embedded into the deployment process, but it is much more scalable in assessing the identified critical situations and deriving appropriate measures afterward.

Model-Based Testing Framework
To validate the behavior of the autonomous shuttle in the identified critical spots virtually, model-based testing introduces appropriate models to represent the ADS in the necessary detail. Models are always abstractions from the real world and are only accurate to a certain degree. The desired fidelity for each of the introduced models of the testing framework depends on the particular use case at hand and is, therefore, purpose-driven. The elements of the general structure used for the simulation framework can be seen in Figure 3 and have the following meaning: Driving function In the case of an autonomous shuttle, a sequence of waypoints through a road network, forming a path, is pre-defined. The specific driving behavior is chosen in the Behavioral Layer, taking perceived road users, obstacles, and signage into account. This motion specification is an input for the Motion Planning, which seizes the estimated pose, as well as the collision-free space, to calculate a reference path. The Local Feedback Control then determines the actuator commands based on a vehicle state estimation (19). Vehicle dynamics This model acts as the virtual representation of the actual vehicle with regard to the vehicle dynamics and executes the actuator commands from the driving function based on pre-defined interfaces. The model fidelity highly depends on the purpose of the intended simulation study. Sensors Sensor models simulate the interaction between the driving function-related vehicle sensors and the virtual environment. These models are, therefore, a prerequisite for virtual testing of ADS. The actual state of the virtual environment (ground truth) is forwarded to the sensor model, where it is modified based on the sensor model category (20). These are: Ideal sensor models: These models generate objectlist data based on perfect ground truth data from the virtual environment (21). Probabilistic sensor models: These models establish a probabilistic relationship between the sensor output and the ground truth of the virtual environment, for example, by adding statistical failure rates (22). Physical sensor models: These models generate sensor data based on actual physical laws and require appropriate virtual environments, which enable the modeling of physical properties as well as material parameters (23).
The output of these sensor models is then forwarded to the driving function.

Virtual environment
In the virtual environment, all necessary elements for scenario generation, depending on the specific use case and the critical area, need to be represented ( Figure 3). This includes all defined road users in the desired scenario, including their behavior, as well as the ego vehicle (dynamic environment). For the static environment, the road network, as well as the surroundings are necessary. Combined, the static and the dynamic environment represent the required virtual environment. Utilizing this generated environment, especially the dynamic environment, makes it possible to execute specific scenarios. Furthermore, the required input for the sensor models (ground truth) needs to be generated.
Since virtual testing environments are a crucial element for successfully conducting simulation studies to support a general safety case for autonomous vehicles, many different environments are available. In general, the different simulators consist of a graphic (for realistic three-dimensional [3D] rendering) and a physics engine (for realistic movements) (24). An overview of currently relevant virtual testing environments to develop algorithms for self-driving vehicles can be found in Kang et al. (25). The requirements for testing the ADS of vehicles are similar and can be conducted by summarizing the first chapters of that paper. This leads to a list of requirements that the virtual environment needs to fulfill to use simulation for V&V procedures effectively, see Table 1. The most important elements of the road, as well as the surrounding environment, need to be represented accordingly in the virtual environment to make a meaningful contribution toward lowering the risk in the critical area, which was identified via the uniform risk assessment methodology. Additionally, the simulation of different environmental effects, like different weather and lighting conditions, concerning the ODD of the vehicle, is essential, as these are common sources of riskprone situations. For a specific use case, the characteristics of high-risk categories need to be represented in the virtual V&V framework. The values of the important and applicable risk potential categories for the identified areas during the uniform risk assessment methodology can be seen in the far-right column of Table 1, together with the identified requirements for the respective V&V subsystems. It should be noted that these requirements are already derived with the usage for a suburban shuttle use case in mind. Therefore the concrete requirements are specifically relevant for the shuttle trial conducted in the Digibus Ò Austria project. However, the presented V&V subsystems are relevant regardless of the use case.

Virtual Risk Assessment in the Digibus Ò Austria Flagship Project
In the Digibus Ò Austria flagship project, the virtual testing environment CARLA (http://carla.org/) is used (26). It is based on the Unreal Engine, which enables the possibility of creating highly realistic 3D environments. In general, this environment fulfills the identified requirements for the autonomous shuttle use case (Table 1) by providing features like the import of road networks, defined in ASAM OpenDRIVE Ò (27), and scenario descriptions (ASAM OpenSCENARIO Ò ) (28). Additionally, changes to the weather, including the lighting conditions, can be applied. To investigate the risk of a potential shuttle deployment, the necessary models for the simulation framework of Figure 3 are derived. To model the LiDAR sensors of the shuttle, ideal sensor models, using the field of view (FOV) and the range of the respective sensor, are used. Information about objects, including other road users, in the FOV is provided as ground truth data by the simulator. As the autonomous shuttle is driving at a cautious speed (\20 km/h), the vehicle dynamics model is kept simple, only changing simple parameters like the total mass and adding rear-wheel steering, in comparison with the standard vehicle model available in CARLA. The driving function enables the shuttle to follow the path. In case of obstacles on the road, it can either re-plan the trajectory for avoidance or stop. Such a decision is based on the input of the relevant sensors. For the virtual environment, the structure that can be seen in Figure 4 was derived. The static environment includes everything that does not change its position and orientation during the execution of a concrete scenario. This includes, most importantly, the road network, which defines how the road users inserted in the scenario can move, based on the ASAM OpenDRIVE Ò standard. Project partners of the Digibus Ò Austria project have already provided a first section of the proposed path of the autonomous shuttle trial. The road network description was derived from a uniform process chain for the generation of such high definition digital map formats and is also part of the Digibus Ò Austria project. In the future, it is planned to include certain aspects of the surroundings (see Figure  3), like houses and trees, in the chosen digital map format as well (3). This would further enhance the usage of the process chain for virtual validation purposes. The dynamic environment contains everything that changes its location and orientation during the scenario execution, including most importantly traffic participants like pedestrians and road users. The combination of the static and dynamic environments leads to the possibility of scenario descriptions, which is necessary for conducting scenario-based testing.

Results
In the Digibus Ò Austria project, six different driving maneuvers were identified, which entail a potential risk for the planned deployment of an autonomous shuttle along the proposed path. Out of these six maneuvers, two are happening at a junction (turning left and right). This junction represents the relevant environmental and geographical part of the modeled ODD. Furthermore, the shuttle is only operated at daytime and in the absence of rain and snow. The maneuver of turning right on to the main street at the specific junction, relevant for shuttle deployment, was analyzed further with the help of the previously explained virtual testing environment. First, the virtual environment was derived, using the digital map for the road layout, provided by partners of the Digibus Ò Austria project, as a foundation. The surroundings, belonging to the static environment, were derived from measurements provided by the project partner AIT, using a reference measurement system, including different LiDAR sensors and stereo cameras. The virtual environment relevant to the right-turn maneuver can be seen in Figure 4.
The dynamic driving task (DDT), which describes all relevant operational and tactical functions required to operate a vehicle in real traffic, is implemented in the virtual testing framework using the respective lateral and longitudinal controllers as well as elements for maneuver planning (16). For the maneuver analysis, two possible paths, which are contrary to each other, were derived and simulated. The paths can be seen in Figure 5. Case 1 describes a path with an optimal FOV for the autonomous shuttle during the right-turn maneuver. This means the path was optimized for the ideal orientation at the stopping point (right at the entry of the junction), including optimization boundary conditions for the path curvature (respecting the physical limits of the shuttle while turning). Keeping a safe distance to the opposite lane was considered as well. This represents the dashdotted line in Figure 5. The objective of case 2 was to minimize the curvature of the proposed path while keeping safe distances from the roadside and opposite lane. This can be seen as the dashed line in Figure 5. The black solid line represents the original shuttle track, which was manually derived at the beginning of the trial. The background of the figure depicts the underlying road network, which is correctly georeferenced, including the critical junction (gray area), static obstacles (dark gray) to analyze potential occlusions from the infrastructure, as well as lane boundaries (light gray solid line) and parking spaces (gray hatched area). The results of this path optimization can be seen in Table 2. Figure 6 shows a detailed comparison between the two considered optimization cases. This includes the

Discussion
Analyzing the shuttle during just one identified high-risk maneuver shows great optimization potential for the proposed path, but also a conflict of objectives. While the first case provides the best view into the junction, especially of approaching road users, it takes nearly 16 s for the shuttle to clear the marked junction area, as can be seen in Table 2. That is because of the high curvature of the calculated path, resulting in a low rotation rate of the shuttle. In contrast, the second case provides significantly less relevant FOV into the junction but it takes slightly under 10 s. In both cases, the shuttle was accelerating very slowly to maintain a certain level of comfort for the passengers. Even if the shuttle were to accelerate faster, there would still be a time difference between the cases in junction clearance. Additionally, the deviations in the sensor FOV can be observed easily from Figure 6. While in case 2 (minimized curvature) there is additional sensor coverage at the right front of the shuttle (top of Figure 6) not much can be gained from that, considering the occlusion from the static obstacle. However, in case 1 (optimal FOV) the additional sensor coverage turns out to be particularly valuable, as it directly points toward the most relevant area for potential oncoming traffic in case of a right-turn maneuver onto the main road.

Conclusion
By utilizing the restricted ODD of autonomous shuttles, the first step toward a V&V methodology, depending significantly on the use of virtual testing, has been made. The proposed virtual risk assessment approach provides significant decision criteria for the process of deployment of a potential autonomous shuttle operation. In the Digibus Ò Austria flagship project, the developed procedure is used to derive alternative shuttle paths for a specific right-turn maneuver, which is one of the six identified critical spots along the track of the shuttle trial. The right-turn maneuver the shuttle is supposed to execute at this location can, therefore, be adapted based on specific parameters, like the exact sensor setup of the autonomous shuttle. Furthermore, critical parameters, which can act as important key performance indicators in future simulation studies, including junction-based maneuvers, are identified and used as risk assessment criteria. This serves as valuable feedback for the whole deployment process and addresses the needs of several stakeholders involved in the process. For public authorities, the simulation studies of critical spots serve as a foundation for decision making concerning the approval of deployments, especially if specific changes of the surrounding environment (e.g., adding traffic lights or changing the speed limit) are solely carried out for enabling shuttle operation. These changes can be validated using the proposed virtual risk assessment approach. The shuttle supplier gains important feedback from the analyzed maneuvers, which could be fed into the development process of future shuttle generations, concerning both software (ADS) and hardware adaptions (sensor setup).

Outlook
In the future, it is planned to analyze all the identified critical maneuvers the shuttle encounters during the proposed path of the trial. For this purpose, the maneuverbased concept, helping to determine the critical spots along the route, will be extended toward a scenario-based concept. Such an enhancement enables even more efficient use of simulation studies to reduce the risk of certain maneuvers by aiming at statistical significance in the coverage of the restricted shuttle ODD. Using the layer Figure 6. Comparison of sensor coverage of the autonomous shuttle for two optimized tracks during a right-turn maneuver.
model for scenario-based testing, a methodical approach for potential changes in the static environment along the shuttle path, which would lead to reduced risk, is also possible by combining the methods for the creation of virtual environments with the scenario-based concepts. Furthermore, the use of real-world data from controlled shuttle test runs along the proposed path serve as an essential source for quantifying and arguing model accuracy. This is mainly critical for the sensor models, as the vehicle dynamics model will remain at low fidelity, considering the shuttle motion.

Author Contributions
The authors confirm contribution to the paper as follows: study conception and design: P. Weissensteiner, G. Stettinger; data collection: P. Weissensteiner, K. Tieber, K. Rehrl; analysis and interpretation of results: P. Weissensteiner; draft manuscript preparation: P. Weissensteiner. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The lead project Digibus Ò Austria is funded by the Austrian Federal Ministry of Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) as part of the ''Mobility of the Future'' program. The publication was written at Virtual Vehicle Research GmbH in Graz and partially funded within the COMET K2 Competence Centers for Excellent Technologies from the Austrian Federal Ministry for Climate Action (BMK), the Austrian Federal Ministry for Digital and Economic Affairs (BMDW), the Province of Styria (Dept. 12), and the Styrian Business Promotion Agency (SFG). The Austrian Research Promotion Agency (FFG) has been authorized for the program management.