Reducing drone incidents by incorporating human factors in the drone and drone pilot accreditation process

. Considering the ever-increasing use of drones in a plentitude of application areas, the risk is that also an ever-increasing number of drone incidents would be observed. Research has shown that a large majority of all incidents with drones is due not to technological, but to human error. An advanced risk-reduction methodology, focusing on the human element, is thus required in order to allow for the safe use of drones. In this paper, we therefore introduce a novel concept to provide a qualitative and quantitative assessment of the performance of the drone operator. The proposed methodology is based on one hand upon the development of standardized test methodologies and on the other hand on human performance modeling of the drone operators in a highly realistic simulation environment.


Introduction, Motivation & Scope of Work
The number of small drones and drone operations is expanding and proliferating tremendously. However, there is a problem. Drones crash. Often [1]. When they do, international studies [2] show that around 80% of the drone crashes can be related to human factors. Combining these two facts, it is clear that -if we want to avoid a massive number of drone incidents in the future -it is required to develop a strategy to incorporate human factors in the drone deployment process and the training of drone pilots. Pilots for regular aircraft or for larger (typically military) drones generally follow extensive simulator training before engaging in any real flight. However, for small rotorcraft, this is much less the case, because it is very difficult to convey a realistic representation to the human sensory system. Both for fixed wing and rotary wing drones, the main problem with current simulator-based pilot training programs is that they are limited to simplistic scenarios (typically flying predefined patterns and practicing take-off and landing operations), without providing much qualitative feedback to the trainee or the supervising entity.
In response to these identified shortcomings, we are developing a drone operator performance assessment tool, which uses a realistic environment and realistic opera-tional conditions to measure the performance of the drone operator, both in a qualitative and quantitative manner. These metrics can then be used by training responsibles to adapt / adjust the theoretical and practical training courses for drone pilots, such that the curriculum (both the practical and the theoretical courses) can be iteratively optimized to best fit the needs.
An important aspect of any qualification assessment procedure is the definition of the test methodologies and of the test scenarios. Within the subject of drone pilot training, these test scenarios are currently most often very limited to simple take-off & landing operations and of following simple patterns in the air. For pilots working in the security sector (military, police, firefighters, civil protection, ...) in tough operating conditions, these highly simplistic scenarios are hardly relevant. Therefore, we also propose a set of standard test methods specifically geared towards the training of drone operators in the security sector.

Previous Work & Main Contributions
Drone operator human performance models have been first developed by the US Air Force in [3], focusing on operations with large military drones, navigated by a crew. These military-oriented operator performance modeling approaches tend to focus on operator workload analysis for optimizing the crew composition, which is less relevant for micro-UAS systems. Bertuccelli et al. [4] proposed a new formulation for a single operator performing a search mission with multiple drones in a timeconstrained environment. Wu et al. [5] expanded on this idea by proposing a multioperator multi-drone operator model. The main criticism with respect to these approaches is that they focus heavily on aspects such as attention and fatigue modeling and neglect other aspects that are paramount for operations in the security sector such as mission stress, enemy countermeasures, varying operator skill levels, etc.
In response to these identified shortcomings, we propose a holistic drone operator performance model, targeted towards drone operators in the security sector, taking into consideration identified parameters that are critical towards these end users.
Efforts to integrate drones into standard operating procedures and into the operational toolbox of security operatives would benefit from quantitative evaluations of individual aircraft capabilities and associated remote pilot proficiencies. The National Institute of Standards and Technology (NIST) is leading an international effort to develop the measurements and standards infrastructure necessary to quantitatively evaluate such aircraft and pilots in the framework of urban search and rescue (USAR) operations [6]. The resulting standard test methods enable any user to generate statistically-significant performance data to evaluate airworthiness, maneuvering, sensing, payload functionality, etc. While extremely valuable, these standard test methodologies developed by NIST are heavily focused on USAR operations and not generically useable for all type of security operations.
Therefore, we propose a set of standardized test methodologies for security operations, based upon the existing NIST framework for USAR operations.

Conceptual overview of the methodology
In order to assess the relationship between human factors and the human operator performance, we followed a user-centered design [7] to come to the methodology which is graphically depicted on Figure 1 and which can be summarized as follows: 1. Identify which human factors could potentially impact the performance of drone pilots, via a set of interviews with experienced drone operators. From this set of interviews, the following human factors were discerned as most important: It should be noticed that these scores were given by expert operatives. We inquired for many more potentially influencing factors and some (e.g. distraction) scored suspiciously low, so they did not make it to the list of important factors of Table 1. Notwithstanding this, we also test against these factors in the evaluation process. Each of these identified parameters is re-identified with the test subjects (drone pilots) during an intake questionnaire to assess the state of the pilot when she or he starts the simulation exercise. 3. Development of a simulation environment for complex drone operations. Driven by the innovation in the field of game development and the increasing graphical processing power of computers, current simulator engines provide a very realistic environmental representation, and the integration with virtual and augmented reality systems allows to increase the level of realism even further. All this means that the visual quality provided by the existing engines is generally very high. However, they also have some important disadvantages, as most existing simulator engines are closed solutions and thus provide no possibility to integrate added functionalities. Therefore, we use the Microsoft AirSim simulation engine [8], which is an open-source simulator for RPAS built on the Unreal Engine. This simulation environment is completely open and customizable, which enables us to incorporate the standard test scenarios, multiple customizable drones and to quantitatively measure the performance of the pilots on-line while executing the mission. Next to this interoceptive sensing of the human physiological state, we also plan to use (in a later stage) exteroceptive sensing of the human physiological state by a camera system targeted at the pilot, estimating fatigue etc. We make use of a human-machine interface with a curved monitor, and not of a virtual reality interface, as may be expected. While the simulation engine supports virtual reality and we have the equipment available, we have especially opted for not making use of a virtual reality interface for two reasons: 1. We want to avoid measuring the side-effects of virtual embodiment, where some pilots may be subject to. 2. Virtual reality would obstruct the use of exteroceptive sensing tools for measuring the physiological state of the pilot during the test. At this moment, we work on two scenarios within the simulation: • Stealthy detection and observation of enemy forces in a rural environment • Management of a hostage situation in an urban environment In each of these scenarios, the pilots are confronted with large-scale dynamic environments, changing weather conditions and time pressure in order to deliver quality data in a minimal amount of time, which are all factors that can induce human errors that can dramatically impact the performance. 4. After completing the mission, the test subjects will be asked to fill in another questionnaire in order to assess their physiological state, as well as to assess any differences with respect to the moment of performing the intake survey.
5. At the end of this procedure, this means that we have the following data at our disposal: a. Human factors & human physiological state prior to beginning the mission (through the intake questionnaire) b. Human factors & human physiological state during the mission (through the exteroceptive sensing, though this is still under development) c. Human factors & human physiological state after completing the mission (through the outtake questionnaire) d. Human performance data as quantitatively measured using the simulation engine (interoceptive sensing), which is directly usable for pilot performance assessment. Given enough test subjects, this enables us to set up a mathematical model between on one hand the human factors and the human physiological state and on the other hand the human performance. This model enables us to predict human performance given a certain input state. In the next section, we will explain how such a model can also be effectively used in the drone pilot accreditation process and the drone certification process.

Incorporation of the human performance model in the drone and drone pilot accreditation process
In manned aviation, there exist extremely strict procedures for pilot accreditation and aircraft airworthiness certification. For small unmanned aircraft, however, the rules are less tight and also less harmonized globally. In the European Union, a risk-based approach [9] is followed, where tighter and tighter rules are imposed (both for the pilot license as for the aircraft airworthiness assessment) with increasing risk associated to the drone operation to be performed. A crucial point is thus to assess the risk to a drone operation, which is dependent of the scenarios that are going to be performed and that are written down in the operational handbook. Therefore, a set of standard scenarios are defined and in order to get a permission to fly, the performance of drone pilots and drones for a specific scenario needs to be assessed. This concept of operation for the accreditation has two pitfalls that our work tries to address: 1. The drone pilot accreditation process happens once, once a year or once every few years. However, we know that a varying physiological state of the pilot on the date of the flight may impact the performance drastically. Using our human performance model, we can predict -given a certain physiological input state -what would be the flight performance of the human operator. As such, a much more fine-grained case-based accreditation is possible, which is specifically useful for stressful operations, such as is often the case in the security sector. 2. The drone accreditation process is not pilot-agnostic. Indeed, when a new drone is tested in a real or simulation environment, it is controlled by a human pilot. Each pilot is of course different and the performance of each pilot will differ, which will have its impact on the evaluation of the drone under investigation. Using our human performance model, we can create a generic computer pilot that is able to perform a flight operation to test a drone system without any influence of a human pilot. This would provide a much fairer assessment for the accreditation of drones and a valuable extra metric to be taken into consideration for the airworthiness assessment.

Conclusions & Future Work
In this paper, we have presented a methodology for the qualitative and quantitative assessment of drone pilots, which can then be used as a tool for improving the training curriculum for these drone pilots. The methodology is based upon a virtual training environment and a set of standard test methods. Importantly, the proposed methodology also enables the development of a human performance model, interlinking the human factors and physiological state on one hand and the human performance on the other hand. This is a crucial tool, as it would not only teach us the relationship between these parameters, but it would in a later stage also support completely pilotagnostic qualitative [10] and quantitative [11] evaluation of drones and drone pilots.