Benchmarks for Aerial Manipulation

This letter is devoted to benchmarks for aerial manipulation robots (drones equipped with robotic arms), which are demonstrating their potential to conduct tasks involving physical interactions with objects or the environment in high altitude workspaces, being a cost effective solution for example in inspection and maintenance operations. Thus, the letter deals with different methods and criteria to evaluate and compare the performance of aerial manipulators. This is not an easy task, taking into account the wide variety of designs, morphologies and implementations that can be found in recent works. In order to cope with this problem, this letter analyzes the capabilities and functionalities of several aerial manipulation prototypes (aerial platform + manipulator), identifying a set of relevant metrics and criteria. A number of benchmarks are defined to evaluate the performance of the aerial manipulator in terms of accuracy, execution time, manipulation capability, or impact response. Experimental results carried out with a compliant joint aerial manipulator in test-bench and in indoor-outdoor testbeds illustrate some of the benchmarks.


I. INTRODUCTION
A ERIAL manipulation is a recent robotics field that proposes the integration of lightweight robotic arms in aerial platforms like multirotors [1]- [5] or autonomous helicopters [6], [7], allowing the realization of different operations in high altitude workspaces that cannot be easily accessed by ground, thus reducing the time, cost, and risk for the human workers. Some illustrative application examples include the inspection and maintenance of pipe structures in chemical plants [8], the inspection by contact using ultrasonic devices to measure the thickness of pipes and tanks [9], [10], the installation [11] and retrieval [12] of sensors, or insulation of cracks [13]. Several prototypes of aerial manipulation robots have been developed and validated both in indoors [1], [4], [5], [10], [11], [14], [16] and outdoors [2], [9], [11], [12], [17], [18], demonstrating the possibility to conduct tasks like grasping [1], [3], [6], [12], visual servoing [15], accurate position control with wrench estimation [5], planning and trajectory control [10], or contact force control [11], [20], [21]. Note that, unlike a fixed base manipulator, the  physical interactions with the environment in an aerial robot [16] should be supported by the multirotor platform in such a way that the performance of the manipulator is not affected [22], [23]. However, since most of these robotic manipulators are designed for a particular platform or task, it is difficult to evaluate and compare their performance given the variety of morphologies and implementations that can be found [24], [25]. This problem motivates the interest in the definition of specific benchmarks that facilitate the evaluation of different aerial manipulation robots, proposing a set of methods and metrics that can be easily reproducible and that are illustrative. Based on previous work on benchmarking for manipulation [26], [27], mobile robots [28], and mobile manipulation [29], this letter is focused on the definition of benchmarks for aerial robots equipped with lightweight and compliant robotic arms capable to interact with objects or the environment on flight.
In our previous work we developed several prototypes of aerial manipulation robots with stiff [18] and compliant [12], [20] joints, conducting test-bench and outdoor flight tests to evaluate the positioning accuracy and repeatability of the end effector [18], analyze the effect of the dynamic coupling over a standard autopilot, or demonstrate the feasibility to carry out bimanual grasping tasks [12]. Later we introduced the long reach aerial manipulators with dual arm [17], motivated by the necessity to improve safety during the operation on flight with close obstacles. The benefits of mechanical compliance were evidenced in those tasks involving physical interaction with the environment, allowing the estimation and control of the force/torque/impedance [21] and preventing that the actuators are damaged due to impacts or overloads [12], [20].
The main contribution of this letter is the definition of a set of benchmarks for aerial robotic manipulation, motivated by the convenience to facilitate the evaluation and comparison in the performance of different prototypes in terms of time metrics, positioning accuracy and repeatability, control errors in grasping/contact tasks, or response to physical interactions. The functionalities and capabilities of the aerial robots are analyzed in the first place from recent works in multirotor control and aerial manipulation, identifying some evaluation criteria and the features of interest. Two groups of benchmarks are then proposed, evaluating the performance of the manipulator in test-bench and in indoor/outdoor flight tests. Some of these tests are demonstrated experimentally with two prototypes of compliant aerial manipulation robots.
The rest of the letter is organized in the following way. Section II identifies the capabilities and functionalities of the aerial manipulation robots based on recent works in the field. Section III describes the benchmarks intended to evaluate the performance of the robotic manipulators in fixed based testbench as a preliminary step before their integration in an aerial platform, whereas Section IV presents a set of benchmarks for the aerial manipulation robots, considering indoor and outdoor flight tests. Section V discusses how to extend the application of the proposed benchmarks to different morphologies and hardware implementations, also describing the testbeds where the experiments will be conducted. Section VI illustrates some of the proposed benchmarks, applied over two prototypes of lightweight and compliant aerial manipulators with single and dual arms, summarizing the conclusions in Section VII.

A. Literature Review on Multirotor Design and Control
Unlike most industrial manipulators that are firmly attached to a fixed base, the performance of an aerial manipulation robot strongly depends on the ability of the aerial platform to support the reaction wrenches and the external forces exerted over the manipulator on flight, in such a way that its operation is not significantly affected by undesired position deviations and the stability of the system is not compromised. Roughly speaking, the aerial platform operates in two different modes during the execution of a task: 1) trajectory/position control for positioning the manipulator within the workspace, like in a grasping task [1], [3], [12], [15], and 2) force/impedance control during the interaction with the objects or the environment, for example in the installation of sensor devices [11] or in other physical interactions with the environment [9], [10], [14], [21].
Traditionally, most aerial manipulators are developed over multirotor platforms with coplanar rotors [2], [5], [14]- [16] for efficiency reasons, that is, for maximizing the flight time. According to the equations of motion derived for a quadrotor [3], [4], this feature makes the system under-actuated in such a way that only the height and orientation (roll, pitch, and yaw) can be directly controlled through the rotor thrust, whereas the translational dynamics (XY) is coupled with the rotational dynamics. This implies that the position of the aerial robot has to be controlled indirectly through the attitude, which is not convenient for the accurate positioning of the end effector. In any case, it is still possible to estimate and control the external forces [22], [23] and moments [24], [25] acting over a multirotor on flight.
Tilted rotor hexarotors, also known as fully actuated aerial platforms, [30], [31], [32], enhance the maneuverability and the controllability with respect to coplanar multirotors, allowing the translation of the platform without changing the attitude, or the application of 6-DOF (degrees of freedom) wrenches decoupling the force and torque. This kind of platforms results especially suitable for aerial manipulation as they contribute to increase the positioning accuracy and control capabilities. However, the efficiency of this configuration is lower, and therefore the flight time decreases.

B. Lightweight and Compliant Arms for Aerial Manipulation
Due to the strong constraints imposed by the aerial platform in terms of payload, flight time, or available space, the usual approach in the development of an aerial manipulation robot consists of integrating a customized lightweight robotic arm built with servo actuators, typically Dynamixel [2], [5], [15] or Herkulex [12], [18]. The integration of "lightweight" industrial manipulators (∼15 kg) would require bigger (high scale) aerial platforms as the autonomous helicopter used in [7]. Besides the usual upper arm-forearm morphology implemented in the indicated works, it is possible to find other configurations like  [13], [14], or linear actuators [11]. Table I compares the features of these prototypes, paying attention to the weight, lift load capacity, reach, or number of joints.
The performance of the prototypes listed above is evaluated through different experiments. The positioning accuracy and repeatability of the dual arm in [18] is measured with a visual marker attached at the end effector. Ref. [20], [21] evaluate the force control and collision detection capabilities in test-bench, whereas [11], [14] demonstrate the application of contact forces on flight, and [15] shows a grasping task with visual servoing.
Mechanical compliance extends the functionalities and capabilities of lightweight robotic arms built with smart servos like Herkulex or Dynamixel (conventional RC servos do not provide any feedback), allowing the estimation and control of the torques and forces from the deflection measurement of an elastic element introduced between the servo shaft and output link [16], [20], [21]. The spring-lever transmission mechanism developed in [12] is a particular implementation of the series elastic actuators described in [33], in which the force control is reduced to a position control problem in the joint [20] or in the Cartesian [21], [34] space. The ability of springs and other elastic elements to absorb the energy of impacts or overloads in a passive way and at higher rates than the actuator provides also contributes to increase significantly the safety, reliability and lifespan of the aerial robot during its operation on flight.

A. Preliminary Considerations
The robotic manipulators typically integrated in multirotors are extremely light and low inertia due to the severe payload limitations and the dynamic coupling with the floating base. In order to simplify the design and development phases, many prototypes are built with smart servo actuators like Dynamixel [1], [2], [15] or Herkulex [12], [18], whose performance is lower compared w.r.t. industrial manipulators built with Harmonic Drive gears and accurate force-torque sensors. Therefore, it is necessary to take into account the technological limitations in the definition of benchmarks for lightweight manipulators. This motivated the introduction of mechanical compliance with deflection measurement [12], [21], allowing the estimation and control of the forces and torques in the manipulator.
According to the capabilities and functionalities identified in Section II, the performance of a lightweight and compliant manipulator intended for aerial applications can be evaluated by means of five features of interest: positioning / trajectory accuracy or repeatability, payload capacity, force control, and    Table II, and detailed in next subsections.
For clarity and simplicity reasons, the benchmarks described below are illustrated considering a manipulator with the usual two-link morphology (upper arm -forearm), denoting by L 1 and L 2 the link lengths, and L = L 1 + L 2 the maximum reach. The Cartesian position of the tool center point (TCP) will be expressed in the reference frame of the manipulator, X 0 Y 0 Z 0 , with the X 0 axis pointing in the forward direction. The metrics used to evaluate the system performance are defined in such a way that the score is invariant with the particular features of the robot (size, weight, DOF's, or kinematic configuration). Section V extends the application of the proposed benchmarks to other morphologies and hardware implementations.

B. Accuracy and Repeatability Benchmarks
These two features are evaluated in a similar way, drawing a circular pattern or a group of marks (dots) with a pen attached to the end effector of the manipulator over a panel placed in front of it, as Fig. 1 illustrates. The trajectory accuracy is evaluated in the first place comparing the circle drawn w.r.t. a printed circle used as reference, considering the three possible planes: X 0 Y 0 , X 0 Z 0 , and Y 0 Z 0 . The method is similar to the one described in the norm ISO 9283:2003. Since the effective workspace of the manipulator is usually constrained by the landing gear of the multirotor and the joint limits [12], [18], the amplitude of the reference circle should be in the range L/4 ≤ A ≤ L/2, being L the reach of the manipulator. The reference trajectory of the TCP in any of the three planes (for example in the Y 0 Z 0 plane) can be computed in the following way: where T is the time required to complete the circumference. Imposing that this is drawn moving the TCP at the 50% of its maximum speed v max T CP , the period is given by T = 2πA/v max T CP . For accurate data representation, the sample time should be below 20 ms (50 Hz update rate). The manipulator must draw N = 10 consecutive circles so the results are reliable enough.
The trajectory accuracy AT is defined here as the maximum deviation of the drawn circle w.r.t. the reference circle. This value can be obtained easily by visual inspection or from the reference and current Cartesian position of the TCP, r T CP : The relative trajectory accuracy ρ AT is a dimensionless index that may be useful for comparing the accuracy of different manipulators independently of their size.
The positioning accuracy and repeatability can be evaluated measuring the maximum deviation of a set of N = 10 marks drawn around at the left and right points of a reference line of length L/2, as Fig. 1 depicts. The TCP of the manipulator, initially located at the midpoint of the line, will jump from one point to the other at the 50% of the maximum speed, leaving a mark on each iteration. The left/right position references will be denoted by r L ref and r R ref .
Considering the Y 0 Z 0 plane: According to the ISO 9283, the positioning accuracy AP is defined as the difference between the mean positions reach by the TCP and the reference position, whereas the repeatability RP represents the dispersion of the reached positions. That is: Here r LR T CP,n denotes the position marked by the TCP around the left/right reference points at the n-th iteration. The indices ρ AP and ρ RP are introduced again to facilitate the comparison of manipulators of different sizes. Note that the repeatability RP represents the radius of the circle containing all the marks, as depicted on the right side of Fig. 1.

C. Lift Load Benchmark
In many works, the lift load (payload) capacity of a robotic arm is not properly defined since it is not clear if it is referred with respect to the elbow or the shoulder joint, considering the typical upper arm-forearm morphology. Note that most of the manufacturers of servo actuators only specify the stall torque parameter, which is not reliable as it benefits from the friction of the gearbox. In practice, the maximum dynamic torque that a Herkulex or Dynamixel servo may deliver is 2 or 3 times lower than the stall torque. A simple way to demonstrate the payload capacity consists of lifting a payload mass attached at the end effector following the procedure depicted in Fig. 2. For this purpose, the manipulator generates a sequence of joint rotations at fixed time intervals of Δt = 5 seconds: The metrics of interest in this case are the payload mass at elbow m P L elbow and shoulder m P L shoulder , the payload-to-weight ratio (Eq. (6)), the torque or PWM signal of the servos, and the play time P T , that is, the time needed to lift the load. In order to facilitate the comparison, it is convenient to consider common objects as payloads, as 33 ml cans (370 g weight).
ρ P L w = min m P L elbow , m P L shoulder /w manipulator (6)

D. Force/Torque Estimation and Control Benchmarks
Before its integration and application on flight, it is highly necessary to evaluate the force/torque control capability of the manipulator in test-bench [20], [21] in terms of accuracy, setup time, and overshoot. The method consists of applying a step force/torque reference and compare the estimation provided by the controller w.r.t. a calibrated sensor: an F/T sensor [11], load cell, bench scale or calibrated payload mass [21]. Since the torque-to-force transmission depends on the inverse of the Jacobian [21], [34], it is convenient to consider the L-shaped nominal configuration shown in Fig. 3, so the contribution of the actuators can be identified more easily in the Cartesian components of the resulting force: The pushing/pulling force reference is a step signal from 0 N to the 50% of the payload mass applied during T = 5 s, that is: (8) where g is the gravity constant. The metrics of interests are indicated in Fig. 3: the time required to reach the 90% of the final value t 90% , and the overshoot in % w.r.t. the reference.

E. Collision Detection and Reaction Benchmark
The end effector of an aerial manipulator operating on flight in contact with the environment will be frequently exposed to impacts and collisions that may compromise the stability of the aerial platform [11], [16], [21]. Therefore, it is convenient to evaluate the ability of the manipulator to detect and react against obstacles [20]. This can be done modifying slightly the experimental setup depicted in Fig. 3 in such a way that the TCP hits the force sensor at the 50% of its maximum speed, v max T CP , with a displacement equal to L/2 w.r.t. the initial pose. A force detection threshold equal to the 10% of the elbow payload mass (F th x = 0.1 · g · m P L elbow ) will be used to trigger the recoil movement of the manipulator at speed v max T CP . The metrics of interest in this case are the maximum force detected by the sensor and the elapsed time until the force is below F th x .

A. Preliminary Considerations
This group of experiments is focused on evaluating the performance of the aerial manipulation robots in both indoor and outdoor testbeds. The benchmarks correspond to tasks to be executed on flight (grasp an object, apply a pushing force, or lift a load), defining conveniently the conditions of the tests to make them easily reproducible. Since most of the works in the field employ multi-rotors, the diagrams and explanations provided below consider this kind of aerial platform, although helicopters [6], [7], fully actuated hexarotors [31], [32], or any UAV (unmanned aerial vehicle) capable to hover can be used.
In the following, the position of the aerial platform will be referred to the Earth fixed frame {E} (inertial), and denoted by r UAV ∈ 3 . The position control error ε UAV is taken as metric of interest, defining the equivalent dimensionless index ρ UAV : The accuracy of the UAV position measurement should be below the 10% of the reach of the manipulator, L, so the task can be carried out within the workspace in a reliable way.

B. Object Grasping Benchmark
One of the basic tasks that an aerial manipulator is expected to conduct is object grasping [1], [12], [15], considered here as a benchmark for evaluating the performance of the robot. This involves the positioning system (Vicon, visual SLAM, GPS, LIDAR, etc.), the position control of the multirotor platform, the perception system for detecting and localizing the object, and a grasping method. As Fig. 4 illustrates, the goal is to grab an object located in a tool bench at distance d goal and height h goal w.r.t. the take-off position, following the five references   Table IV, imposing that ε UAV < 0.25 · L for each way-point. The UAV takes-off at a height h 1 , moves towards the tool bench, retrieves the object with a gripper at height h 2 , and moves up to the initial height h 1 . The grasping maneuver at phase 4 should be carried out in such a way that the TCP of the manipulator advances at least the 25% of its reach L (otherwise the robotic manipulator could be replaced by a gripper, being useless). The weight of the tool should be above the 25% of the elbow payload mass m P L elbow , defined in Section III-C. The grasping height h 2 depends on h goal and the position of the manipulator. The parameters of the test are defined in Table V as function of the multirotor size, S UAV .
The grasping test should be repeated N = 5 times in order to evaluate the reliability of the system. The metrics of interest include the success ratio, the time required to complete each of the phases in Table IV, and the maximum deviation of the multirotor during the grabbing phase, ε UAV grab max and ρ grab UAV .

C. Contact Force Control Benchmark
The possibility to exert contact forces on flight results of interest in applications like sensor installation [11], inspection by contact [9], [10], or cleaning [21], where the ability of the multirotor controller to support the reaction wrenches while maintaining its position stable is the critical point. In order to evaluate this capability, the proposed benchmark test (Fig. 5) consists of applying a pushing force in the X-axis against a wall with the end effector of the manipulator for at least T = 5 seconds, taking the parameters from Section III-D. The aerial platform will approach at low speed to the contact point where the force sensor is placed at height h 1 (see Table V). If this is not available, the internal force estimation can be used [21]. The collision detection benchmark described in Section III-E will be applied to enable the force controller once the contact is detected. The metrics are the UAV position deviation given in  Equation (9), taking as reference the multirotor position at the moment of contact, and the ones indicated in Section III-D.

D. Benchmark for Multirotor Position Control
According to the dynamic model of the aerial manipulator [5], [18], the reaction wrenches caused by the motion of the arm will induce an undesired oscillation in the orientation of the multirotor and, consequently, a deviation in its position. As in [5], [18], the performance of the multirotor position controller can be evaluated in terms of positioning accuracy with the benchmark illustrated in Fig. 6. The manipulator generates the sequence of rotations indicated in Fig. 2 while the UAV tries to stay in a fixed position. The metrics in this case are the maximum position deviation defined in Equation (9), ε UAV , and the elapsed time until ε UAV ≤ 0.1 · L, that is, t 10% .

A. Extended Application to Other Morphologies
For clarity reasons, the benchmarks presented before were illustrated considering a revolute joint manipulator with the usual upper arm-forearm morphology. This can be found in references [1], [2], [5], [7], [10], [12], [18]- [21], [24]. However, the proposed tests can be adapted to other morphologies as long as the evaluation methodology and the associated metrics are clearly defined. Fig. 7 illustrates three different designs: the single link arm with end effector employed in [4], [9], [16], the aerial manipulator with linear actuator described in [11], or the delta manipulator in [13], [14]. The performance of these prototypes can be evaluated with the benchmarks described in Sections III and IV with minor modifications. For example, in the force control benchmark (IV-C), the actuation of the single link and the linear actuator will be constrained to the X-axis. The lift load capacity (III-C) of the linear actuator or the delta manipulator can be evaluated moving a payload in the vertical axis. The trajectory accuracy test (III-B) is applicable with the 2-DOF delta manipulator in one of the three Cartesian planes, whereas the positioning accuracy and repeatability of the other two manipulators can be determined only on a single axis.

B. Testbeds
The proposed benchmark tests are intended to evaluate the performance of an aerial manipulation robot in three different scenarios, illustrated in Fig. 8: a fixed base test-bench built with Rexroth bars, an indoor testbed equipped with a highly accurate positioning system (Vicon, or Opti-Track), and an outdoor testbed covered by a safety net. A detailed description of the testbeds and the conditions of the tests (time and date, temperature, wind) should be reported along with the results. The main features of the methods, algorithms, or devices used to measure the metrics of interest should be also specified.

A. Fixed-Base Benchmark Tests
The trajectory accuracy of a prototype of lightweight and compliant arm is evaluated following the method described in Section III-B. The specifications of the arm are given in [36]. The test consisted of drawing a 180 mm Ø circle over a panel in the Y 0 Z 0 plane with a red pen attached at the tip of the end effector, using a printed black circle as ground truth. Fig. 9 illustrates the experimental setup (right), the circle drawn (left-down), and the position of the tool center point obtained from the forward kinematic model (left-up). The position deviations in Fig. 9 reveal systematic control errors as the trajectory tracking method does not take into account the joint deflection. The arm executed N = 4 turns so the repeatability can be also observed. The parameters and metrics of interests defined in Equations (1) and (2) were:   The lift load capacity was evaluated attaching a 33 ml can (370 grams weight) at the tip of the forearm link, executing the sequence of rotations indicated in Fig. 2 with an interval of 5 s and 1 s play time. In order to compensate the deflection of the springs, the angular reference of the servo was set to 120°. The setup is depicted in Fig. 10, whereas Fig. 11 shows the evolution of the servo shaft and output link angles of the shoulder pitch (q 2 ) and elbow pitch (q 3 ) joints, along with the servo speed and PWM signals. It is interesting to observe the recoil rotation of q 2 caused by the displacement of the center of mass when the elbow joint lifts the load, and the underdamped behavior associated to the compliant joints. The metrics identified in Section III-C are indicated in Table VII:  TABLE VII  PARAMETERS AND SCORES OBTAINED IN THE LIFT LOAD BENCHMARK

B. Indoor Flight Benchmark: Bimanual Grasping
This experiment illustrates the application of the grasping benchmark (Section IV-B) with an aerial manipulation robot consisting of the compliant dual arm presented in [12] and a hexarotor platform equipped with the Pixhawk autopilot and an Intel NUC board, using the UAV Abstraction Layer (UAL) [35] to interface the multirotor control. The test is conducted in the CATEC indoor testbed, using a Vicon system to obtain the pose of the multirotor and the object to be grasped with accuracies below one centimeter. Fig. 12 and Table VIII show the scenario and parameters according to Fig. 4, whereas Fig. 13 represents graphically the evolution of the test The duration of the phases (take-off, approaching, grasping, and tool retrieval) is obtained from the trajectory of the multirotor: 11-11-4-9 seconds, respectively. The performance of the multirotor position control can be quantified considering the overshoot in the Y-axis position at t = 16 s (23 cm amplitude), and the maximum displacement

C. Outdoor Flight Benchmark: Contact Force Control
The contact force controller based on Cartesian deflection described in [21] was implemented in the compliant joint arm evaluated in Section VI-A, using the data from Fig. 11 for calibrating the internal estimation. The arm was integrated in a S550 hexarotor with the Pixhawk autopilot and a RTK GPS, weighting 2.7 kg in total. The aerial platform is controlled in position mode by a human pilot, enabling automatically the force controller when the impact is detected using the method described in Section III-E. The evolution of the interaction can be followed in Fig. 14 and Fig. 15. As it can be seen, the displacement of the multirotor due to the reaction force causes the contact loss as the arm reaches the kinematic singularity at t = 3.5 s. The metrics of interest, summarized in Table X, are obtained from Fig. 15. As it can be seen, the aerial robot is not able to maintain the contact more than 2.8 seconds since the autopilot does not takes into account the contact force.

VII. CONCLUSION AND FUTURE WORK
As the aerial manipulation technology approaches to the customer applications with higher TRL prototypes, it becomes more evident the necessity to define methods and criteria that facilitate the evaluation and comparison of the performance of these systems. The benchmarks presented in this letter are also useful for developers and manufacturers as it contributes to focus the effort in increasing the quality of the prototypes and provide a common way to demonstrate their capabilities.