Shared-Autonomy Control for Intuitive Bimanual Tele-Manipulation

Existing dual-arm teleoperation systems function on one-to-one coupling of the human and robotic arms to fully exploit the user's dexterity during bimanual tele-manipulation. While the individual coordination of the robot end-effectors can be necessary for complex and asymmetric tasks, it may result in a cumbersome user experience during symmetric bimanual tasks (e.g. manipulating and carrying objects). In this paper we propose a novel framework that includes the one-to-one direct control and a new shared autonomy strategy. The user can autonomously choose between the two, and if the new one is selected the robots move in a coordinated way, in which desired positions are extrapolated from the movements and gestures of just one users arm. These gesture commands are interpreted and handled by the control, with the purpose of unloading the users cognitive burden. Lastly, the tele-impedance paradigm, i.e., the remote control of robot impedance and position references, is applied to both controls, to improve remote physical interaction performances. The paper reports on the overall proposed architecture, its implementation and its preliminary validation trough a multi subject experimental campaign.


I. INTRODUCTION
Teleoperation technology is attracting renewed attention due to its potential to allow human operators to execute complex tasks in remote, possibly dangerous environments.Three factors have been crucial to this resurgence of interest: 1) the availability of new technologies for accurate torque controlled robots and end-effectors, 2) new lightweight wearable devices for human-robot interface, to sense human inputs and to display visual and tactile feedback, and 3) new control paradigms to safely handle physical interactions with the environment.Examples of collaborative robots include KUKA iiwa [1], Panda [2], and Universal Robot series [3], among others.The interfaces to measure human parameters vary from kinematic tracking systems (motion capture devices such as Optitrack [4], Microsoft Kinect [5], etc.) to dynamic ones such as sensor insoles (e.g.see [6]).Feedback mechanisms to improve human awareness of the remote Correspond to: {marco.laghi,michele.maimeri} at iit.it environment, can range from grounded force feedback (e.g., [7]) to haptic displays [8] and vision (e.g.see [9]).The most advanced control algorithms to create a robotic system for remote presence are based on the bilateral, force-reflecting scheme [10], [11], or more recently, the tele-impedance control [12].
The integration of the above systems in several application scenarios such as space exploration [13], disaster response [14], medical [15] and industrial [16] services have already demonstrated the benefits of the simplicity and effectiveness of modern teleoperation systems.Besides the technical aspects of teleoperation, however, the ultimate dominant factor for its success is the users' acceptability.Although a complete understanding of acceptability in human-robot interaction is not available today, it is of common understanding that both cognitive factors such as intuitiveness and attention needs, and physical factors such as comfort and fatigue, must be taken into account while designing a teleoperation system.In particular, when it comes to teleoperating high degreesof-freedom (DoF) robots, such as bimanual manipulation systems, this requirement becomes significantly important.
The existing dual-arm teleoperation systems function on one-to-one coupling of the human and robotic arms with the aim to increase the dexterity of bimanual tele-manipulation.This provides the users with the ability to individually control each robot end-effector's motion, to successfully execute asymmetric bimanual tasks.The examples include bilateral teleoperation control of dual-arm systems using grounded haptic interfaces [17], [18], [19], [20], exoskeletons [21], and touch-based interfaces [22].One issue here is that, while the individual coordination of the robot end-effectors are necessary in executing certain complex and asymmetric tasks, it can result in a cumbersome user experience during symmetric bimanual tasks.For instance, while holding and carrying a heavy load using two arms, an individual control of the robot movements may impose a high cognitive load, not lastly to compensate for non perfect human-robot movement coupling, and can result in generating high internal forces or an object loss.
Hence, the objective of this paper is to develop a novel framework that integrates a shared-autonomy telemanipulation interface to enable human operators to effectively and comfortably achieve complex bi-manual tasks.This is done by the recognition of human hand-arm gestures in real-time, to implement an independent or a coordinated mode (Fig. 1) to control the movement and the physical interaction performance of the bimanual robotic system.The latter is realized by the tele-impedance paradigm, i.e., the remote control of robot impedance and position references.A preliminary assessment of the proposed interface in improved intuitiveness, acceptability and performance of the sharedautonomy teleoperation framework is carried out experimentally on five subjects.
The paper is structured as follow: in Sec.II the methods used to calculate the human inputs needed for the robots control are explained.In Sec.III the two strategies to control the dual-arm platform and the switching technique between them are presented.Sec.IV describes the experiments conducted and reports the results, which are subsequently discussed in Sec.V. Finally, conclusions are synthesized is Sec.VI.

II. HUMAN INPUTS
This section describes in details the method used for the tracking of human inputs, i.e., bimanual position and stiffness references, and the gestures to switch the control strategy and command the robots.

A. Human arm position tracking
The human arm can be approximated using a seven (DoF) serial chain with three links (as shown in Fig. (2)): the upper arm with length l u , the forearm with length l f and the hand with length l h .The chain's joints are: a spherical joint for the shoulder (3 DoF), a rotational joint for the elbow (1 DoF) and another spherical joint for the wrist (3 DoF).As said in the introduction, new enabling technologies permit to easily calculate the posture of human arms.In the case of IMU based tracking or motion capture, the sensors provide the orientation of the body at which they are connected, in the form e.g., of quaternions or euler angles.If the link lengths are known, the kinematics of a serial-chain is quite straightforward: referring to Fig. 2, once received the orientation of the arm links, it is possible to calculate the homogeneous transformation matrix between the user's hands and the world reference as: where * = L, R, with L standing for left and R for right; 0 l l l s, * is the position vector of the shoulder frame Ψ s, * in the world frame Ψ 0 , considered constant, u l l l u = [l u , 0, 0] T , f l l l f = [l f , 0, 0] T , h l l l h = [l h , 0, 0] T are the constant vectors of the segments expressed in their local frames.Finally, 0 R u, * , 0 R f , * and 0 R h, * are the rotation matrices between the local arm segment frames and the world frame Ψ 0 , which can be obtained with standard conversion from the orientation form (quaternions, RPY-angles, . . . ) returned by the used tracking sensors.

B. Human arm stiffness profile tracking
The tracking of the human arm stiffness and its replication on a robotic arm controlled via impedance controller is called tele-impedance.Various implementations of this paradigm have been presented, depending on the application and the level of desired accuracy.Automatically, the higher is the desired accuracy the more complex is the calibration and identification of the user parameters, and higher is the time required for it.An examples of high complexity implementation is [12], while one more direct and simple is [23].Similarly to the latter, with the purpose of fastreconfiguration of the bimanual interface to various users, a simplified model of human impedance tracking is developed here.To do so, the stiffness is evaluated proportionally to the activity of the upper arm's muscles, monitored through surface electromyography sensors (sEMG).sEMG senses the change of electric potential on the skin due to the muscle activation.This signal, once rectified and cleaned through filtering, gives a measurement of muscle activity a.
Given n sEMGs, positioned on m = n/2 different couples of antagonistic upper arm muscles, the arm stiffness index s a (t) used in this paper is defined as following: where a i (t) is the activity of the i-th muscle at instant t and a max = ∑ n i=1 max{a i } is the sum of the maximum values of each single muscle activity.Consequently: s a (t) is an index of the arm stiffness and can be used to create a reference stiffness profile for the impedance controller of the tele-operated robotic arm.

C. Hand gesture recognition
While the muscles activity of the upper arm can be used to have an indication of the arm stiffness, the one of the forearm can be used to recognize the movements and the gesture of the hand.For this work we implemented a simple recognition technique to identify the gesture of fist and finger spread.The two gestures can be distinguished from the forearm muscles activity because these mainly involves different muscles positioned in opposite sides of the forearm.We use only two sEMG sensors, one on the flexor digitorum superficialis and the other on the flexor carpi radialis.Naming a e (t) the activity of the first and a c (t) the activity of the second, a fist gesture can be detected if a c (t) > a e (t), a finger spread if a e (t) > a c (t).One issue is that, the same muscles are naturally involved also in the wrist movements, but usually with low activity levels.To neglect the activations caused by the wrist movements, a security threshold th is added.The recognition algorithm is then defined as follow: with th that dynamically changes in order to both avoid false positive during hand resting (th max ) and avert to tire subjects when they perform a gesture (th min ).This algorithm, basically, works as a relay with hysteresis.Starting from a resting position the threshold is set to the greater value (th = th max ).When at least one of the two signals exceeds the threshold, this one changes to the lower value (th = th min ).Finally, if both the signals become lower than the new threshold, the value of this one is restored to the starting one (th = th max ).

III. CONTROL STRATEGIES AND SWITCHING TECHNIQUE FOR BIMANUAL TELE-MANIPULATION
As introduced in Sec.I, this paper proposes a novel control architecture for unilateral dual-arm tele-manipulation.This architecture enables the user to choose on-line between an independent dual-arm tele-impedance control strategy and a shared autonomy strategy in which the user can control the dual-arm robotic platform using only one arm and some gestures, as it will be explained in the following.
In this Section the two control strategies are introduced.First, the independent dual-arm control is described.Then, the shared autonomy one is explained, and the technique used to switch between the two controls is clarified.

A. Independent Dual-Arm Strategy
In the independent dual-arm strategy the user left arm controls the left arm of the dual-arm robotic platform, and the user right arm controls the right robotic one.To control a robot through tele-impedance [12], two references have to be sent: a desired position and a desired stiffness.
The desired position and orientation are sent to the robots in homogeneous transformation matrix, b R H ee R ,d (t) for the right arm and b L H ee L ,d (t) for the left, where Ψ b R , Ψ b L are the base frames of right and left arm, and Ψ ee R ,d , Ψ ee L ,d are their desired end-effector frames (corresponding to the right user's hand frame Ψ h R and left one Ψ h L , respectively, see Fig. (2).These two references are obtained from: where 0 H h R (t) and 0 H h L (t) are the homogeneous transformations between the right hand frame Ψ h R and the world frame Ψ 0 and the left hand frame Ψ h L and the world frame Ψ 0 , respectively, both calculated with (1).Similarly, b R H 0 and b L H 0 are the homogeneous transformation matrices between the world frame Ψ 0 and the base frame of the right robotic arm Ψ R and the left robotic arm with b R l l l 0 and b L l l l 0 being the vector position of the world fixed frame Ψ 0 with respect to the two robot base frames.
As for the position, also the desired stiffness are set independently.Once defined the minimum and the maximum linear and rotational desired stiffnesses, k l,min , k l,max , k ω,min , k ω,max , the desired unidimensional linear and rotational stiffnesses k l * ,d and k ω * ,d are defined as: with * = R, L. Finally, the desired stiffness matrices at the end effector sent to the robot controllers are defined as follow: The desired homogeneous transformation matrix and stiffness are sent to the local robot controller, that computes the desired joint torques τ d (t) with the classic Cartesian impedance control: where J ee (t) is the robotic arm Jacobian at the end-effector, x(t) is the position and orientation error vector between b R H ee,d (t) and b R H ee (t) expressed in the end-effector frame Ψ ee and ẋ(t) is its derivative.D ee (t) is the damping matrix at the end-effector, that can be defined through a relation with K ee (t).In this work, it is defined imposing a damping factor ξ = 1.Finally, g(t) represent the gravity compensation torque contributions.

B. Synergistic Single-Arm Strategy
The control strategy explained in this subsection moves the two robotic arms always through (8), but with the reference positions and impedance profiles that depend on only one of the user arms, i.e. the right one.
The control is switched from independent dual-arm to shared autonomy when the user spread the fingers of the left hand.The gesture is detected through the recognition explained in Subsec.II-C.When at instant t s the switching is commanded, a virtual frame Ψ v is created in the middle point between the left and right user hand frames (that until the instant t = t s − δt coincide with the desired left and right end-effector frames), orientated as the right hand frame and linked to it, as shown in Fig. 3: with Then, for t ≥ t s the desired homogeneous transformation matrix of the right and left robotic arm's end effector of (4) are defined as: with: where R x,π indicates a rotation of π around the x axis (Fig. 3).In this way, the two end-effector always face each other.l o (t) is an offset length that regulates the distance between the two end-effectors, that will always be 2l o (t).l o (t) is initialized at t s to l o (t s ) = 0 l l l o (t s ) to allow a continuity in position after the switch.l o (t) varies with the following law: In this way, the spread of right hand fingers commands the increase of the two end effector distance with velocity α, while the right hand fist gesture commands a reduction of the distance with the same velocity α.This behavior is entirely handled by the robot controllers.With ( 13), l o (t) is saturated between a lower limit l o,min and an upper limit l o,max , arbitrary chosen, in order to avoid a clash between the two robotic arm and command unreachable positions.Note that the right hand gestures affect only the distance between the two robot end-effector, that will continue to move together at the same distance from the virtual frame Ψ v (11) (that is linked to the movement of the right arm through (9)(10)), with an orientation determined by the one of the right hand.
The desired stiffness is obtained always through (6), using only the right arm.The stiffness matrix resulting from ( 7) is assigned to the virtual frame Ψ v and mapped to the end effector frames Ψ ee R ,d and Ψ ee L ,d as follows: where Adi H j is the Adjoint matrix associated to i H j .The local robot control law is then the same of ( 8), with the stiffnesses given by ( 16) and the desired positions and orientations given by (11).
Eventually, if the user spreads again the left hand fingers, the control framework switches back to the independent dual-arm one.Fig. 4 shows a block diagram of the control framework just described.

IV. EXPERIMENTS
This section opens with a description of the setup used for the validation of the proposed control architecture, summarized in Fig. 4. The definition of the tasks used for this validation and the relative data results will follow.The discussion of the results will be held in the next section.

A. Experimental Setup
The orientations and the muscles activity of the arms links were obtained using three Myo armbands ( [24], Thalmic Lab™) on each arm, as illustrated in Fig. 1.The IMU embedded in each Myo provides the device orientation using quaternions.Converting each quaternion in rotation matrix it was possible to find the hand pose using (1).Furthermore, each armband provides eight sEMG signals, relative to the muscles embraced by the armband, already filtered and rectified.The sEMG signals of the upper arm's armbands were filtered again and used as muscle activation signals a i (t) of ( 2).The value a max of (2) was calculated with an initial calibration for each arm, where each user was asked to stiffen as much as possible his/her arms.The desired unidimensional stiffnesses were calculated through (6), with k l,min = 100 N/m, k ω,min = 10 Nm/rad, k l,max = 600 N/m, k ω,max = 60 Nm/rad, defined experimentally and taking in consideration the robots capabilities.The gestures were recognized with the procedure described in Subsec.II-C, using the signals of the forearm Myo armbands EMGs relative to the interested muscles, and used by the framework to switch control and determine c(t) of ( 14), with α = 3 cm/s.The bimanual robotic platform was composed by two Panda robots by Franka Emika [2] and the proposed control architecture was implemented in ROS [25] in a Linux environment.
The robots were posed with the base frames equally oriented and aligned on the y axis at a distance of 0.6 m.The fixed world frame Ψ 0 was put in the middle of the segment connecting the two robotic base frames and oriented in the same way, the user's torso was considered virtually centered in it.Therefore, the user shoulder frames Ψ s,L and Ψ s,L result to be symmetric with respect to the plane x − z of Ψ 0 at a distance depending on the user's shoulder distance.

B. Experimental Tasks
To evaluate the performance of each strategy, we developed three experiments.Each one of these was designed to test the difficulty of performing basic movements of a carried object, such as rotations and translations, in teleoperation with both strategies.Five subjects, between 26 and 33 years old, performed the experiments, which consisted in executing different tasks with both the dual-and single-arm strategies.All subjects were naive to the experimental purpose of the study and had no history of neuromuscular disorders.For each participant, the order of the strategies used during each task was randomized.During each task subjects had to accomplish some actions/movements in repetition, such as picking up a box and placing it in a specific area.We evaluated the number of actions/movements n s correctly performed in a period of 2 minutes.Before each combination of task and control strategy, participants performed one minute of training to become familiar with the system.Subject were asked to fill out a Likert scale questionnaire after the execution of each task with both control strategies.The statements of the questionnaire were the following: 1) It was easier to execute the asked task with the dualarm strategy than the single-arm; 2) It was more tiresome to execute the asked task with the dual-arm strategy than the single-arm; 3) It was easier to move the robotic arms with the singlearm strategy than the dual-arm; 4) It was easier to manipulate the object with the singlearm strategy than the dual-arm; 5) Overall, I felt I was improving the performances together with the trial numbers; 6) Overall, I preferred the Single-arm strategy.The possible answers ranged from completely disagree to completely agree, with an assigned score of −3 and +3, respectively.For the last question, three possible answers were allowed: 1-completely disagree (−3 points); 2-Undecided (0 points); 3-completely agree (+3 points).  .score of the six statements of the questionnaire for this task.
2) Task 2 -Box rotation: The second task consisted of rotating the same box used for the first task 90 • around an axis parallel to the ground and passing through the centers of the end effectors.Fig. 5(f)-(j) show the various phases of this task.The box was positioned on the table centered with respect to the two Pandas.Subjects had to pick up the box (Fig. 5(f)), rotate it through the abduction of the wrist (Fig. 5(h)-5(i)) and then position it back on the table, in the new configuration.The trial was successfully executed if the user completed the asked rotation of the box without touching the table.The two bars in the middle of Fig. 6(a) show the average n s of this task both with dual-and single-arm strategies, together with their standard deviations.The average scores of the six statements of the questionnaire for this task are depicted in Fig. 6(c).
3) Task 3 -Box tilting: The third task consisted in tilting the same box of the the other tasks around an axis parallel to the y of world frame Ψ 0 (Fig. 3) and passing though the center of the box.Fig. 5(k)-(o) shows the various phases of this task.The box was positioned on the table centered with respect to the two Pandas.After picking it up (Fig. 5(l)), each subject had to lift it, tilt it clockwise (Fig. 5(m)), tilt it counter-clockwise (Fig. 5(n)) and place it back in the initial position.The trial was successfully executed if the box did not fall and both tilt angles resulted greater than 45 • .The two bars at the right of Fig. 6(a) show the average n s of this task both with dual-and single-arm strategies and their standard deviation.The average scores of the six statements of the questionnaire for this task are depicted in Fig. 6(d).
V. RESULTS AND DISCUSSION Fig. 6(a) results suggest that the subjects performed better with the proposed single-arm strategy than the dual-arm in the first and the third tasks.Furthermore, in these two cases the variance registered for the dual-arm strategy is consistently larger than for the single-arm.This last observation seems to suggest that the performances achieved with the dual-arm strategy are strongly user-dependent, while the single-arm strategy is able to compensate in part for the user's lack of experience and dexterity.This consideration is supported by the questionnaire results (Fig. 6(b) and 6(d)), in which none of the subjects preferred the dual-arm strategy.The analysis the interaction forces can shed more light on these results.Fig. 7 depicts the external forces normal to the contact with the object, sensed at the left and right end-effectors during a single repetition of the box tilting task, for both dual-arm (Fig. 7(a)) and single arm (Fig. 7(b)).During the contact, the average values during the manipulation phase are similar in both cases.Instead, the variance of the signals during the manipulation phase with the dual-arm strategy is significantly higher then the one with the single-arm strategy.This indicates that the control of the closure is particularly hard for the user and affects its performance.Indeed, during single-arm strategy the closure is autonomously handled by the control, leaving the user free to move without focusing on it.
Differently from the first and the third tasks, in the second, the subjects performed better with the dual-arm mode (middle bars of Fig. 6(a)).Nonetheless, the variance is high and similar in both strategies.The questionnaire (Fig. 6(c)) confirms that the subjects preferred the dual-arm strategy, suggesting that a direct and independent control of the robotic arms is more suitable for this task.
The results of these experiments show the particular usefulness of a combined control strategy, like the one presented in this paper.Neither the single-nor dual-arm strategy are likely to be optimal in all situations, thus offering the user to switch freely between them could grant maximum dexterity and efficiency.Fig. (8) shows an example of a complex task structured in different phases: a ball has to be put inside a box, which is then closed with its cover.Once assembled, the box has to be moved and manipulated.Naturally, in the first two phases (ball in the box,

VI. CONCLUSIONS
This paper presented a two levels autonomy framework for bimanual tele-manipulation.With the proposed architecture, the user can choose between a classic dual-arm direct control and a new shared-autonomy control strategy, in which the robots move in a coordinated way.The provided preliminary results suggested that both strategies are useful, depending on the desired task.Indeed, while the dual-arm direct control is more convenient and sometimes necessary for complex and asymmetric tasks, the other can perform better in moving or manipulating objects and assure a more stable and constant grasp.
Future works will focus on an extensive evaluation of the proposed strategy, involving an higher number of subjects and tasks.Furthermore, different inputs for the control switching and grasping will be considered.Ultimately, the possibility of increasing the number of autonomy levels among which the user can choose the most suitable depending on the task will be investigated.

Fig. 1 :
Fig.1: An example frame of single-arm synergistic manipulation operated with the proposed approach.

Fig. 2 :
Fig. 2: Representation of the frames involved in the dual arm telemanipulation strategy of Subsec.III-A and length of the human arm segments: l u for the upper arm, l f for the forearm and l h for the length of the hand palm.The user is virtually positioned in the middle of the dual-arm robotic platform.

Fig. 3 :
Fig. 3: Representation of the frames involved in the single arm teleoperation strategy of Subsec.III-B at the switching instant t s .The frames of the user's left hand are not depicted because not involved in this strategy.
th min / * f i n g e r s p r e a d d e t e c t e d * / e l s e th(t) = th min / * f i s t d e t e c t e d * / e l s e th(t) = th max / * hand r e s t i n g * /

Fig. 4 :
Fig. 4: Block diagram representation of the control strategy flow and switching technique described in Sec.III depending on the user's hand gesture.

Fig. 5 :
Fig. 5: Photo sequences of the experimental tasks: (a)-(e) (first raw) are the phases of the first task -Pick and Place (IV-B.1)-,(f)-(j) (second raw) are the phases of the second task -Box rotation (IV-B.2)-,while (k)-(o) (third raw) are the phases of the phases of the second task -Box tilting (IV-B.3).

1) Task 1 -Fig. 7 :
Fig. 7: External forces normal to the contact with the object, sensed at the left and right end-effectors during a single repetition of the box tilting task (IV-B.3),for both (a) dual-arm and (b) single arm.

Fig. 8 :
Fig. 8: Photo sequence of a complex task example: ball inserted in a box (a)-(c), which is than closed with its cover (d)-(e) and manipulated in the air (f)-(j).The first two actions are executed using the dual-arm strategy (a)-(e), while the in-air manipulation is executed switching to the one-arm strategy.In frame (f) it is possible to appreciate the moment in which the subject switches the control strategy, through the spread of the left hand finger.
Fig 8(a)-(c), and box assembly, Fig 8(d)-(e)) the dual-arm strategy is more convenient, because the user can independently adapt the position of the two arms to the possible misalignment of the parts and carefully complete the assembly.Then, the third phase (box manipulation, Fig.8(f)-(j)) is easier to be executed using the one-arm strategy, as demonstrated with the results of the experimental tasks (IV-B).