Active Deformation through Visual Servoing of Soft Objects

In this paper, we propose the ADVISEd (Active Deformation through VIsual SErvoing) method, a novel model-free deformation servoing method able to deform a soft object towards a desired shape. ADVISEd relies on an online estimation of the deformation Jacobian that relates the motion of the robot end-effector to the deformation behavior of the object. The estimation is based on a weighted least-squares minimization with a sliding window. The robustness of the method to observation noise is ensured using an eigenvalue-based confidence criterion. The ADVISEd method is validated through comparisons with a model-based and a model-free state-of-the-art methods. Two experimental setups are proposed to compare the methods, one to perform a marker-based active shaping task and one to perform several marker-less active shaping and shape preservation tasks. Experiments showed that our approach can interactively control the deformations of an object in different tasks while ensuring better robustness to external perturbations than the state-of-the-art methods.


I. Introduction
Controlling the deformation of an object can be of importance, for instance in surgery [1], car manufacturing [2], or food manipulation [3]. The use of robots might bring repeatability, safety and high throughput in the manipulation of deformable objects. However, most of the current industrial robots are not able to control the deformation of an object because it requires both knowledge of the object physical properties and real-time tracking of the deformations. Several studies have focused on real-time deformation tracking [4]- [7] but not simultaneously with deformation control. A few methods have been proposed to find a grasp configuration on a deformable object [8] but without interactive control of the deformation. In this paper, we focus on methods that can perform interactive deformation servoing tasks, i.e. the manipulation of the shape of a deformable object at an interactive framerate.
Our contributions are as follows: • the ADVISEd method, a novel deformation servoing method that can work without knowledge about the physical properties of the object; • two experimental setups to compare the ADVISEd method with state-of-the-art methods. The remainder of this paper is organized as follows: Sect. II presents the related work on deformation servoing, Sect. III describes the ADVISEd method while Sect. IV and Sect. V detail the experimental setups used for the comparison with state-of-the-art methods. Finally Sect. VI ends the paper with a discussion on the approach.

II. Related work
A possible categorization of deformation servoing methods is to distinguish those that depend on a prior model of the object (model-based methods) and those that do not rely on any characterization of the object (model-free methods). Modelbased methods require prior knowledge about the object that is deformed, such as the geometry and/or the physical properties of the soft object. These methods are mostly combined with physically-based simulations to predict the deformation of the object and compare it with the real observation. For instance, Smolen and Patriciu proposed a physics-inspired model based on particles which does not require a mesh of the object [9]. Another possibility is to model the contact between the object and the end-effector instead of modeling the whole object. Navarro-Alarcon et al. proposed such a method where the contacts are modeled as springs [10]. It combines offline information, the system acquiring at different locations the effect of the end-effector motion on the object deformation, with online adaptation of variable parameters. The same authors extended their deformation servoing method in [11] by performing simultaneously positioning and shaping tasks. Other approaches use the massspring formulation, which exhibits good computation time performances. These mass-spring models were even adapted to model the plastic behavior of the object [12]. However, mass-spring models do not express properly actual physical properties of the object and therefore often lack accuracy. For more accurate simulation, the Finite Element Method (FEM) has been introduced to better estimate the deformation of an object. Thus, Zhang et al. used it for controlling soft robots [13]. Frank et al. presented a method for robot endeffector path planning while avoiding the deformation of a soft object [14]. The FEM physics simulation has also been used for automatic flexible needle insertion in deformable tissue using a predefined path, for instance in the work of Adagolodjo et al. [15]. Assuming that the geometry and the physical properties of the object are available, model-based methods are able to provide accurate representations of the deformation of an object. However, the object representation and mesh reconstruction as well as the mechanical properties measurement remain still very time-consuming as-of-today and most of the proposed approaches are not able to reach on a real system the same accuracy that it could be expected from simulations.
Some deformation servoing methods that do not require any prior knowledge of the object have been proposed, such as the one of Berenson able to locally deform 1D and 2D soft objects [16]. Alonso-Mora et al. proposed a method to perform simultaneously the positioning and the shape preservation of 2D deformable objects [17]. However, shape preservation is only a secondary task neglected when its achievement induces a collision. Navarro-Alarcon et al. proposed a method to deform objects based on an online Jacobian estimation [18]. The online estimation is based on the Broyden algorithm that infers the variation of the Jacobian from the current and previous observations [19]. However, Broyden algorithm is very sensitive to noise: when the system is converging towards the desired configuration, some directions of the Jacobian are not stimulated and their update is performed using only measurement noise. Navarro-Alarcon et al. proposed another method that considers the contour of the object to perform shape deformation [20]. This method was based on the expression of the 2D shape of the object with a compact vector of Fourier coefficients. However, the system falls in local minima if the desired contour is not reachable or if the number of Fourier coefficients is not enough to well describe the shape of the object.

Positioning of our work
Our method is inspired by the Uncalibrated Image-Based Visual Servoing (IBVS) methods and belongs to the online model-free method category. We aim at controlling the deformation of an object without requiring any prior model or physical characteristics of the targeted object. Our approach is based on an online estimation of a deformation Jacobian that links the variation of the object deformation to the velocity screw vector of the robot end-effector that is in contact with the deformable object. This deformation Jacobian is used in the control law to deform the object.
There exist several ways for estimating online the Jacobian of a function. Hosoda and Asada proposed a method using the Lyapunov theory to tackle the noise sensitivity issue of the Broyden algorithm [21]. Their method was extended by Navarro-Alarcon et al. for deformation servoing [18]. This later method needs an accurate tuning of several parameters to ensure robustness. Moreover, it relies on 2D information to deform the object, which makes it difficult to control the whole shape of the object. In our approach, we propose to use only a small set of parameters to tune for estimating online the deformation Jacobian. We use a RGB-D sensor to quickly get 3D deformation observations allowing to control the global object shape. We combine these real-time observations with a weighted least-squares minimization with a sliding window [22]. Compared to the method proposed in [19], this method is more robust to noise since each row of the deformation Jacobian matrix is updated using a confidence criterion computed from the deformation observations. To the best of our knowledge, this method has been used only for standard IBVS by Krupa [23]. His objective was to control the orientation of a 2-Degree-Of-Freedom (2DOF) surgical instrument in the context of laparoscopic surgery using the image 2D coordinates of a laser spot projected from the instrument on the scene. Our approach aims at controlling the 3D shape of a deformable object by automatically applying a motion to the end-effector that performs the shaping task.

III. Description of our method
This paper aims at proposing a novel adaptive model-free deformation servoing method to control the 3D shape of an object. The formulation that will be used in this article is the following: • k denote the current discrete time. It is precised only when confusion can arise in a formula, • v ctrl ∈ R m is the computed velocity control law vector expressed in the robot end-effector frame, with m the number of DOFs of the end-effector, • e = s − s * ∈ R n is the deformation error vector, s and s * being respectively the current and desired deformation feature vectors. The size of the deformation and deformation error vectors is denoted n, •Ĵ + is the Moore-Penrose pseudo-inverse of an approximation of the deformation Jacobian J, which links the evolution of the deformation with the end-effector actual velocity v e i.e.ṡ = Jv e .
Our novel model-free deformation servoing method estimates the deformation Jacobian using a weighted leastsquares minimization method with a sliding window. The Jacobian is updated only w.r.t. the DOFs for which a confidence criterion has been met. This confidence criterion permits to filter noise measurement when the system converges towards the desired deformation. Assuming that the variation of the deformation featuresṡ is linear with regards to small motions of the end-effector, their relation can be written as: where the vector v e designates the velocity of the endeffector, the vectorĴ i,: designates the i th row of the deformation Jacobian matrix that we want to estimate and ν i is the i th element of the noise measurement vector. To minimize the predictive errorṡ i (k + 1) − v eT (k)Ĵ T i,: (k), we propose the following quadratic cost function C Ĵ T i,: ∈ R: where N is the size of the sliding window, the index j indicates that the sum is performed over previous discrete times, 0 < λ ≤ 1 is a constant forgetting factor giving less importance to the oldest measures and m(k) is a normalization signal. Let's define: The cost function can be rewritten as follows: The cost function is convex with regards toĴ T i,: thus its minimum is obtained for the value ofĴ T i,: that nullifies its gradient C with regards toĴ The estimatedĴ i,: that nullifies the gradient can be written: In practice, it is not possible to implement directly (6). Indeed, if the system is not well stimulated, which is notably the case when the error tends towards 0, the eigenvalues of the matrix R tend towards 0. Consequently, it is not possible to invert the matrix. To avoid this problem, a confidence criterion based on eigenvalue decomposition is used to determine how the Jacobian will be updated. The matrix R being a positive-definite symmetric m × m matrix, it can be decomposed such as: where γ 0 ≥ · · · ≥ γ m−1 ≥ 0 are the eigenvalues of R and φ 0 , . . . , φ m−1 are its eigenvectors. Finally, the update of the estimated deformation Jacobian is performed depending on a user-defined confidence threshold : with φ sorted such as φ j > and φ j+1 ≤ The subscript ":" indicates that all the rows (respectively columns) are selected when it appears before (respectively after) the comma, while the subscript "0:j" indicates that all the rows (or columns) from index 0 to index j are selected. The V 1 term in (8) allows to update the Jacobian in the directions where the eigen-value-based confidence criterion is met, while the term V 2 allows to ignore the information in the noisy directions.
Once the Jacobian has been updated, the velocity control law is computed such as: where α is the control law gain.
IV. Comparison with state-of-the-art methods on a marker-based active shaping task To evaluate our approach, we compared it with two stateof-the-art methods: a model-free deformation method proposed by Navarro-Alarcon et al. [18] and a model-based deformation method similar to the one proposed by Adagolodjo et al. [15]. First, we recall the main principles of the two state-of-the-art approaches. Then, we present a performance evaluation of the three methods when performing a markerbased active shaping task.
A. Description of the state-of-the-art methods 1) Model-based deformation method: We implemented a model-based deformation servoing method similar to the one proposed in [15]. We used a physically-based corotational FEM simulation using the SOFA framework [24] to estimate the deformation Jacobian. The velocity control law is given by v ctrl = −αĴ + e ∈ R m and α is the control law gain. Several physically-based simulations are run to estimate the deformation Jacobian. Each simulation performs a stimulation of the end-effector on the object by applying a translation or rotation dir j along the axis of the end-effector frame and observe the resulting simulated object deformation. The partial JacobiansĴ dir j ∈ R n,1 are estimated such as: where v j is the constant input velocity of the simulated endeffector in the j th direction and the subscripts indicate the indices of an element in a vector or a matrix. Then, all the individual deformation Jacobians are gathered in order to form the global deformation Jacobian matrix J such asĴ = Ĵ dir 0 . . .Ĵ dir m−1 ∈ R n,m . This updated Jacobian will be used until the next update to control the deformation of the object.
2) Model-free deformation servoing: We also compared our approach with the model-free deformation servoing method proposed by Navarro-Alarcón et al. in [18]. This method iteratively estimates the deformation Jacobian matrix J from the deformation feature vector and the robot velocity observation using the Broyden update rule [19]: where 0 < ε ≤ 1 ∈ R is a user-defined responsiveness gain and x ∈ R 3 is the relative linear displacement of the endeffector position. Only the three linear velocities of the endeffector are controlled with this method, i.e. m = 3. The estimated deformation Jacobian is thereafter used for the computation of the control law. The control law was developed according to the Lyapunov theory to ensure the robustness of the controller. The velocity control law is: where p is a numerical state vector, C ∈ R m,m is a symmetric positive damping matrix, c is a positive scalar acting like a damping-like feedback gain, sat is a saturation function, K s ∈ R + is a saturation gain and α is the control law gain.

B. Initialization of the Jacobian
The deformation Jacobian of both model-free deformation methods is initialized with a coarse guess of its value. An offline open-loop deformation Jacobian estimation is done: the end-effector is moved in each direction successively to stimulate the deformable object while recording the observed deformation. Then, the Jacobian is initialized as follows: where in j represents a constant velocity input that is applied successively on each axis during a period ∆ t and the subscript i denotes the i th raw of a vector or a matrix. This offline initialization takes a few seconds.

C. Experimental setup for the marker-based active shaping task
The experimental setup is composed of a 6-DOF anthropomorphic robot arm (Viper 650 from ADEPT) where the endeffector distal tool corresponds to a 3D-printed stylus (see Fig.3 for the setup, the foam is represented in an inset). A sheet of foam is deformed in order to reach a desired curved shape. A remote Intel Realsense D435 RGB-D camera is used for tracking both the robot and the soft object. The depth information is accurate within ±0.5mm.
For the marker-based active shaping task, the deformation of the object is tracked using η passive visual markers attached to the object surface. These markers are tracked in the RGB image using ViSP blob tracking abilities [25]. The deformation vector groups the 3D position of each marker w.r.t. the camera that can directly be retrieved from the depth sensor data, i.e. s = x 0 y 0 z 0 . . .
To determine the constant transformation between the remote camera frame R c and the robot base frame R r , we used a computer vision approach to determine the homogeneous transformation between the robot end-effector frame R e and the camera frame. We then chain this transformation to the homogeneous transformation between the end-effector frame R e and robot base frames frame R r provided by the robot forward kinematics. This determination is performed at the initialization of the experiment. The comparison of performances between the different methods takes into account several criteria: • t r : response time of the system, expressed in seconds, which corresponds to the time needed to reduce the error to 10% of its initial value e 0 , • t s : settling time of the system, expressed in seconds, which corresponds to the time after which the system stays in the interval [e ∞ − 0.05e 0 ; e ∞ + 0.05e 0 ], • e ∞ : final error, expressed in millimeters, between the desired and observed deformation vectors when the system finally is in steady-state, • d e f f : total distance traveled by the end-effector during the manipulation, expressed in millimeters For the marker-based active shaping task, the considered error function is the Euclidean norm of the deformation error vector i.e. e 0 = e(0) and e ∞ = e(k → ∞) . Fig.1 illustrates the evolution of the Euclidean norm of the deformation error vector obtained with the different methods when performing an active shaping task consisting in reaching a desired curved shape. Two passive markers were used during this experiment, so the size of the deformation and deformation error vectors was n = 6. Each algorithm was optimally tuned to achieve the best trade-off between accuracy, stability and responsiveness. For ADVISEd, the parameters were: λ = 0.99, N = 10, = 0.001, α = 0.05. For the model-based method, the parameters were: Young modulus = 13kPa, Poisson's ratio = 0.3, Rayleigh stiffness = 0.125, mass density = 40kg.m −3 . For the model-free method, the parameters were: The model-based method, whose results are depicted in blue in Fig.1, was less stable (d e f f = 63.5mm w.r.t. 16.3mm for both model-free methods) and less accurate (e ∞ = 3.5mm) than the two model-free methods (0.7mm for ADVISEd and 0.8mm for the other model-free method). This is due to the difficulty to accurately obtain the physical parameters that are required for modeling the geometry and physical properties of a deformable object as well as the surrounding scene. This enforced our idea of proposing a model-free method to avoid modeling constraints. The ADVISEd method and model-free state-of-the-art method had similar results in terms of stability and accuracy. The model-free state-of-the-art method was more responsive than the ADVISEd method (t r = 35s compared to 70s) but needed more time to become stable than the ADVISEd method (t s = 175s compared to 125s). We thus decided to conduct additional experiments to compare these two methods.
V. Comparison with a model-free state-of-the-art method on a marker-less active shaping task

A. Experimental Setup
We used the same hardware as presented in Sect. IV-C. Several soft objects were manipulated: a flat piece of foam, a soft ball and a soft toy. The experimental setup is shown in Fig. 3, the soft objects being presented as insets.
For this set of experiments, the deformation of the object is tracked using a marker-less method. A 3D point-cloud composed of η 3D points located in a given  Surface of Interest (SoI) and captured by the RGB-D camera is now selected as deformation feature vector s = x 0 y 0 z 0 . . . x η−1 y η−1 z η−1 T . The projected area of the SoI on the surface of the object was set to 3cm × 3cm. Different downsamplings of the point-cloud are tested in order to determine the impact of the number of 3D points onto the convergence to a desired configuration s * . To reduce measurement noise, M-estimation is used in order to remove outliers from the deformation error vector, such as the lack of depth data at one point due to sensor failure or occlusion due to the end-effector presence [26]. We apply Iteratively Reweighted Least-Squares (IRLS) method in order to convert the M-estimation problem into an equivalent least-squares problem, as proposed in [27]. The error used in the computation of the control law becomes e = W s − s * ∈ R n , where W is a diagonal weighting matrix containing the weights reflecting the confidence in the deformation features. W is recomputed each time a new deformation vector is made available using Tukey M-estimator [28]. This new expression of the deformation error vector changes the control law (9) in: To ensure a fair comparison between the ADVISEd method and this method, the velocity control law shown in  (11) has been adapted as follows: The Hausdorff distance is a metrics commonly used to compare two surfaces [29]. Let e H (k) be the Hausdorff distance between the desired and observed surface at the discrete time k. For the different marker-less tasks, the final error that is considered is e H∞ = e H (k → ∞).
Several kind of experiments were conducted: • a) folding a piece of foam to reach a desired shape, • b) deforming the surface of a soft ball to reach a desired shape (Fig.2a to 2c), • c) bringing back a deformed shape of the soft toy to a desired less deformed shape (Fig.2d to 2f), • d) same as exp. a) with external disturbances (upwards and downwards 3-centimeter displacements). Experiments a) and b) correspond to active shaping tasks of soft objects. Experiment c) is a shape preservation task. Experiment d) permits to evaluate the robustness of the method with regards to external disturbances. Before all of these experiments, the end-effector is moved in order to deform the object. The resulting deformed surface will be used as desired surface. The object and end-effector are thereafter put back in their resting state.
For experiments a) b) and c), several downsamplings of the point-cloud located in the SoI were tested to determine the impact of the number of points η on task completion. During these experiments, the ADVISEd method parameters were as follows: α = 0.045, λ = 0.99, N = 10 and = 0.001. The results of these experiments are presented in Table I. One can notice that the ADVISEd method was able to reach the same accuracy as the model-free state-of-the-art method. The number of considered points η in the SoI impacts the convergence speed of the methods because they are more constrained. The state-of-the-art method is more reactive than the ADVISEd method in terms of response time. This is due to the forgetting window of the ADVISEd method that tends to smooth the update of the deformation Jacobian.  Finally, experiment d) has been conducted to evaluate the robustness of the methods to external disturbances. The disturbances could either move the object upwards or downwards. Different sets of parameters have been tested for the ADVISEd method. The results are summarized in Table II. Increasing the size of the weighting window and decreasing the value of the forgetting factor both increase the robustness of the method w.r.t. external perturbations by decreasing the impact of occasional outliers. Increasing the value of the confidence threshold increases the robustness of the method by filtering noise measurements.
The Hausdorff distance being sensitive to the camera sensor noise, we chose to represent instead the mean square error e ms (k) = √ e T (k)·e(k) η for display purpose only. Fig. 4 depicts the evolution of e ms for each method when external disturbances were applied during an active shaping task. During this experiment, the ADVISEd method parameters were as follows: α = 0.045, = 0.99, N = 10 and = 0.001. The disturbances occurred when the system reached a steady-state and are depicted by arrows. The stateof-the-art method was more responsive than the ADVISEd method. However, the high responsiveness of the state-of-theart method made it lose contact with the object in several occasions, depicted by filled areas in Fig. 4. On the other hand, the ADVISEd method had a smooth response to the external perturbations ensuring an exponential decrease of the error and a permanent contact with the object. This different behavior between the two methods make them suitable for different kind of applications: the ADVISEd method is welladapted for applications where smooth behavior is required whereas the state-of-the-art method is more suitable when high responsiveness is desired and overshoot is not critical.

VI. Conclusion
In this paper we presented the ADVISEd method, a modelfree deformation servoing method based on a numerical estimation of the deformation Jacobian. Our method combines the advantages of requiring a small number of parameters while not relying on any prior knowledge on the manipulated soft object. Our method is based on a weighted leastmean squares minimization with a sliding window to online estimate the deformation Jacobian. This later is efficiently updated by determining the reliability of the observations using eigenvalue decomposition and a confidence threshold.
Several experiments were conducted to compare the AD-VISEd method to model-based and model-free state-ofthe-art methods, and to evaluate its ability to control the deformation of a soft object. A marker-based active shaping task showed that the model-based method required accurate knowledge of the physical parameters of the object and its neighborhood to ensure correct results whereas the our method and the model-free method did not suffer from this issue.
Further experiments of both active shaping and surface preservation tasks have been conducted to compare the ADVISEd method to the state-of-the-art model-free method. The accuracy of the ADVISEd method is independent of the number of points located in the Surface of Interest as well as external disturbances. Experiments showed that the ADVISEd method was more robust to external disturbances and as accurate as the state-of-the-art methods while relying on a smaller amount of parameters to tune. As such, the ADVISEd method is a promising alternative for interactively controlling the deformations of soft objects.