Assistance Strategies for robotized laparoscopy

Robotizing laparoscopic surgery not only allows achieving better accuracy to operate when a scale factor is applied between master and slave or thanks to the use of tools with 3 DoF, which cannot be used in conventional manual surgery, but also due to additional informatic support. Relying on computer assistance different strategies that facilitate the task of the surgeon can be incorporated, either in the form of autonomous navigation or cooperative guidance, providing sensory or visual feedback, or introducing certain limitations of movements. This paper describes different ways of assistance aimed at improving the work capacity of the surgeon and achieving more safety for the patient, and the results obtained with the prototype developed at UPC.


Introduction
The introduction of robotics in laparoscopic surgery started in the early 2000s, initially with systems equipped with three robotic arms, two for operation and the third for the support and guidance of the laparoscope. Subsequently, a fourth arm was introduced so that the surgeon could manipulate an additional instrument for holding other auxiliary tasks without the need of an external cooperation. Since the introduction of robotic laparoscopic surgery, with the Da Vinci system being so far the only robotic system with certification to be marketed since 2000, Fig. 1, about 5,000 systems have been installed, of which about 50 in Spain. In the coming years this figure will increase significantly, especially with the appearance of new systems, such as the CMR Versus now recently certified [1], Fig. 2. It is also expected that in the coming years the offer diversifies with the appearance of other systems that can become more competitive. The system developed at UPC [2], Fig. 3, is expected to be one of them in the time frame of two years, the minimum necessary to fulfil all legal requirements. The systems currently available have been incorporating new features, basically aimed at three objectives: facilitating the work of the surgeon, increasing the patient's safety, and making this type of surgery less and less invasive. In the first aspect, in order to provide a better assistance to the surgeon, new instruments have been developed. To put an example, by using staples the number of sutures to be performed can be reduced, with the advantage this entails in terms of required ability and operation time. On the Casals, A. [et al.]. Assistance strategies for robotized laparoscopy. A: "Robot 2019: Fourth Iberian Robotics Conference: Advances in Robotics, volume 2". Berlín: Springer, 2020, p. 485-496. The final authenticated version is available online at https://doi.org/10.1007/978-3-030-36150-1_40 2 other hand, assistance systems have been developed to streamline the set-up, a process that still takes too long. This aspect is relevant, since the introduction of robots has been based on the high levels of surgical efficacy achieved, more than on efficiency, due to the high cost of current systems and the longer operation time still required in front of conventional surgery.
In the second aspect, the increase of patient's safety, visualization systems are developed, which provide the surgeon with a better vision of the position of each instrument or with means of fixing protection boundaries, virtual fixtures. These systems tend to compensate the lack of tactile feedback in teleoperated systems, since despite being a demand of the surgeons, it has not been achieved due to the impossibility of developing sensory systems integrable in the instruments that measure interaction efforts and are compatible with the required miniaturization and the needed robustness in front of sterilization processes. In the third aspect, to minimize trauma to the patient, suitable articulated instruments have been developed to be able to operate through a single hole, which has allowed to develop both, tools oriented to the manual SILS (Single-Incision Laparoscopic Surgery), and robotic systems oriented to this type of surgery, as the TSpot of Titan Medical, fig. 4.
There is therefore a line of progress that aims to look for improvements based on the design of architectures, or new mechanical devices, such as new stapler tweezers or specific tools to operate on a single port, fig.4, and another that with the available architectures and systems, enable the introduction of better control strategies or supervisory algorithms that allow improving the medical practice or patient safety. And this is the focus of this work, surgical improvements using the resources provided by multisensory perception, learning techniques and artificial intelligence or big data.

Robot based assistance systems for laparoscopic surgery
In robotic laparoscopic surgery the typology of soft tissues, the object of the intervention, does not allow a preprogramming of the tasks to be performed. Robotization here is implemented by means of teleoperation, which with the classical control structure can provide advantages over manual execution, despite not having a capacity for sensory feedback. Apart from the assistance functions already developed, the fact of robotizing allows introducing clear advantages since the trajectories of each arm can be provided with correction factors, either a gain factor or keeping the orientation of the working axes during a movement, which allows to increase precision and the work comfort.
The lack of force feedback does not prevent providing the operator with an alternative visual information, for example on the status of the instruments when they are not in the visual field or when they reach extreme positions within their working space.

Aids based on the geometric control of trajectories
Due to the large computing capabilities available in teleoperation control systems, many aids based on geometric calculations can be introduced, surpassing the capacity of a human operator, and thus facilitate teleoperation. The ability to memorize trajectories allows automatic and secure removing of an instrument and its automatic return to the point of origin. Considering this point as safe, which is reasonable if the removal has been carried out without incompatibilities with the environment, and if there have been no other changes in position in the surgical field, it is unlikely that the inserting and accessing movements change. The automatic execution of the removal and reentry movements to a specific working point is of especial interest to speed up the operation of cleaning the camera's objective when there is drowsiness that makes vision difficult, or to release an instrument of tissues that may have remained adhered during the intervention.
To guarantee a safe automatic realization of trajectories, such as the return to a certain point previously visited, and in order to anticipate the presence of unforeseen obstacles, some assisted control is necessary. That is, the generation of trajectories that are modifiable referring to both, execution speed and position, in which the position in each point of the trajectory is the vector sum of the planned or memorized trajectory, and the gesture control orders made by the operator, either directly on the arm of the robot, or on the controls, Fig. 5.
The geometric computing capacity provided by the control unit is also applicable to make the geometric measurements that help certain operations, such as the sizing of meshes or other tools, and to be able to automatically guide the camera, focusing the field of view on the tools working area, which position is known since these tools are governed by the own control unit.

Aids based on haptics
All teleoperated robotic systems in laparoscopic surgery are operated from a workstation with motorized controls, both to be able to do the initial positioning and to guide the operator in order to maintain the orientation of the instrument along a change of position, as well as to perceive limit situations. The perception of efforts in manual laparoscopic surgery, even being affected by the friction of the tool with the trocar, which appreciably distorts their values, is lost in robotized surgery due to the difficulties involved in the incorporation of force sensors in the different working tools. Thus, at present, the robotic systems of application in laparoscopic surgery do not have yet haptic capacities, although they are common in other teleoperation fields. Direct force/torque measurement, F/T, is achieved placing a F/T sensor on the laparoscopic tools to detect the interaction forces between tool and tissues. The placement of the F/T sensor is crucial to obtain the best measurements. Two are the feasible solutions: interpose the sensor between the tool actuator and the shaft or between tool and robot. The first solution offers optimal results, but its implementation passes through complex mechanical solutions. The most successful approach was developed by DLR, [3], where a F/T sensor based on a Stewart platform was placed to obtain direct measures. More recently, in [4] a grasping force sensor placed on the wrist of a robotic laparoscopic grasper was proposed. The sensor is based on a double layered capacitance sensor attached to a deformable wrist and a torque sensor on the pulley. This approach requires high precision and miniaturization, resulting into high cost solutions with sterilization and electromagnetic compatibility problems that are still pending to be solved.
Placing the F/T sensor outside the patient relaxes the restrictions of sensor size enabling the use of industrial force sensors. This solution suffers from measurement distortions induced by the trocar and tool flexions [5]. The contribution of these two factors is difficult to be modelled. In [6] a force sensed robotic trocar is presented where the force sensor is out of the patient. The force sensor is placed between a double concentric access channel. Laparoscopic tool interacts with the internal channel while the external 5 one is passively guided. This difference is measured as the interaction force between the tool and the surgical environment.
Unlike direct sensing, indirect F/T is estimated from sensors that directly measure deformations using other parameters. Most of the proposed solutions are based on measuring the intensity of the current of the motors that control the end effector driving cables, [7]. Following a similar principle, it is possible to estimate F/T measuring the tension of the driving cables. In [8], authors guide the grasper actuators cables through rigid channels. Tension variations on that cables generate measurable deformation of the elastic bodies. Using this deformation, the contact forces of the gripper can be estimated.
The use of surgical environment visual information obtained from the laparoscopic camera allows estimating the force from images. Tissues deformation produced by the contact between laparoscopic tools and tissues can be used to estimate interaction forces. Visual force estimation methods present several benefits and are a promising research field. Unfortunately, they require from a correct surface reconstruction to determine the depth and direction of the contact; tissue modelling of dynamic properties to estimate reaction forces against contacts and, finally, occlusions preclude correct penetration computation.
Although not having haptic capacity, as a perception of the efforts, interaction forces, made during the execution of the task, these robotic systems are provided with the necessary motorization at the joints of the master device to generate some haptic perception. This force feedback allows the human operator to perceive the mechanical limits of movements derived from the mechanical structure of the robotic arms, the virtual operation limits that may have been defined by the operator in advance or the security blockages that can be generated through the monitoring and supervision of the task.

Aiding systems based on the supervision of the task
Other support systems can be based on the autonomous or supervised execution of tasks, or part of tasks, which due to their predictability can be generated from a planner, based on predetermined procedures that can be executed with visual feedback. Many efforts are currently being done for their autonomous execution, or to guide the assisted manual execution of these procedures. The study of an intervention workflow allows acquiring the knowledge of the phases of the procedure and find the way to automatize some of them. In [9] this information is used for autonomously guiding the camera and for the planning of minimal invasive port positions in combination with an initial setup. In [10] a series of simple tasks are studied aiming to be automated and thus releasing the surgeon from the need to execute systematic and repetitive operations, allowing him or her to focus on the most difficult aspects. The operations analyzed were puncturing, cutting and suturing. A wider study for the modelling of surgical procedures is described in [11] aimed to be applied for the evaluation of surgeon skills, analyzing clinical team workload, optimizing the operating room management and predicting next surgical task or their duration. 6

3.
Aiding Systems in the frame of the Bitrack project Bitrack is a teleoperated robotic system for laparoscopic surgery, a system that is expected to be able to reach the market after obtaining the certifications of the CE and the FDA. The prototype has been tested with in vivo, with pigs, with various surgeons in different surgical specialties with satisfactory results and with a short learning curve, which is less than two sessions, thanks to the friendly user interface. Apart from regulations, further research is going on to provide new teleoperation aids, more than those conventional ones. The new aids developed that the surgeon can use when appropriate consist in haptic perception and guiding assistance for certain pre-established tasks.

Force estimation for the generation of haptic feedback
To provide some feeling of the efforts exerted during the intervention, the interaction forces can be estimated from the visual perception of the deformation caused by the end effector in the working environment and its quantification generates the equivalent reactive forces to be fed back to the master device.
The contact with the environment to be detected, which must generate the estimated force to perceive, can be either due to the contact between the instrument and the tissues, or between an element held by the tweezers and the environment too, being this element either a suture needle, a clip or so. The perception system developed should be able to segment the image distinguishing between the instrument handled by the robot arm, its grasped element and the detectable elements of the environment. With this aim, the perception system operates from a pair of contour images, obtained from the stereo endoscope used. To speed up the computing time for obtaining the perception of efforts in real time, these images are simplified. First, using working windows around each instrument and second extracting some singular points to reduce the amount of data to process. Fig. 6 shows the resulting segmentation. The instrument (labeled in red) has been identified from the window corresponding to the perimeter of each instrument and registered with the contour obtained by the projection of the CAD model of each instrument, seen from the 3D point corresponding to the position of the optics within the work space. This position is known by the control unit of the robot arm that guides it. When performing the growth of the points in contact with the instrument, it is also necessary to carry out the 3D points matching with the geometric model of the needle used, to detect its presence (in blue). Once isolated the singular points corresponding to the instrument, a growth of the rest of the singular points found identifies those considered as belonging to the background (in yellow).
Once achieved the segmentation that allows to obtain the singular points of the instrument, or of any element transported by the instrument, which interact with the environment, the image processing system obtains the displacement vector P (t) of each singular point, and proceeds to subtract each vector from the average of all the vectors considered, in order to eliminate the movement of the scene, Fig.7. Once the movement is filtered, compensating in part the movement of the scene, the exerted forces are inferred, those that better justify the displacement vectors obtained from the points in the environment. These calculated forces, which must emulate the F/T sensor that the tool should have had, generate the corresponding reaction forces on the control device, information that can be complemented with synthetic images on the screen.

Guidance aids
The developed system is also provided with guidance aids, if this function is activated by the user, which allows generating forces of attraction that facilitate the surgeon to carry out these tasks. These guiding aids functions can be geometric, or they can be for goals. The first ones are those that can generate geometrically trajectories from known passing points. The second ones are those needing a planner, able to generate a trajectory that allows the execution of a concrete objective. Initially, the aiding functions introduced are of geometric type to be able to achieve the high reliability that surgical robotics requires. In this line, the developed aids which have given good and reliable results have been the guidance in operations of insertion and retrieval of the instruments, and in the realization of sutures.
For the retrieval assistance the system generates the forces that maintain the trajectory of the instrument axis along the entire path, since there are no other elements in the space occupied by the tool. The guided entry has two clearly differentiable variants by the own system: a first entry or a new entry towards the previous working point. In the first case, the guidance is oriented towards the center point of the field of view of the camera. This operation facilitates the advance when the instrument is still outside the camera field of view, and despite being blinded, the initial placement of the trocars guarantees that each instrument can be visible to the camera if the trajectory of the entry follows this direction. In the second case, the return to the working point is very useful both for cleaning the camera's optics, and for the cleaning or replacement of an instrument. With respect to the suture assistance, the surgeon after the puncture must rotate the needle as much centered as possible on the geometric center of the needle used. Since the segmentation allows to perceive an object at the tip of the instrument, the aiding system can register it with the different needles that can be used and determine the position in space of its center with respect to the distal point of the instrument used, Fig. 8. The accuracy in the computation of the virtual center of rotation improves as the movements of the surgeon allows to get more points of view of the needle. The turning made along the suture can then be assisted at the surgeon's will, with a guiding force that maintains the rotation without displacement, which facilitates the task accomplishment with less stress of involuntary displacement of the tissue to suture.

Future possibilities
Since the beginning of laparoscopic surgery in the 2000, its progress has been more focused on the mechanics, passing from a robot with three arms to a new with four, and later on in modifying the robot architecture in order to increase accessibility and compatibility between the arms to avoid constraints and collisions. Currently, much work is oriented to introduce new aiding systems to more evolved robotic systems and to the robots that have appeared in the market recently or that will appear in the next years.
In this sense, different research efforts have been done from different perspectives. One of them is the Horizon 2020 SARAS project, Smart Autonomous Robotic Assistant Surgeon, aimed to provide autonomous assistance to the main surgeon, that teleoperates the robotic arms, with two additional robot arms acting as auxiliary surgeons. We are contributing with the development of a vision-based force estimator to be able to provide haptic feedback, and with the system architecture.
In what refers to the haptic feedback, all Robotic Assisted Minimal Invasive Surgery (RAMIS) include an endoscope to visualize the scene during a procedure. Therefore, this visual information from the endoscope can be used to relate the deformation in the tissue surface to estimate the applied force [12]. From the conservation principle of the continuum mechanics, the forces applied to an elastic object is directly proportional to its change of shape. Following this principle, we propose to reconstruct the surface of the tissue using the visual data from the endoscope and find the relationship between the deformation and the applied forces.
Then, the force estimation is developed in a three steps process. First, a stereo reconstruction of the surface of the tissue that generates a dense point cloud. In the second step, a computation of the surface deformation of the tissue from the point cloud. Finally, in the third step, the computation of the force estimation using the relationship with the computed deformation and the type of tissue.

Surface Reconstruction.
In this part of the algorithm, we deal with the reconstruction of the workspace. Our approach is based on stereovision using the stereo endoscope of the system. This method uses the calibration of two different cameras, where the relative position of both is known, to search in one image all the pixels from the other via maximum correlation. These correspondences create a map of depths of the image pixels. Afterwards, this depths map is transformed in a 3D point cloud of the object. Fig.  9 shows the reconstruction of the surface of a placenta.

Deformation Computation.
From the previous recovered surface in this step the algorithm computes the deformation in the surface as the changes produced in a set of linearly independent vectors (3D lattice). We use this lattice in order to obtain a compromise between reducing computation cost and accuracy. So, in consequence, we use the lattice as our deformation model. Fig. 9. Stereo reconstruction of a placenta surface 10 The 3D lattice can be parametrized by the following formula: ; for 1, … ,3 where are a cubic basis spline and denotes the displacement of a control point with y 1 y 2 y 3 number of points. So, the lattice is calculated from the reconstruction by minimizing the following equation: where is the discrepancy measure term, is the regulation parameter, is the penalization term and is a term to preserve the lattice shape. Finally, the parameters of the lattice are the inputs to the neural network used to determine the relation function between the deformation and the applied force of the last part of our approach.
Force Estimation. As previously mentioned, our strategy to compute force feedback relies on using a Recurrent Neural Network (RNN) to get the relationship between the deformation and the applied force. The input to this neural network will be the lattice parameters and the geometric information. Specifically, we propose the usage of a Long-Short Term Memory (LSTM) based architecture to compute the force feedback.
Other approaches to calculate the force from the deformation are based on an experimental relationship between the deformation and the force [13]. In [14], the force is estimated from a combination of the deformation from the images and the electrical current from the motor during the interaction of the tool with the tissues. Application to reconstruct surface by mosaicking. An application of the computer vision in surgery is its application in a robust real time tracking for Fetoscopic Photocoagulation (FLP) surgery for Twin-Twin Transfusion Syndrome (TTTS) [15]. TTTS is a syndrome where both twins share the same placenta and the blood flood has an unbalanced intertwin from the donor twin to the recipient twin throught some vessel connections called anastomoses. The treatment of the TTTS relies on a fetoscopic laser to photocoagulate these anastomoses.
The system has been tested using videos from a real TTTS intervention. One of the principal problems of these videos is the low quality of images due to the dirty environment and liquid inside the amniotic sac jointly with the low quantity of light produced by the fetoscope. In consequence, the system must deal with images that are noisy, blurred and poor colour components.
This method assists in the robotic system to track and stabilize the region of interests during the coagulation of the anastomoses by compensating the placenta's movement by calculating the relative position of the fetoscope tool tip with respect to the placenta surface via local vascular structure registration. More precisely, the system works in the following three steps. First, the method binarizes the images from the endoscope to segment the local superficial vascular structures of the placenta. Second, using these vascular structures, the algorithm searches for a sequence of relevant points of interest (POIs) that are distributed among the image. Third, the system uses these points to calculate the transformation between two consecutives images. Fig. 10 shows some placenta mosaics created using this method.

Conclusions
The aiding systems developed, based on geometric criteria, have shown that they facilitate the task of the surgeon after a relatively short learning time by performing a pyeoplasty and a clamp in the renal artery, after trying with different surgeons. These aids that robotization of laparoscopic surgery can provide, constitute just a first step that robotics can bring in the future in the field of surgery. In this first stage, only the aids for guidance were addressed since these control strategies can be justified rigorously in the necessary certification process. The trajectories that are taken as reference of a task can be geometrically defined from points provided by vision systems or from the calculation of the kinematics chain robot-trocar-instrument. On the other hand, performing tasks in a more autonomous way, based on expert systems, experience acquired by learning or by databases are currently more difficult to certificate. However, the good results obtained and the acceptance it deserved by the professionals that have collaborated, foresee great advances in this field in the coming years. Much can be done; however, the challenge is to achieve the required reliability that surgery requires.