Tactile augmented reality for arteries palpation in open surgery training

. Palpation is an essential step of several open surgical procedures for locating arteries by arterial pulse detection. In this context, surgical simulation would ideally provide realistic haptic sensations to the operator. This paper presents a proof of concept implementation of tactile augmented reality for open-surgery training. The system is based on the integration of a wearable tactile device into an augmented physical simulator which allows the real time tracking of artery reproductions and the user finger and provides pulse feedback during palpation. Preliminary qualitative test showed a general consensus among surgeons regarding the realism of the arterial pulse feedback and the usefulness of tactile augmented reality in open-surgery simulators.


Introduction
In open surgery, surgeons rely heavily on haptics, including kinesthetic (force) and cutaneous (tactile) feedback, e.g. when palpating tissue to distinguish healthy from diseased and to localize hidden anatomical structures [1].The haptic feedback can enable the identification of anatomical landmarks such as muscle, bones and blood vessels [2]. For all this reasons, the haptic sense plays a paramount role in many surgical procedures allowing the surgeon to avoid damages to healthy tissues and/or unintentional perforations/ruptures of blood vessels and life-threatening exsanguination. In particularly, palpation is an essential step of several surgical procedures for locating arteries, hidden beneath tissues, by arterial pulse detection. Surgical simula-tion would ideally provide realistic haptic sensations to the operator. At the current time, arterial palpation is mostly neglected in medical training simulators, with the exception of very specialized simulators [3]. These latter can be grouped into three main classes: physical simulators such as life-sized anatomical human models(i.e. SimMan® by Laerdal, HPS by CAE) with simulated feel able arterial pulse (i.e. carotid, femoral and brachial pulse), virtual simulators with haptic interfaces for simulation of pulse palpation in a virtual environment [4,5], hybrid simulators with a physical replica of a pulsating arterial vessel and augmented reality functionalities (i.e. to improve the simulation realism by adding a "virtual jet" of blood in case of a possible needle insertion after pulse palpation) [6,7].Examples of implementing a pulse simulator include: using an hydraulic interface (with an hydraulic pump to inflate a tube) to provide a physical replica of a pulsating vessel [6,7]; using the acoustic Radiation Pressure (ARP) phenomena generated by focusing multiple ultrasonic transducers to create perceivable vibrations trough an haptic interface [8,9]; integrating different tactile devices, based on a piezoelectric pads, a micro speakers, or a pin array, into a commercial force feedback device (Falcon by Novint Technologies, Inc.) [5]. In this work we propose a tactile augmented reality system for open-surgery simulation. The main idea is to merge benefits of haptic feedback in surgical training with those offered by an augmented physical simulator comprising high-fidelity physical models of the human anatomy and augmented reality functionalities. This paper presents a proof-of-concept implementation of the aforementioned approach for the simulation of artery pulse palpation. The system is based on the integration of a wearable haptic device into an augmented physical simulation platform allowing the real time tracking and augmented reality visualization of deformable tubular structures [10,11]. In particularly, we propose a method to accurately track the arterial replicas and the user finger [12] and to provide pulse feedback during palpation through a wearable haptic device. As detailed in the following paragraphs, we have chosen cholecystectomy simulation for the preliminary demonstration of the proposed concept. During this surgical procedure, pulse palpation is an essential step for the recognition of the cystic and right hepatic artery.

Fig. 1. Anatomy of Calot's Triangle
This latter, which is the most challenging step of the entire procedure, requires a meticulous dissection of the Calot's triangle contents [13] for a proper ligation and division of the cystic artery and cystic duct. The simulation platform comprises: realistic physical replicas of the involved anatomical structures, an electromagnetic (EM) tracking system, an optical calibration-acquisition system, and a laptop. In particularly the AR-Sim includes patient-specific physical replicas of: liver, gallbladder, pancreas, abdominal aorta, stomach, duodenum (fabricated as described in [10,11]) and realistic physical replicas of biliary ducts, arterial tree and connective tissue (fabricated as described in [10,11]). All the anatomical replicas match the shape, geometry and consistency of real human organs to enable a realistic interaction between the surgical tools/surgeon hand and the simulated anatomy (Fig.2). The biliary ducts and arterial tree are sensorized with EM sensors and tracked in real time for the implementation of augmented reality functionalities: tactile augmented reality and visual augmented reality. Indeed, the current version of the simulator, which was originally design for laparoscopic procedures, includes visual augmented reality information (Fig.2), displayed on a laptop screen, to aid the trainee in the recognition of the Calot's triangle, covered by connective tissue. In this work, this visual cue is integrated with the tactile augmentation: the surgeon can not only visualize the position of the cystic artery through the AR scene but can also feel its pulse by means of the haptic device.
The optical calibration-acquisition system has a twofold function indeed it allows the user to perform a calibration routine, and it acquires the real scene and displays it on the laptop screen. The AR-Sim simulator: a) patient-specific physical replicas; b) augmented reality scene allowing the trainee to visualize the current position of the cystic artery and biliary tree (respectively represented by the red and green virtual structures) which are covered by the synthetic connective tissue.

Wearable Fabric Yielding Device W-FYD
W-FYD (Wearable Fabric Yielding Device) (Errore. L'origine riferimento non è stata trovata.) is a tactile display for softness rendering and multi-cue delivery that can be worn on user's finger [14]. The mechanical structure, inspired by the grounded version of the device [15][16][17][18], is also similar to the one reported in [19], where two DC motors can vary the stiffness of the fabric stretching it, if independently controlled, provide tangential force (Errore. L'origine riferimento non è stata trovata.). The fabric is wrapped around two pins connected to the frame through suitable supports. When the user's finger is inserted inside the device, two different interaction modes can be used: active one, where the finger actively explores the fabric surface for softness, and passive mode, where the finger pad interacts with the fabric passively, thanks to the lifting mechanism that put the fabric, whose stiffness is regulated using the two DC motors, in contact with user's finger, which is still. In the active mode, the device is attached to the back of the finger, hence the only movement the user can perform is the flexion of the distal phalanx, which provokes the indentation of the fabric. To enable also passive mode, a lifting system is implemented trough a servomotor and a camshaft-based lifting mechanism, which puts the fabric in contact with the user's finger pad, making the platform where the fabric is wrapped around moving vertically of distance hp (Errore. L'origine riferimento non è stata trovata.).

Fig. 5.
Passive Mode: the fabric frame is put in contact by the camshaft lifting mechanism and servo-motor, inducing a variation of the height h p of the frame.
The fabric stretching system is actuated separately from the indentation mechanism, hence it is so possible to change the fabric stiffness independently from the force exerted on the finger of the user. Different levels of stiffness can be generated changing the stretching state of the fabric, obtained changing the relative angle between the two DC motors. The control of the stretching state of the fabric, and hence of the stiffness stimulus to be delivered, relies on the characterization of the system (Fig. 4 right side). Finally, thanks to the presence of the two independently controlled DC motors, W-FYD is endowed with an additional translational degree of freedom, which can induce a sensation of sliding/slipping on the user's fingertip (for further details the interested reader is invited to refer to [14]). For the application described in this work, we customized the device to have the fabric directly wrapped around user's finger pad. In this manner we enabled a more ecological and naturalistic interaction when the user palpated the simulated artery. Furthermore, only the passive mode was used since the goal was to exert a pressure on the subject's finger (to simulate artery pulse), without enabling stiffness rendering. We decided not to use the active mode, neither the stiffness rendering, because the task we wanted to perform did not require stiffness variation. The idea was to leave the user to freely interact with the simulated physical artery. Thanks to fabric deformability, we were able to superimpose artery pulse without compromising the sensation derived from artery palpation. To achieve this goal, we commanded a saw-tooth height variation to the lifting mechanism, in order to induce a deformation of the finger pad that simulates the pulsing artery. We sew a load cell (FSR model 400 round, thickness 0.1-0.3 mm) on the part of the fabric that was not in contact with the finger pad: the goal was to detect the pressing force exerted by the user. Future works will aim at investigating multi-digit implementation of the device and other haptic effects.

Wearable Haptic Device sensorization and calibration
The Wearable Haptic Interface was sensorized with a 6DOF EM Aurora Sensor positioned as illustrated in Errore. L'origine riferimento non è stata trovata..

Fig. 6. W-FYD sensorized with a 6DOF EM Aurora Sensor
A calibration procedure is required to precisely derive the position of the user finger from the EM sensor data so that to activate the pulse simulation when the user fingertip touches the artery models. At this aim a simple method based on static digitization was employed. In particularly, at each time t the position of the user fingertip, ⃗( ), can be calculated as: where: GRF indicates the Aurora Reference Frame (Global Reference Frame); LRF indicates a Local Reference Frame at the EM coil center ; ⃗( )is a 3D vector expressing the position at time t of the Local Reference Frame origin in the Aurora Reference Frame (GRF); ( )is a 3-by-3 rotation matrix which indicates the orientation of the Local Reference Frame with respect to the Aurora Reference Frame (GRF) at time t; ⃗is a 3D vector expressing the position of the user fingertip in the Local Reference Frame.
⃗( )and ( ) are directly derived from the 6DOF EM sensor which is tracked in real time.
⃗can be considered constant over a simulation session (since the EM sensor is rigidly attached to the Wearable Haptic Interface) and it can be evaluated at the beginning of each session by positioning the tip of a digitizer at the user fingertip.
⃗indeed can be calculated as follow: where k denotes the calibration time instant, ⃗( )is the digitized position (it is a 3D vector expressing the position at time k of the Digitizer Tip in the Aurora Reference Frame).

Arterial Pulse Simulation
The human pulse wave has different shapes depending on the patient constitution and medical conditions, and also on the pulse region [7,20]. In this paper, as a first approximation of the pulse pressure waveform, we have used a saw-tooth waveform model. Errore. L'origine riferimento non è stata trovata. shows a typical arterial pressure waveform and the simulated pulse profile used in this study. The simulated wave form is characterized by the pulse period T and by the maximum amplitude Amax.

Fig. 7. Approximation of pulse pressure waveform and simulated profile pulse
The length of the pulse period T can be varied to simulate different heart rate. The normal resting heart rate for adults ranges from 60 to 100 beats per minute (BPM), although this can vary depending on different circumstances. During a cholecystectomy procedure the heart rate changes depending on the patient decubitus, drug effects and carbon dioxide insufflation (in case of laparoscopic interventions), and it ranges from 70 to 90 beats per minute [21].
A max is calculated as: where F measured is the pressing force exerted by the user (measured by the force sensor) µ is the coefficient corresponding to the maximum deformation induced on the finger pad by the device F max and F min are the force bounds (see below). In our simulation a pulsating force feedback is rendered when two criteria, based on a proximity condition between the user fingertip and artery and on the force exerted by the user on the artery, are contemporaneously satisfied. As for the proximity condition, we employed a "bang-bang" control model making use of a threshold to take into account the Aurora sensor precision. In particularly, the pulse simulation is activated when the distance between the user fingertip and the sensorized artery satisfies where: d is distance between the user fingertip and the sensorized artery R artery is the sensorized artery radius h tissue is the sensorized thickness of the connective tissue which cover the artery δ Aurora is the Aurora dynamic accuracy which is 0.5 [22]. The pulse is deactivated when where ε Aurora is the Aurora dynamic precision which is 0.57 [22]. As regards to the force exerted by the user on the artery the following criteria should be satisfied: where F max and F min are the forces measured by the pressure sensor corresponding respectively to a standard systolic (120 mmHg) and diastolic (60 mmHg) pressure. These values were defined by using the following simple calibration procedure. The pressure sensor was attached to the skin of a volunteer in correspondence to the brachial artery (Fig.8).The cuff of the sphygmomanometer was then positioned over the sensor; the cuff was inflated to a pressure of 60 mmHg and to a pressure 120 mmHg and the forces measured by the pressure sensor were recorded.

Experimental design
Qualitative tests were performed to preliminary assess the proposed simulation concept.

Experimental set-up
The experimental set-up included the W-FYD and a simplified version of the AR-Sim simulator comprising only a sensorized arterial tree replica covered with synthetic connective tissue (Fig.9). To preliminary test the system the arterial tree and the connective tissue were positioned over a box (made of expanded polystyrene) as showed in Fig.9. The box was used as a spacer over the EM field emitter and as a physical support for both the sensorized arteries and the camera calibration grid [10,11]

Qualitative evaluation
A total of ten surgeons (eight general surgeons and two vascular surgeons) participated in this study. All participants were asked to try the simulator and to complete a questionnaire. The surgeons were asked to wear the tactile display and to palpate the mockup for the recognition of the cystic or right hepatic artery. During palpation, the W-FYD conveyed a pulse feed-back on user's finger. The questionnaire comprises 6 items (listed in Table 1) assessed using a five point Likert scale (1= strongly disagree to 5 strongly agree). Participants were also asked to comment on any other aspect of the simulator which they felt were important but had not been directly included in the questionnaire, and to list any suggestions to improve the simulation realism. Statistical analysis of data was performed using the SPSS ® Statistics Base 19 software. The central tendencies of responses to a single Likert item were summarized by using median, with dispersion measured by interquartile range. The Wilcoxon signed-ranks test was used to determine the significance of the responses to each item evaluating if the operators were significantly more likely to agree or disagree with each of the statements. A p-value <0.05 was considered statistically significant. Table 1 summarizes results of preliminary qualitative evaluation indicating that there is overall significant agreement with the following statements: "The pulse feedback is realistic", "The location of the arterial pulse in the real environment is correct and realistic", "The integration of the W-FYD in the surgical simulator is useful", "The integration of W-FYD in the laparotomic cholecystectomy simulator is useful for the recognition of the arterial tree". Moreover participants agreed with items 3 and 4 even if a larger study is required for statistical evidence. Pulse palpation, as well as pulse diagnosis, is a subjective process depending on a practitioner's palpatory skill and ability to discriminate changes in pulse variables. For this reason, the variability among the surgeons responses may reflect differences in user's ability and sensitivity.
Obtained results, however are very positive and encourage further studies to develop the proposed simulation platform. Responses to item 2, validate the accuracy of the user fingertip localization (which depends on the accuracy of the EM tracking system and of the haptic device calibration), since the pulse is correctly perceived when the user touches the physical arterial replica. The most prevalent theme within participants comments was the need for: reducing the dimension of the haptic interface to improve the human factor, and enabling a multi-digit implementation allowing the trainee to palpate the pulse with two fingers. Moreover participants suggested to add the possibility of differentiating the amplitude of the simulated arterial pulse according to the palpated artery. Indeed, whereas the mean arterial pressure is known to be identical in all parts of the arterial tree, the pulse pressure has a greater amplitude in peripheral than in central arteries due to the buffering function of the arteries [23,24].More particularly, one surgeon suggested to reduce the minimum force to be exerted by the user on the artery to feel the pulse. No concerns were raised on the realism of the synthetic tissue whose constructive materials were selected to feel natural to the touch according to the surgeon feedback [11]. To conclude, the benefits of haptic feedback in surgical training has been already recognized by several research groups and companies working in this field (e.g. Immersion Medical, Surgical Science and Mentice). Most of the work is focused on haptic rendering for simulation of tool-tissue interactions in virtual reality simulators for training of minimally invasive surgical procedures. In this work we firstly presents a proof-of-concept implementation of tactile augmented reality for open-surgery training. Preliminary qualitative test showed a general consensus among surgeons regarding the realism of the arterial pulse feedback and the usefulness of integrating this kind of haptic interface in open-surgery simulators. In the future the W-FYD will be optimized according to the surgeon feedback, enabling a multi-digit implementation while reducing the haptic device size. Moreover future studies will address ergonomics and human factors issues in the design of the augmented reality functionalities, considering the possibility to integrate an head-mounted display in the surgical training platform.