text: Purpose: In surgical navigation, pre-operative organ models are presented to surgeons during the intervention to help them in efficiently finding their target. In the case of soft tissue, these models need to be deformed and adapted to the current situation by using intra-operative sensor data. A promising method to realize this are real-time capable biomechanical models. Methods: We train a fully convolutional neural network to estimate a displacement field of all points inside an organ when given only the displacement of a part of the organ's surface. The network trains on entirely synthetic data of random organ-like meshes, which allows us to generate much more data than is otherwise available. The input and output data is discretized into a regular grid, allowing us to fully utilize the capabilities of convolutional operators and to train and infer in a highly parallelized manner. Results: The system is evaluated on in-silico liver models, phantom liver data and human in-vivo breathing data. We test the performance with varying material parameters, organ shapes and amount of visible surface. Even though the network is only trained on synthetic data, it adapts well to the various cases and gives a good estimation of the internal organ displacement. The inference runs at over 50 frames per second. Conclusions: We present a novel method for training a data-driven, real-time capable deformation model. The accuracy is comparable to other registration methods, it adapts very well to previously unseen organs and does not need to be re-trained for every patient. The high inferring speed makes this method useful for many applications such as surgical navigation and real-time simulation.
  @cite_30: Abstract Objectives Accurate reconstruction and visualisation of soft tissue deformation in real time is crucial in image-guided surgery, particularly in augmented reality (AR) applications. Current deformation models are characterised by a trade-off between accuracy and computational speed. We propose an approach to derive a patient-specific deformation model for brain pathologies by combining the results of pre-computed finite element method (FEM) simulations with machine learning algorithms. The models can be computed instantaneously and offer an accuracy comparable to FEM models. Method A brain tumour is used as the subject of the deformation model. Load-driven FEM simulations are performed on a tetrahedral brain mesh afflicted by a tumour. Forces of varying magnitudes, positions, and inclination angles are applied onto the brain's surface. Two machine learning algorithms—artificial neural networks (ANNs) and support vector regression (SVR)—are employed to derive a model that can predict the resulting deformation for each node in the tumour's mesh. Results The tumour deformation can be predicted in real time given relevant information about the geometry of the anatomy and the load, all of which can be measured instantly during a surgical operation. The models can predict the position of the nodes with errors below 0.3 mm, beyond the general threshold of surgical accuracy and suitable for high fidelity AR systems. The SVR models perform better than the ANN's, with positional errors for SVR models reaching under 0.2 mm. Conclusions The results represent an improvement over existing deformation models for real time applications, providing smaller errors and high patient-specificity. The proposed approach addresses the current needs of image-guided surgical systems and has the potential to be employed to model the deformation of any type of soft tissue.
  @cite_18: In this paper, an approach for 'virtual' real-time deformation of complex structures is developed. The approach combines the finite element method and Neural Network calculations to allow the user to perform interactive shape changes and view the resultant deformation changes in a virtual environment.A tennis racket and ball are used to illustrate the capability of the proposed method.Because real time deformation simulation is a time consuming repeated analyses, the neural networks are employed in this investigation as numerical devices for substituting the finite element code needed for the tennis racket and ball deformation.The input data for the artificial neural network are a set of parameters generated randomly (ball impact velocity, impact angle and the zone impact). The output data are the deformation of the ball strings and the impact force-feedback value.The work contribute toward the development of real virtual simulator which uses a haptic device that "feels" the feedback force generated by the deformation.
  @cite_14: This paper presents a new method for simulating the deformation of organ models by using a neural network. The proposed method is based on the idea proposed by [2] that a deformed model can be estimated from the superposition of basic deformation modes. The neural network finds a relationship between external forces and the models deformed by the forces. The experimental results show that the trained network can achieve a real-time simulation while keeping the acceptable accuracy compared with the nonlinear FEM computation.
  @cite_13: Surgical planners are used to achieve the optimal outcome for surgery. They are especially desired in procedures where a positive aesthetic outcome is the primary goal, such as the Nuss procedure which is a minimally invasive surgery for correcting pectus excavatum (PE) - a congenital chest wall deformity which is characterized by a deep depression of the sternum. The Nuss procedure consists of placement of a met al bar(s) underneath the sternum, thereby forcibly changing the geometry of the ribcage. Because of the prevalence of PE and the popularity of the Nuss procedure, the demand to perform this surgery is greater than ever. Therefore, a Nuss procedure surgical planner is an invaluable planning tool ensuring an optimal physiological and aesthetic outcome. We propose the development and validation of the Nuss procedure planner. First, a generic model of the ribcage is developed. Then, the computed tomography (CT) data collected from actual patients with PE is used to create a set of patient-specific finite element models (FEM). Based on finite element analyses (FEA) a force-displacement data set is created. This data is used to train an artificial neural network (ANN) to generalize the data set. In order to evaluate the planning process, a methodology which uses an average shape of the chest for comparison with results of the Nuss procedure planner is developed. Haptic feedback and inertial tracking is also implemented. The results show that it is possible to utilize this approximation of the force-displacement model for a Nuss procedure planner and trainer.
summary: The idea of using neural networks to estimate the outcome of an FEM simulation is not new. @cite_18 have used a fully connected net to predict the velocity and angle of a tennis ball after hitting a racket. A similar approach has been adopted by @cite_30 for predicting the movement of a tumor during brain surgery, but the displacement of the healthy tissue is not considered. @cite_13 estimate the displacement of a patient's rib cage, but only predict the displacement of surface nodes. In contrast, @cite_14 estimate the displacement of a full liver model using neural networks by superimposing basic deformation modes. These approaches show that neural networks can indeed be trained to estimate soft tissue behavior. However, they work with the data of a single patient, requiring re-training (and in some cases even re-designing) the network for every new patient. Additionally, they use the acting surface forces as input which are very difficult to obtain intra-operatively.