![](https://www.grad.ubc.ca/sites/default/files/styles/card_vertical_767/public/students/7x2a8576.jpg?itok=S9w_-ZZd&c=85ae7a11c9c5ed3c9c64fab1011e2992)
David Black
Doctor of Philosophy in Electrical and Computer Engineering (PhD)
Research Topic
Novel approaches to teleoperation and human computer interaction in medical robotics
Dissertations completed in 2010 or later are listed below. Please note that there is a 6-12 month delay to add the latest dissertations.
Ultrasound elastography is a medical imaging technique that quantifies tissue mechanical properties such as elasticity. It involves applying a force to the tissue, and measuring the resulting deformations to calculate tissue elasticity by solving an inverse problem. 3D imaging provides more spatial information on tissue deformation compared to conventional 2D frames. However, 3D imaging has several bottlenecks that limit its practicality in clinical settings, including a long data acquisition time that can introduce artifacts from patient or sonographer motion and a low frame rate that limits the applied force’s frequency range by the Nyquist frequency. A wider frequency range can enable a more comprehensive tissue modeling and characterization. Furthermore, most ultrasound elastography techniques measure displacement only in one axial direction, but it is important to consider that any force applied to the tissue leads to 3D tissue deformation due to tissue incompressibility. Therefore, it is crucial to explore new approaches to overcome these limitations and fully utilize the potential of 3D imaging in clinical applications. Shear wave absolute vibro-elastography (S-WAVE) is an elastography technique where an external vibration source generates steady-state mechanical vibrations inside the tissue. This thesis presents two novel techniques for high-frame-rate S-WAVE volumetric data acquisition as well as the algorithms to calculate elasticity volumes. The standard quasi-real-time technique using a wobbler transducer involves sweeping the transducer mechanically and acquiring 2D images at each location. The images are then reconstructed into a 3D volume which takes less than 2 s to collect 100 data volumes. Alternatively, using a matrix array transducer, we introduce the first S-WAVE method with real-time volumetric data acquisition which takes 0.05 s to collect 100 data volumes. We propose a novel method for estimating axial, lateral, and elevational displacements which in turn enables the use of the curl of the displacements in the elasticity reconstruction to reduce artifacts. By employing high volume rates, we extend the range of S-WAVE excitation to 800 Hz. The proposed method is validated using homogeneous and heterogeneous phantoms and in ex vivo bovine liver studies.
View record
The most common treatment for organ confined prostate cancer is radical prostatectomy (RP), where cancerous prostate is incised out. Nowadays, mostly the da Vinci system is used to do robot assisted laparoscopic radical prostatectomy (RALRP), providing improved dexterity and significantly faster patient recovery times. However similar to open surgery, the RALRP has higher numbers of reported positive surgical margins. This is potentially due to a surgeon’s effort to remove the cancer and preserving the healthy tissue, when the cancerous and non-cancerous boundaries are indistinguishable in the endoscope. Therefore, the objective of this thesis is to clearly display these boundaries and tumors, using Augmented/Mixed Reality (AR/MR) technology, utilizing imaging data such as, magnetic resonance imaging and ultrasound (US). The successful intra-operative AR/MR depends on primarily two things. First, the surgically compatible calibration steps, to map the imaging data correctly in camera image. Second, the visualization of the co-located data to give reliable depth of subsurface structures, which is imperative to patient safety. Unlike existing methods, in our first work, we propose a method that performs the required hand-eye and camera calibrations without using external markers during surgery. To further streamline the process, in another work, we use an optimization scheme to combine both the calibrations in a single step. The method allows to register the robotic data to the camera within minutes. Additionally, we presented an evaluation of a full AR system that registers the phantom US to the camera image. Next, we address the well-known problem of occlusion while visualizing the overlayed imaging data. For this, our deep learning-based method segments surgical instruments in the human RALRPs videos without using any labelled data. In our other two works we explore the color and motion parallax as depth cues to provide a reliable depth judgement. The usefulness of these are validated through user studies showing significantly better depth perception when using our methods. In conclusion, this thesis presents multiple methods to make AR/MR guidance feasible for RALRP by addressing two pressing challenges in the field of surgical AR, i.e. surgically compatible calibrations and reliable visualization of the registered data.
View record
This thesis addresses the problem of tracking surgical tissue using camera data in robotically assisted minimally invasive surgery. Millions of robotically assisted surgeries are performed yearly. These surgeries are performed by a surgeon who uses a teleoperation console. This console gives the surgeon a 3D view of their environment as they perform surgery. If we would like to overlay preoperative scans onto the surgeon's field of view using augmented reality, we must understand where the tissue is and where it is moving. Additionally, to enable automation of tasks, or remembering measurements made at tissue locations, such as a biopsy, we require robust tracking.In this thesis, we first perform an in-depth review of the field. We then propose multiple interlocking pieces to enable tissue tracking. To enable tracking salient features, we design a real-time keypoint descriptor that is trained in an unsupervised manner, demonstrating improved performance over classical methods. Afterwards, we propose a novel method that uses these keypoints to parameterize motion in 2D space using graph neural networks. We demonstrate the efficiency of this graph interpolation method. We then incorporate a temporal model into this graph interpolation paradigm. Finally, we extend our graph interpolation algorithm into 3D.In developing our methodology, we realized that there is an acute need for more datasets for quantification. We develop a novel and complete dataset for tissue tracking and mapping and release our dataset for public use by researchers. We close with a summary of important future work.
View record
Prostate cancer (PCa) represents a substantial health concern for men, and this thesis focuses on improving its diagnosis and treatment. The first contribution introduces a registration framework that localizes transperineal template-guided mapping biopsy (TTMB) cores in volumetric ultrasound (US) using only standard TTMB information. With an average target registration error of 1.2 mm and ≈97 s of registration time, this framework provides fast and accurate TTMB core localization. The pathology of these registered cores can serve as ground truth for training US-based PCa classifiers.To improve diagnosis, three US systems are developed to perform Shear Wave Absolute Vibro-Elastography (S-WAVE), a technique for quantitative tissue stiffness assessment. The first system, compatible with TTMB, employs multi-frequency transperineal excitation for the first time and correlates with magnetic resonance elastography (MRE) imaging at 96%. Clinical studies revealed a positive correlation between its measurements and PCa detected from histopathology. A PCa classifier, trained using clinical S-WAVE data taken with this system, achieved an AUC of 0.87±0.12. The second system is the first hand-operated 3D S-WAVE system with a transducer-mounted exciter designed for targeting systematic biopsies. When compared to MRE, it achieves a cross-correlation of 99% and 94% on quality assurance phantoms and in vivo human livers, respectively. In the third system, both S-WAVE and strain elastography are implemented for the first time in a commercial microUS system thus enabling the possibility of multi-parametric microUS imaging.For localized PCa, low-dose-rate prostate brachytherapy (LDR-PB) is an effective treatment where small radioactive seeds are permanently implanted within the prostate. The arrangement of these seeds is pre-planned manually, which is an iterative and time-consuming process. To automate this, two new methods based on conditional generative adversarial networks are proposed. Comparable results to the high-quality manual plans were achieved, demonstrating 98.9% coverage of the clinical target volume (CTV) receiving the minimum prescribed dose. Planning times are significantly reduced, with the proposed methods running approximately 7 and 400 times faster.
View record
Magnetic resonance elastography (MRE) is a biomechanical imaging tool that reconstructs the tissue elasticity map by capturing tissue displacement with magnetic resonance imaging (MRI). MRE has the unique advantage of capturing all three directional displacement components with a moveable and deep field of view. In addition, MRE offers the functionality to incorporate other MRI contrast seamlessly to extend MRE for advanced biomechanical models. However, MRE is limited by its long scanning time, low signal-to-noise ratio, and low resolution. This thesis focuses on iterative model-based reconstruction techniques with structured sparsity for faster MRE acquisition and robust elastogram reconstruction. A new optimization technique for elastogram reconstruction is proposed using the bi-convexity of the elastodynamic model and the alternating direction method of multipliers (ADMM). The proposed optimization technique allows easy integration of structured sparsity and is robust to noise and initialization. Four elastography reconstruction methods were implemented in this thesis with different models and regularization prior: (a) scalar 2D shear wave model (2D-ERBA), (b) scalar 3D shear wave model (3D-ERBA), (c) vector 3D elastodynamic wave model (ERSA), and (d) multifrequency vector 3D elastodynamic wave model (MERSA). The former two elastography reconstruction methods used a sparsity prior to the shear modulus, while the latter used sparsity prior on both shear modulus and displacement. These methods were extensively validated for in silico, in vitro, and in vivo datasets and compared with state-of-the-art methods. Experiments showed that MERSA provides higher robustness to noise, higher robustness to wavelength-to-voxel ratio, and higher accuracy for both elasticity and viscosity. A comparison study of diagnostic performance in detecting liver disease of scalar 2D shear wave model, scalar 3D shear wave model, and vector 3D elastodynamic wave model with MERSA implementation concludes the reconstruction part of this thesis. A bi-convex optimization-based displacement regularized compressed sensing (DRCS) method is proposed for fast MRE acquisition. The proposed method uses separate sparsity prior on the magnitude, phase and displacement to recover the MR signal from a highly undersampled k-space. We compare the performance of DRCS with other compressed sensing methods for highly undersampled data from in silico, in vitro and in vivo datasets.
View record
Neural networks (NNs) have reached remarkable performance in computer vision. However, numerous parameters and complex structures make NNs opaque to humans. The failure to comprehend NNs may raise serious issues in real-world applications. My research aims to explore the NN interpretability in diverse visual tasks from post-hoc explanation and intrinsic interpretability perspectives.Convolutional neural networks (CNNs) have outperformed humans in image classification. However, the logic of network decisions remains a puzzle. As such, we propose concept-harmonized hierarchical inference, a post-hoc explanation framework, to explain the decision-making process of CNNs. Firstly, we interpret layered feature representations of NNs with hierarchical visual semantics. Then we explain the NN feature learning as a bottom-up decision logic from low to high semantic levels in which a deep-layer decision is decomposed as a sequence of shallow-layer sub-decisions.With the evolution of virtual reality, researchers are focusing increasingly on inverse rendering: reconstructing a 3D scene from multi-view 2D images. In this field, NNs achieved superior performance in novel view synthesis and 3D reconstruction. For both tasks, learning a 3D representation from input views is the key process where prior methods separately designed a CNN-based single-view feature extraction and a pooling-based multi-view fusion. This incoherent design damages their intrinsic interpretability and performance. Therefore, we aim to design coherent, interpretable NNs that can adequately exploit knowledge of relationships from data. For novel view synthesis, we propose a unified Transformer-based neural radiance field (TransNeRF) conditioned on source views to learn a generic 3D-scene representation. TransNeRF explores deep relationships between the target-rendering view and source views. TransNeRF also improves intrinsic interpretability by enhancing the shape and appearance consistency of a 3D scene. In experiments, TransNeRF outperforms prior neural rendering methods, and the interpretation results are consistent with human perception.We reformulate 3D reconstruction as a sequence-to-sequence prediction and propose an end-to-end Transformer-based framework (EVolT). EVolT jointly explores multi-level associations between input views and the output volume-based 3D representation within our encoder-decoder structure. EVolT achieves state-of-the-art accuracy in multi-view reconstruction with fewer parameters (70% fewer) than prior methods. Experimental results also suggest the strong scaling capability of EVolT.
View record
This thesis addresses the problem of assisting humans in robot-assisted surgery. Our research investigates this problem using the following two approaches: (i) designing interfaces that facilitate human control of the surgical robotics platform, and (ii) developing autonomous systems to perform repetitive parts of the surgical task, allowing humans to focus on the more demanding ones.Following the first approach, we explored how to use multiple types of data to facilitate the surgeon’s control of surgical robots, such as motion data of expert surgeons, video data from an additional camera, and eye gaze data. The main application area of this approach is surgical training and skill assessment. Our results show that combining hand-over-hand and trial and error training approaches, based on expert motion data, enables trainees to balance both the speed and accuracy of performing tasks better than using only one of these approaches alone. Furthermore, our results show that a two-view system can improve both training and skill assessment, with its application in training showing the most promise, compared with the traditional case of using only a single view. In addition, we discovered a gaze-based phenomenon called “Quiet Eye” in multiple minimally invasive surgery settings. We report how this phenomenon changes with surgeons’ experience level and/or successful task completion. This opens the door to leverage this phenomenon in surgical training and skill assessment.Following the second approach, in the context of autonomous robotic surgery, we worked on automating tasks such as moving the surgical camera and suturing. To automate the surgical camera, we proposed a rule-based method that uses both the position and 3D orientation information from structures in the surgical scene. We tested the effectiveness of using our autonomous camera method in video-based surgical skill assessment. To automate surgical tasks such as suturing, we leveraged the surgical robot’s capability to move multiple arms in parallel to devise autonomous execution models that go beyond the humans’ way of performing these tasks. Our simulation experiments show that our proposed parallel execution models can lead to at least 40% decrease in the tasks’ completion time, compared with the state-of-the art ones.
View record
The progression of chronic liver disease, including hepatitis B, C, and non-alcoholic steatotic hepatitis, is closely associated with advancing fibrosis which stiffens the liver tissue over time. Magnetic resonance elastography is a quantitative imaging method that measures the liver tissue shear modulus over a volume and provides the most accurate imaging-based fibrosis staging results when compared to biopsy. While ultrasound-based elastography methods have been developed for liver fibrosis staging, they are mostly confined to providing measurements for a 1D or a 2D region of interest and with a limited imaging depth. In this thesis, a novel 3D ultrasound liver shear wave absolute vibro-elastography (S-WAVE) imaging technique is developed. With the aid of state-of-the-art 3D ultrasound hardware, 3D S-WAVE aims to provide quantitative and volumetric hepatic stiffness measurements comparable to magnetic resonance elastography. The new 3D S-WAVE utilizes a high element count matrix array transducer. With a modified 3D color power angiography imaging sequence, large field-ofview multi-frequency shear wave imaging could be achieved with a significantly shorter scan time. Learning-based automatic image segmentation and cross-modality registration techniques have also been developed to streamline the image processing pipeline and assist the comparison study between the S-WAVE and magnetic resonance elastography methods. Real-time shear wave imaging capability has also been enabled to improve the efficiency and repeatability of the S-WAVE exam. Validation studies with liver tissue phantoms and in vivo subjects have been designed and implemented using magnetic resonance elastography as the ground truth. The results showed that 3D S-WAVE had a high consistency with magnetic resonance elastography and outperformed conventional ultrasound transient elastography. The collected 3D S-WAVE data were also utilized to assess the performance difference between 2D and 3D elasticity reconstruction techniques. The results indicated that the 3D method effectively overcame the overestimation and the spatial bias issues which were commonly introduced by the 2D imaging method.
View record
This thesis focuses on the automatic analysis of radiological and pathological images for cancer detection and classification. It addresses ultrasound imaging for prostate and breast cancer, and histopathology analysis for prostate cancer, where we propose several classification approaches based on novel features and deep learning. The goal of this thesis is to develop machine learning methods to assist clinicians in diagnosis, prognosis, and treatment planning for patients.To tackle the inherent data heterogeneity in prostate cancer research, we develop a novel framework based on the generative adversarial network to discard extraneous information. For benign vs. malignant classification, it achieves area-under-the-curve of 93.4%, sensitivity of 95.1%, and specificity of 87.7%, respectively, representing significant improvements of 5.0%, 3.9%, and 6.0% compared to using heterogeneous data.We propose novel methods that improve prostate cancer classification and risk stratification using multi-stain digital histopathology. For classification, we demonstrate that: (1) other stain types (Ki67, P63) improve classification performance upon H&E; (2) even without the presence of Ki67 and P63, by mimicking the stain types, H&E stain can better report the presence and severity of prostate cancer.For risk stratification, our proposed risk stratification pipeline, integrating clinicopathologic data and learned image features from multi-stain digital histopathology, outperforms the currently most common grading system, the Gleason grading system, in predicting clinical outcomes such as metastasis-free and overall survival. Using our risk models, 3.9% of low-risk patients are reclassified as high-risk and 21.3% of high-risk patients are reclassified as low-risk. These results demonstrate our risk stratification pipeline’s potential to guide the administration of adjuvant therapy after radical prostatectomy.For breast cancer, we propose a novel automatic pipeline for data processing, feature extraction, feature selection, and classification using ultrasound data. Our best results (95% confidence interval, area-under-the-curve = 95%±1.45%, sensitivity = 95%, and specificity = 93%) outperform the state-of-the-art results using shear wave absolute vibro-elastography. Moreover, our study proposes novel directions in the field of elasticity imaging for tissue classification.All the proposed methods have been tested on held-out sets and have demonstrated promising results, which would be useful in future cancer diagnosis, prognosis, and patient management.
View record
Despite the rapid advancement in medical imaging technologies, health care systems in Canada as well as in many other countries still cannot provide patients with sufficient medical imaging resources due to the ever-growing demands, lack of imaging devices, and fully trained medical practitioners. What makes the situation worse are persistent and prevalent work-related injury situations for medical practitioners, especially sonographers. To help these problems, the overall objective of this thesis is to improve the user interface design for ultrasound machines and contribute to automated medical imaging-based diagnosis with the help of the gaze tracking technology.Multi-themed study and research were conducted to achieve the objectives. On one hand, for the interface design improvement, we started by analyzing the characteristics of gaze signals through statistical modelling. Then we surveyed among sonographers to understand their daily usage of the ultrasound machines. Also, we analyzed different kinds of ultrasound machines to understand common design patterns and suboptimal control logic. To improve the suboptimal control logic and observed usage difficulties on ultrasound machines, we designed and implemented a gaze tracking contingent ultrasound machine. For the validation of our design ideas and machine effectiveness, comparative user studies were conducted. Additionally, we enhanced the user experience by solving the gaze tracker accuracy deterioration problem during normal usage of the devices by proposing a disruption-free auto-recalibration method. On the other hand, for the automated medical image diagnosis, with the integration of the gaze tracking dataset, we proposed deep learning methods that not only can provide a label that predicts which kind of abnormality is observed in the medical image, but also some reasoning that can guide humans to understand how the prediction is made. Multi-task learning methods and neural network attention mechanisms were used for enhanced automated diagnostic performance and better interpretability of the deep learning models.Our study has demonstrated the usefulness of integrating gaze tracking with human-computer interaction and image understanding in the medical field. Future work can be done to further processing and quantify gaze tracking data.
View record
Breast cancer affects millions of women worldwide each year, and is responsible for hundreds of thousands of deaths. Mammography, the standard screening method for these cancers, underperforms for women with dense breast tissue, who account for half of all women under the age of 50. Given that these women are also at higher risk of developing breast cancer, there is a significant need for better screening methods to serve this population. In this thesis, we develop a breast imaging platform which combines several modalities that have been clinically shown to aid in the detection and staging of breast malignancies. This operator-independent, completely automated scan, simultaneously acquires B-mode ultrasound, absolute elasticity, Doppler flow, and photoacoustic tomography of the entire breast in 20 minutes. We describe the hardware and software components which comprise each of these imaging subsystems, and conduct a preliminary study testing the combined system by imaging a phantom which we designed to incorporate inclusions which are uniquely visible in either elasticity or photoacoustic imaging. The photoacoustic tomography system constitutes the most significant contribution, and as such is the primary focus of the thesis. We have designed, built, and tested a custom, fiber-based tissue illuminator to accommodate the unique scanning geometry of the automated breast ultrasound scanner upon which we have based our system. We have also developed a novel data reconstruction scheme which can account for the spatial non-uniformity of this illumination. We tested this system in vitro, as well as a purpose-built wire phantom. Finally, we developed a data processing pipeline which uses generative adversarial networks to improve the signal-to-noise ratio of our raw photoacoustic data, and implemented a state-of-the-art regularized reconstruction scheme to remove imaging and reconstruction artifacts. We tested this method using several phantoms, including an anatomically realistic blood vessel phantom generated from real breast imaging data.
View record
In this thesis, we researched design developments for multi-axis force sensing at the surgeon and the patient consoles of the da Vinci® classic system. A systematic survey on the force sensing literature in Minimally Invasive Surgery (MIS) was conducted. It summarizes the design requirements, compares different technologies, and lists the pros and cons of different locations for sensor integration. While more than 100 articles were published on MIS force sensing, no prior work that addresses force sensing at the surgeon console, without limiting its dexterity, was found. We propose modifications in the wrist’s yaw link of the da Vinci’s Master Tool Manipulator (MTM) for integration of a commercial 6-axis force-torque sensor. The new design does not change the original manipulator’s kinematics and its dexterity. Two example applications of the MTM’s impedance control and joystick control of the Patient Side Manipulator (PSM) were presented to demonstrate the successful integration of the force sensor into the MTM.The mechanical design, electronics hardware, and firmware and software architectures of a novel 6-axis optical force sensor are discussed. The mechatronic design features simple integration, no overload, low-noise, wide dynamic range opto-electronics, and signal conditioning, coupled with co-located digital electronics based on a Field Programmable Gate Array (FPGA) that samples all sensing channels synchronously, enabling very low noise displacement sensing with a resolution of 1.62 nm, low measurement signal latency of 100 μs, high measurement bandwidth of 500 Hz, and high data transfer rates over 11.5 kHz for transmission of six-axis transducer data to a host computer. The transducer’s resolution is better than 0.0001% of the full-scale.The optical force sensor was used for measuring the forces applied to the distal end of a da Vinci® EndoWrist® instrument by mounting it onto its proximal shaft. A new cannula design comprising an inner tube and an outer tube was proposed. A mathematical model of the sensing principle was developed and used for model-based calibration. A data-driven calibration based on a shallow neural network architecture is discussed. The proposed force-sensing requires no modification of the instrument itself; therefore, it is adaptable to different instruments.
View record
During prostate surgeries, there are critical structures around the prostate that should be preserved. Therefore, an additional intra-operative prostate cancer (PCa) imaging method is needed to help the surgeon localize the cancer. Photoacoustic (PA) imaging as an emerging imaging modality shows great potential to detect cancerous tissue. In this thesis, we focus on intra-operative PA imaging of the prostate using the da Vinci robotic system.Towards this objective, we developed a PA reconstruction technique that works in the presence of the challenges of the linear transducers. These challenges include the directivity effect of the transducer and limited-view PA imaging that cause the rank deficiency of the reconstruction system. Therefore, a sparse representation of the PA absorber distribution using the Discrete Cosine Transform was proposed. This sparse representation helps improve the numerical conditioning of the system of equations and reduces the computation time of the approach.In addition, we evaluated the feasible scanning configurations for intra-operative PA tomography (PAT) of the prostate. There are two ultrasound transducers that can be used in prostate PAT: a transrectal ultrasound (TRUS) transducer located posterior to the prostate, and a pick-up ultrasound transducer carried by the da Vinci robotic system and located anterior to the prostate. We proposed a PAT acquisition system that includes a da Vinci system controlled by the da Vinci Research Kit. The configurations using the pick-up and the TRUS transducers to perform intra-operative prostate PAT were investigated.Finally, we developed intra-operative prostate PA imaging using the da Vinci robotic system and a pick-up ultrasound transducer. We proposed a new approach in which the da Vinci robot is programmed to acquire trajectories in a shared control configuration with virtual fixtures; the pick-up transducer is manually controlled but virtual fixtures keep it parallel to a single tomography axis, and keep its translation fixed to a single plane normal to this axis. The surgeon controls the transducer motion on the tissue along this virtual fixture. This thesis confirms that intra-operative da Vinci robot-assisted PA imaging with a pick-up transducer is feasible.
View record
The kidney is a vital organ within the human body and improvements in the ability to characterize the kidney tissue can create benefits for patients with kidney tumors and for kidney transplant recipients. Often, changes in tissue health or development of cancer are manifested in changes in tissue structure that affect tissue elastic properties. For example, the cancerous tissue of renal cell carcinoma is stiffer than healthy kidney tissue, and the development of fibrosis, which impairs kidney function, also causes the tissue to become stiffer over time. These changes can be imaged with ultrasound elastography, a technique for quantitatively assessing tissue elasticity. If proven effective, elastography tissue characterization can replace biopsy.The ultrasound elastography method used in this thesis, called Shear Wave Absolute Vibro-Elastography, or SWAVE, measures the wavelength of induced steady-state multi-frequency mechanical shear waves to calculate tissue elasticity. SWAVE can employ standard ultrasound transducers that image the kidney though the skin above the organ, or custom miniaturized transducers that are placed directly on the surface of the organ during surgery. The accuracy of SWAVE is vastly improved by the use of 3D ultrasound data. We propose and evaluate 3D SWAVE imaging based on the use of a tracked intra-operative ultrasound transducer designed for use with the da Vinci Robot. Different tracking methods are evaluated for future intra-operative use. Elasticity images of tissue phantoms are obtained using interpolated 3D tissue displacement data acquired with the da Vinci robot and the intra-operative transducer. The use of tracked ultrasound transducer opens the way for introducing registered preoperative imaging, including elastography, to improve surgical guidance. Different methods of characterizing kidney tissue using SWAVE imaging are examined. The elastic and viscous properties are estimated kidney tissue ex-vivo. The effect of arterial pressure on the measured kidney elasticity is characterized. It was found that increasing input pressure increases the measured elasticity. Finally, ultrasound and ultrasound elastography are applied to kidney transplant recipients in-vivo to assess the level of fibrosis development. A preliminary study indicates that it is possible to transmit shear waves into the transplanted kidney and measure the elastic properties of the kidney tissue.
View record
Changes in tissue elasticity are correlated with certain pathological changes, such as localized stiffening of malignant tumours or diffuse stiffening of liver fibrosis or placenta dysfunction. Elastography is a field of medical imaging that characterizes the mechanical properties of tissue, such as elasticity and viscosity. The elastography process involves deforming the tissue, measuring the tissue motion using an imaging technique such as ultrasound or magnetic resonance imaging (MRI), and solving the equations of motion. Ultrasound is well suited for elastography, however, it presents challenges such as anisotropic measurement accuracy and providing two dimensional (2D) measurements rather than three dimensional (3D). This thesis focuses on overcoming some of these limitations by improving upon methods of imaging absolute elasticity using 3D ultrasound. In this thesis, techniques are developed for 3D ultrasound acquired from transducers fitted with a motor to sweep the image plane, however many of the techniques can be applied to other forms of 3D acquisition such as matrix arrays. First, a flexible framework for 3D ultrasound elastography system is developed. The system allows for comparison and in depth analysis of errors in current state of the art 3D ultrasound shear wave absolute vibro-elastography (SWAVE). The SWAVE system is then used to measure the viscoelastic properties of placentas, which could be clinically valuable in diagnosing preeclampsia and fetal growth restriction. A novel 3D ultrasound calibration technique is developed which estimates the transducer motor parameters for accurate determination of location and orientation of every data sample, as well as for enabling position tracking of a 3D ultrasound transducer so multiple volumes can be combined. Another calibration technique using assumed motor parameters is developed, and an improvement to an existing N-wire method is presented. The SWAVE research system is extended to measure shear wave motion vectors with a new acquisition scheme to create synchronous volumes of ultrasound data. Regularization based on tissue incompressibility is used to reduce noise in the motion measurements. Lastly, multiple ultrasound volumes from different angles are combined for measurement of the full motion vector, and demonstrating accurate reconstructions of elasticity are feasible using the techniques developed in this thesis.
View record
Surgical removal of the prostate gland using the da Vinci surgical robot is the state of the art treatment option for organ confined prostate cancer. The da Vinci system provides excellent 3D visualization of the surgical site and improved dexterity, but it lacks haptic force feedback and subsurface tissue visualization. The overall objective of the work done in this thesis is to augment the existing visualization tools of the da Vinci with ones that can identify the prostate boundary, critical structures, and cancerous tissue so that prostate resection can be carried out with minimal damage to the adjacent criticalstructures, and therefore, with minimal complications. Towards this objective we designed and implemented a real-time image guidance system based on a robotic transrectal ultrasound (R-TRUS) platform that works in tandem with the da Vinci surgical system and tracks its surgical instruments. In addition to ultrasound as an intrinsic imaging modality, the system was first used to bring pre-operative magnetic resonance imaging (MRI) to the operating room by registering the pre-operative MRI to the intraoperative ultrasound and displaying the MRI image at the correct physical location based on the real-time ultrasound image. Second, a method of using the R-TRUS system for tissue palpation is proposed by expanding it to be used in conjunction with a real-time strain imaging technique. Third, another system based on the R-TRUS is described for detecting dominant prostate tumors, based on a combination of features extracted from a novelmulti-parametric quantitative ultrasound elastography technique. We tested our systems in an animal study followed by human patient studies involving n = 49 patients undergoing da Vinci prostatectomy. The clinical studies were conducted to evaluate the feasibility of using these systems in real human procedures, and also to improve and optimize our imaging systems using patient data.Finally, a novel force feedback control framework is presented as a solution to the lack of haptic feedback in the current clinically used surgicalrobots. The framework has been implemented on the da Vinci surgical system using the da Vinci Research Kit controllers and its performance hasbeen evaluated by conducting user studies.
View record
Noninvasive detection and localization of prostate cancer in medical imaging is an important, yet difficult task. Benefits range from diagnosis of cancer, to planning and guidance of its treatment. In order to characterize cancer and evaluate its localization in volumetric images, such as ultrasound or magnetic resonance imaging (MRI), their spatial correspondence with the "gold standard" provided by histopathology must be established.In this thesis, we propose a general framework for a multi-slice to volume registration that is applied to register a stack of sparse, unaligned two-dimensional histological slices to a three-dimensional volumetric imaging of the prostate. The approach uses particle filtering that allows deriving optimal pose parameters of the slices in a Bayesian approach.We then propose a novel registration method between in vivo and ex vivo MRI of the prostate to facilitate its registration to histopathology. The method incorporates elasticity information, acquired by magnetic resonance elastography (MRE), to generate a patient-specific biomechanical model of the prostate and periprostatic tissue.Next, we propose a registration method between preoperative MRI and intraoperative transrectal-ultrasound. The method can be incorporated with a robotic surgical system to augment the surgeon's visualization during robot-assisted prostatectomy. We also study the use of elasticity-based registration of ultrasound elastography and MRE.We then present an image processing approach for enhancing MRE data. The approach employs registration to compensate for motion of patients during the scan to improve the accuracy of the reconstructed elastogram. A super-resolution technique is employed to increase the resolution of the acquired images by utilizing unique properties of MRE.Finally, we develop a theory for optimization-based design of motion encoding in MRE that allows reducing scanning time and increasing signal-to-noise ratio of elasticity reconstruction. We formulate the displacement estimation of the mechanical wave as an experimental design problem, by which we quantify performance of sequences, and optimize multidirectional designs.The proposed methods have been evaluated in simulations and on a diverse set of clinical data. Results may pave the way for a broader clinical deployment of elastography and elastography-based image processing.
View record
This work presents new approaches to in-vivo and ex-vivo human prostate cancer imaging using magnetic resonance elastography (MRE) – a method to non-invasively image tissue elasticity using magnetic resonance imaging (MRI). From a clinical perspective, stiffness correlates with underlying tissue disease processes and has been traditionally probed with palpation. Thus, diagnosis based on mechanical properties may have great implications in terms of staging of prostate cancer, monitoring disease progression, treatment planning and post treatment follow up. In MRE, mechanical shear waves generted by an external transducer are imaged using an MRI scanner. From the acquired wave field it is possible to reconstruct mechanical properties such as the elastic modulus based on the wave equation. In this work two MR compatible trans-perineal transducers are developed for imaging of the human prostate on a 3T MR scanner. A new MRI pulse sequence is also developed to acquire the three dimensional wave fieldinduced by these transducers. The methods are validated in quality assurance phantoms and volunteer repeatability studies. The system is used for a patient study and the results are compared to the gold standard (whole-mount histopathology marked with Gleason score). Similarly, a transducer is developed for ex-vivo prostate studies on a 7T MR scanner. After validation, prostate specimens of patients are examined and the results are compared to the Gleason score. The overall conclusion are: (i) trans-perineal excitation is well tolerated by the subjects, (ii) the transducers do not interfere with the MR acquisition, (iii) the three dimensional wave field are successfully captured using the new pulse sequence, (iv) phantom validation studies prove that the methods are in fact repeatable and that thestiffness values match with the manufacturer’s specifications, (v) patient motion and the standing wave pattern degrade the repeatability of the reconstructed images, (vi) the prostate gland stands out in the stiffness and shear strain images, (vii) the central gland and in particular the transition zone are stiffer than the peripheral zone, (viii) cancer could indeed be detected with MRE with an area under the receiver-operator-curve of approximately 0.7, and finally (ix) the chemical fixation process degrades the stiffness contrast.
View record
Prostate cancer is the most prevalent type of cancer among men. Accurate delineation and appropriate visualization of the prostatic region can greatly affect treatment of prostate cancer and has the potential to reduce some of the treatment side-effects. The main goal of this research is to develop a prostate segmentation tool which is suitable to replace manual delineation. Manual segmentation, the current standard in procedures such as low dose rate prostate brachytherapy, is tedious, time consuming and observer dependent. We propose a 3D semi-automatic segmentation tool to overcome these limitations. To show the clinical value of this method we perform extensive dosimetric evaluation on in-vivo ultrasound images. This tool is currently being clinically used as part of the prostate brachytherapy treatment procedure at the BC Cancer Agency.Ultrasound is the common modality for imaging the prostate. Although safe and simple to use, it can not always allow the prostate to be reliably delineated. Vibro-elastography is a relatively new imaging method which is used to characterize mechanical properties of tissue. We investigate the suitability of vibro-elastography for visualizing the prostate. We compare in-vivo B-mode ultrasound and vibro-elastograpy images with the gold standard MRI in terms of contrast, edge visibility and the shape and size of the gland as seen in these images. Based on our results we develop a 3D automatic prostate segmentation tool in which, in addition to B-mode, information from vibro-elastography images is used in an iterative model-based segmentation approach.We conclude this work by studying the visibility of cancer itself in vibro-elastography images. Areas suspected for cancer are manually marked on the images and the results are compared to the marked cancer in registered pathology slices. Our preliminary results show that vibro-elastography has the potential to be used for detecting prostate cancer; however, we suggest a combined use of various modalities or image types to improve cancer detection.
View record
Low dose rate prostate brachytherapy is one of the most effective treatments for prostate cancer currently available. It involves the implantation of approximately one hundred small radioactive sources, or ‘seeds’, into the prostate gland. This is accomplished by depositing the seeds transperineally via 16-30 long needles. In British Columbia, over three thousand patients have been treated since 1998 using this technique, and fewer than 10% have suffered a recurrence to date. One of the principal challenges in low dose rate prostate brachytherapy is the limited reliability with which precise doses to the prostate and surrounding organs can be achieved. This is due both to the difficulty in accurately delivering the seeds to their planned positions, as well as movement of the seeds in the post-implant period during the resolution of procedure-induced edema. This uncertainty can lead to undetected deficits in the dose necessary to control the cancer. As patients with more advanced disease are being considered for brachytherapy, these deficits may have greater consequences on oncological outcomes. Treatment uncertainty also increases the risk of side effects, which have the potential to be severe.The overall aim of this thesis is to improve the scope and accuracy with which the dose distribution of stranded seeds can be measured after implant. This involved the development of an algorithm to uniquely identify seeds in post-implant CT data, along with a method to determine their orientation to improve dosimetric accuracy. An analysis of the displacement and migration patterns of seeds in the interval during the resolution of prostate edema was also undertaken. These results identified dosimetric deficiencies and modes of seed loss which have the potential to be rectified by the use of implants containing seeds of non-uniform strength (‘mixed-activity’ implants). Such treatments use fewer needles and may also reduce the incidence of treatment related urinary morbidity. Although the concept of mixed-activity implants is not novel, the algorithm developed to identify seeds after implant enables the post-implant assurance of their dosimetric quality in a clinically feasible way. This thesis concludes with a study investigating the dosimetric benefits of mixed-activity implants.
View record
Tissue deformation is common during many medical interventions. An accurate simulation of these procedures necessitates accounting for tissue displacements by modeling tissue deformation and medical tool interaction. In minimally-invasive procedures, due to lack of visibility, physicians rely on haptic feedback and medical imaging to assess the immediate anatomical configuration and relative medical tool position. These procedures are often difficult to learn and therefore extensive training becomes essential.Computerized training systems offer an alternative to cadavers and training on patients. To accurately model the tissue deformation, most such systems require a mesh representation of the anatomy. To replicate the medical imaging feedback offered during procedures, a realistic image simulation approach is also needed. In this thesis, a novel energy-based meshing technique taking medical images and producing desirable meshes for the finite element method is introduced. This method employs an image-based discretization energy combined with a geometry-based element quality energy. The former promotes each mesh element to cover similar intensity image regions, while the latter ensures the element suitability for finite element simulation. A method that can mimic realistic B-mode ultrasound images under deformation is also presented in this thesis. This method first maps the pixels of an image from a deformed mesh configuration back to the nominal configuration, and then interpolates them in a B-mode voxel volume reconstructed a priori.Needle insertion is involved in several medical procedures. These percutaneous procedures will benefit significantly from advances in simulating needle-tissue interaction, for which a 3D model is proposed in this thesis. Simulating needle flexibility is achieved fast and accurately using a novel approach employing torsional springs. The needle insertion simulation with haptic feedback is presented for a training scenario for prostate brachytherapy, where simulated ultrasound images coupled with deformation are also displayed. A scheme to generate patient models for this system is also devised using both the conventional meshing techniques in the literature and the proposed variational meshing method.
View record
Tissue motion estimation in ultrasound images plays a central role in many modern signal processing applications, including tissue characterization, strain and velocity imaging, and tissue viscoelasticity imaging. Therefore, the performance of tissue motion estimation is of significant importance. Also, its computational cost determines if it can be implemented in real-time so that it can be used clinically. This thesis presents several efficient methods for accurate estimation of tissue motion using digitized ultrasound echo signals.First, sample tracking algorithms are presented as a new class of motion estimators. These algorithms are based on the tracking of individual samples using a continuous representation of the reference echo signal. Simulations and experimental results on tissue mimicking phantoms show that sample tracking algorithms significantly outperform common algorithms in terms of accuracy, precision, sensitivity, and resolution. However, their performance degrades in the presence of noise.To improve the performance of motion estimation in multi-dimensions, pattern matching interpolation techniques are studied and new interpolation techniques are presented. Simulation and experimental results show that, with small computational overhead, the proposed interpolation techniques significantly improve the accuracy and the precision of motion estimation in both 2D and 3D. Employing these techniques, real-time 2D motion tracking software is developed. Furthermore, the performance of the proposed 2D estimators is compared with that of 2D tracking using angular compounding. The results show that the proposed interpolation methods bring the performance of pattern matching techniques close to that of 2D compound tracking. Finally, angular compounding is combined with custom pulse sequencing and delay cancellation techniques to develop a system that estimates the motion vectors at very high frame rates (> 500 Hz) in real-time. The application of the system in the study of the propagation of mechanical waves for tissue characterization is also presented.
View record
Prostate brachytherapy is a widely used treatment of localized prostate cancer. Intra-operative dose feedback would bring many benefits to patients and healthcare practitioners. Detection of brachytherapy seeds and segmentation of prostate boundaries play key roles in dosimetry for prostate brachytherapy. However, seed detection and prostate segmentation using conventional B-mode transrectal ultrasound still remains a challenge for prostate brachytherapy, mainly due to the small size of brachytherapy seeds in the relatively low-quality B-mode ultrasound images and due to the poor contrast between the prostate gland and surrounding tissues, speckle noise, shadowing and refraction artifacts.In this thesis, a new method called the reflected power imaging is presented to enhance the visibility and imaging of implanted seeds. It directly measures the reflected energy of ultrasound radio-frequency signals without logarithmic compression. Based on this method, we propose a new solution for brachytherapy seed detection in a 3D reflected power image computed from ultrasound radio-frequency signals, instead of conventional B-mode images. Then implanted seeds are segmented in 3D local search spaces that are determined by α priori knowledge, e.g. needle entry points and seed placements in a pre-operative dosimetry plan. Needle insertion tracks are also detected locally by using the Hough Transform. Experimental results show that the proposed solution works well for seed localization in a tissue-equivalent ultrasound prostate phantom implanted according to a realistic treatment plan with 136 seeds from 26 needles.As the prostate is a firm organ relative to surrounding tissues, elastography is a potential imaging modality for the guidance of prostate brachytherapy. A dynamic ultrasound elastography method named vibro-elastography can provide more complete dynamic tissue description in terms of transfer functions and coherence functions. In this thesis, we develop fast computational algorithms and programs to implement vibro-elastography imaging in real time. Phantom experiments demonstrate that the vibro-elastography techniques produce stable and operator-independent strain images with high contrast-to-noise ratio. Furthermore, the software for a 3D vibro-elastography imaging system has been designed, implemented and used in the data collection. Over 15 patients have been scanned at the British Columbia Cancer Agency, Vancouver Centre, and the results are encouraging.
View record
Low dose rate prostate brachytherapy has emerged as a treatment option forlocalized prostate cancer. During prostate brachytherapy, tiny radioactivecapsules - seeds - are implanted inside the tissue using long needles. The quality of the treatment depends on the accuracy of seed delivery to their desired positions. Prostate deformation and displacement during insertion and lack of sufficient visual feedback complicate accurate targeting and necessitates extensive training on the part of the physician. Needle insertion simulators can be useful for physician training. In addition, insertion of theneedle with optimized parameters can compensate for prostate deformation,can decrease the targeting errors and, subsequently, can increase thepost-treatment quality of life of the patients. Therefore, needle insertionsimulation and path planning have gained a lot of attention in the research community in the past decade. Moreover, several robots have been designed for brachytherapy; however, they are yet to be coupled with proper needle insertion path planning algorithms.In this thesis, steps toward a path planning algorithm for needles are taken. An optimization method is proposed that updates the initial insertion parameters of a rigid needle, iteratively, based on the simulated positions of the targets, in order to reduce the error between the needle and several targets in a 3D tissue model. The finite element method is used in the needle insertion simulator. The simulator requires a model for the needle-tissue interaction. Therefore, an experimental method has been developed to identify the force model and the tissue elasticity parameters usingmeasurements of insertion force, tissue displacement and needle position.Ultrasound imaging is used to measure tissue displacement. Ultrasound is a common imaging modality in the operating room and does not need beads or markers to track the tissue motion. Therefore, the experimental method can be used in patient studies. The needle-tissue modeling method and insertion parameter optimization were validated in experiments with tissue-mimicking phantoms.In order to facilitate the accommodation of needle flexibility for future applications, three flexible needle models have been compared. Based on these comparisons, it was determined that an efficient and accurate angular spring model is best suited for future studies.
View record
Commonly used medical imaging techniques can render many properties of the anatomy or function, but are still limited in their ability to remotely measure tissue mechanical properties such as elasticity and viscosity. A remote and objective palpation function would help physicians in locating possible tumors or malignancies. The branch of medical imaging that characterizes tissues mechanical properties in a non-invasive manner has enjoyed increasing interest in the past two decades. The basic principle is to apply an excitation, such as tissue compression, to a region of interest and measure the resulting tissue deformation. Tissue mechanical properties can then be inferred from the observed deformation at multiple locations in the region, and the properties can be displayed as an image. If the excitation is dynamic, the deformation is considered as a motion field that varies in time and location over the region of interest. Ultrasound is particularly well suited for measuring motion fields due to its ability to image in real-time, low cost, low risk and ease-of-accessibility. The focus of this thesis is the estimation of the viscoelastic parameters such as Young's modulus, viscosity and relaxation-time. For this purpose, a motion estimation method is proposed to measure axial tissue displacements from the peak of the ultrasound radio frequency signals. The displacements can be further processed to identify the mechanical properties. Two methods were developed: the first one is based on a one dimensional Voigt's model of soft tissue and the second one is based on a finite element model. In the first method, a single frequency or wide-band excitation is applied to the tissue and the local relaxation-time is recovered from the phase difference between the strains or displacements. In this method, the elasticity can also be reconstructed from the magnitudes of the spectra. In the second approach, a novel dynamic finite element model is proposed for the incompressible soft materials. An inverse problem of viscoelasticity is solved iteratively to reconstruct the viscosity and elasticity based on a two or three dimensional model. The theoretical aspect of compressional elastography and longitudinal wave propagation is investigated. It is shown to be feasible to apply dynamic or transient compressional excitation to recover the dynamic properties of soft tissue.
View record
A system for vessel characterization aimed at detecting deep vein thrombosis (DVT) in the lower limbs has been developed and evaluated using ultrasound image processing, location and force sensors measurements, blood flow information and a protocol based on the current clinical standard, compression ultrasound. The goal is to provide an objective and repeatable system to measure DVT in a rapid and standardized manner, as this has been suggested in the literature as an approach to improve overall detection of the disease.The system uses a spatial Kalman filter-based algorithm with an elliptical model in the measurement equation to detect vessel contours in transverse ultrasound images and estimate ellipse parameters, and temporal constant velocity Kalman filters for tracking vessel location in real-time. The vessel characterization also comprises building a 3-D vessel model and performing compression and blood flow assessments to calculate measures that indicate the possibility of DVT in a vessel. A user interface designed for assessing a vessel for DVT was also developed.The system and components were implemented and tested in simulations, laboratory settings, and clinical settings. Contour detection results are good, with mean and rms errors ranging from 1.47-3.64 and 3.69-9.67 pixels, respectively, in simulated and patient images, and parameter estimation errors of 5%. Experiments showed errors of 3-5 pixels for the tracking approaches. The measures for DVT were evaluated, independently and integrated in the system. The complete system was evaluated, with sensitivity of 67-100% and specificity of 50-89.5%. System learnability and memorability were evaluated in a separate user study, with good results.Contributions include a segmentation approach using a full parameter ellipse model in an extended Kalman filter, incorporating multiple measurements, an alternate sampling method for faster parameter convergence and application-specific initialization, and a tracking approach that includes a sub-sampled sum of absolutes similarity calculation and a method to detect vessel bifurcations using flow data. Further contributions include an integrated system for DVT detection that can combine ultrasound B-mode, colour flow and elastography images for vessel characterization, a system interface design focusing on usability that was evaluated with medical professionals, and system evaluations through multiple patient studies.
View record
Theses completed in 2010 or later are listed below. Please note that there is a 6-12 month delay to add the latest theses.
Most current deep learning models for hematoxylin and eosin (H&E) histopathologyimage analysis lack the power of generalization to datasets collected from otherinstitutes due to the domain shift in the data. While graph convolutional neural networkshave shown significant potential in natural and histopathology images, theiruse in histopathology images has only been studied using a single magnification ormulti-magnification with late fusion.In this thesis, we study the domain shift problem with multiple instance learning(MIL) on prostate cancer datasets collected from different centers.First, we develop a novel center-based H&E color augmentation for cross-centermodel generalization. While previous work used methods such as random augmentation,color normalization, or learning domain-independent features to improvethe robustness of the model to changes in H&E stains, our method first augmentsthe H&E color space of the source dataset to color space of both datasets and thenadds random color augmentation. Our method covers the larger range of the colordistribution of both institutions resulting in a better generalization.Next, to leverage the multi-magnification information and early fusion with graphconvolutional networks, we handle different embedding spaces at each magnificationby introducing the Multi-Scale Relational Graph Convolutional Network (MSRGCN)as a novel MIL method. We model histopathology image patches and theirrelation with neighboring patches and patches at other magnifications as a graph.To pass the information between different magnification embedding spaces, we defineseparate message-passing neural networks based on the node and edge type.Our proposed color adaptation method improves the model performance on boththe source and target datasets, and has the best performance on the unlabeled target dataset compared to State-Of-The-Art (SOTA), showing promise as an approach tolearning generalizable features for histopathology image analysis.We also compare our MS-RGCN with multiple SOTA methods with evaluations onseveral source and held-out datasets. Our method outperforms the SOTA on all ofthe datasets and image types consisting of tissue microarrays, whole-mount slideregions, and whole-slide images. Through an ablation study, we test and show thevalue of the pertinent design features of the MS-RGCN.
View record
Visualization in robot-assisted minimally invasive surgery, performed with the da Vinci surgical system, is through a single stereo endoscope. The endoscope’s stereoscopic vision provides the surgeon with a sense of depth which is critical for surgery. Stereoscopic camera separation, or baseline, contributes to this perception and research in related fields has shown the benefits of an increased camera baseline. Minute surgical incisions, however, restrict current endoscopic baselines to small values. Furthermore, current clinical endoscopes have only four degrees of freedom within the surgical cavity, limiting the viewing angles provided. Restricted endoscopic movement, along with complex tumours, can make surgeries like partial nephrectomy (removal of kidney tumours) challenging. To overcome these limitations, alternate cameras have been proposed in the literature. One consistent drawback across the proposed research, however, is the lack of visual-motor alignment when working with these alternate camera views, bringing unnecessary mental strain to the surgeon.In this thesis, we present a stereoscopic "pickup" camera that has been designed to specifically address the endoscopic limitations outlined above. The camera has a special grasping interface that mates securely with a surgical instrument, and offers six degrees of freedom to provide a wider range of positions and angles to view the surgical site from. We also present a method to establish visual-motor aligned control of surgical instruments when viewing the surgical cavity through the "pickup" camera. Additionally, the placement of stereo cameras on the lateral side of the "pickup" camera's cylindrical body provides an opportunity to employ a larger camera baseline to improve the perception of depth.Two experimental user studies were conducted to validate the proposed system. First, the performance gained when visual-motor alignment is established with respect to the "pickup" camera is clearly exhibited with a 72.9% improvement in completion time and 80% improvement in accuracy. Secondly, the effect of an increased camera baseline on depth perception is studied. The results show that, over 28 subjects, depth perception was the strongest with a 20 mm baseline, which cannot be achieved with current endoscopic technology. The study also demonstrated that such large baselines, however, can be achieved with our "pickup" camera.
View record
Robot-assisted laparoscopic radical prostatectomy is a surgical operation where the entire prostate gland is removed. This complex procedure requires the surgeon to establish a fine balance between completely removing the cancer while sparing critical anatomy responsible for continence and potency. Much of the difficulty of this surgery lies in the inability for the surgeon to actively localize the tumours during the surgery and distinguish between cancerous tissue to be removed and healthy tissue to be left behind. This thesis details the design and development of an image guidance system for radical prostatectomy with the goal of improving patient outcomes by providing the surgeon with improved cancer localization. An image guidance system was proposed, consisting of pre-operatively segmented Magnetic Resonance (MR) images registered to intraoperative transrectal ultrasound rendered in a 3D virtual scene. This system builds upon prior work done by previous members of the lab who developed a TRUS robot which is able to perform a da Vinci-TRUS registration as well as a separate system for MRI-TRUS registration. The registered MRI and TRUS images are presented to the surgeon through the da Vinci surgical console's TilePro display system. Improvements to this prior system in this thesis include specifically: rendering the MRI and TRUS in a visual representation of the scene, an imaging pipeline framework for capturing video from multiple sources, improvements to the calibration between da Vinci and TRUS coordinate systems, an augmented reality overlay for the ultrasound image displaying the da Vinci instrument location, prostate and tumour boundaries and a real-time registration algorithm for tracking prostate motion over the course of a surgery was integrated into the main workflow. A series of experiments were conducted in order to validate the system. First, a phantom study was conducted to evaluate the accuracy of the entire registration process. Second, experiments were conducted to measure the runtime performance and latency of the application. Third, the accuracy of the real-time registration algorithm was tested. Lastly, the results of 15 of the ongoing surgical studies using the image guidance system are presented.
View record
Prostate cancer is the most prevalent form of male-specific cancer. Minimally invasive robot-assisted laparoscopic radical prostatectomy, or the surgical removal of the prostate using the da Vinci surgical system, has become the most common treatment for organ-confined prostate cancer. Compared to traditional open surgery and laparoscopic surgery, the da Vinci system provides stereoscopic vision andhigh tool dexterity while also minimizing risk of infection and patient recovery times. However, this is still a difficult procedure for the surgeon, as there is often a trade-off between complete removal of cancerous tissue and preservation of structures responsible for maintaining urinary and sexual function. Delineating between these structures and cancerous tissue in a visually homogeneous workspacerequires a high level of expertise.This thesis evaluates the feasibility of using transperineal ultrasound for image guidance to improve visualization of anatomical structures and cancer localization during robotic prostate surgery. The proposed image guidance system utilizes a matrix array ultrasound transducer capable of real-time imaging the prostate anatomy in 3D. A calibration method is detailed to register the ultrasound volumeto da Vinci robot kinematic frame. In dry lab experiments, calibration error was determined to be 0.84±0.49mm. Next, using a deformable finite element model-based deformable registration, annotated preoperative MRI is registered the ultrasound volume for intraoperative visualization of cancerouslesions. A virtual surgical scene containing the prostate is rendered in 3D to the da Vinci surgeon’s console, where the surgeon can use the surgical instruments to traverse the virtual prostate volume and view the axial MRI slices. To evaluate this registration pipeline from MRI to ultrasound to da Vinci system, a prostate phantom was used to test a user’s ability to locate targets in the registered prostatevolume. Overall target registration error was 2.55±0.94mm. To evaluate clinical feasibility, an exploratory study was conducted to determine the efficacy of transperineal imaging during standard robotic prostatectomy. Prior to surgery, the ultrasound transducer was placed on the perineum, and the patient anatomy, including prostate and surrounding structures, was monitored throughout the procedure. Results from four patient cases showed good delineation of the prostate and other anatomy.
View record
We are investigating the potential and the challenges of integrating eye gaze tracking support into the interface of ultrasound machines used for routine diagnostic scans by sonographers. In this thesis, we follow a user-centred approach by first conducting a field study to understand the context of the end user. As a starting point to a gaze-supported interface, we focus on the zoom functions of ultrasound machines. We study gaze-supported approaches for the often-used zoom function in ultrasound machines and present two alternatives, One-step Zoom (OZ) and Multi-step Zoom (MZ). A state-based analysis on the zoom functions in ultrasound machines is presented followed by a state-based representation of the gaze-supported alternatives. The gaze-supported state representation extends the manual-based interaction by implicitly integrating gaze input to OZ and offering a gaze-supported alternative to moving the zoom box in MZ. Evaluations of the proposed interactions through a series of user studies, seventeen non-sonographers and ten sonographers, suggest an increased cognitive demand and time on task compared to the conventional manual-based interaction. However, participants also reported an increased focus on main tasks using the gaze-supported alternative, which could offer benefit to novice users. They also report a lowered physical interaction as the gaze input replaces some functions of the manual input.
View record
Robot-assisted surgery allows surgeons to have improved control and visualization in minimally invasive procedures. Eye gaze tracking is a valuable tool for studying and improving the surgeon experience during robot-assisted surgery. Eye gaze information gives insight on how surgeons are interacting with surgical systems as well as their intentions during surgical tasks. This thesis describes the development of an eye gaze tracker for the da Vinci Surgical System. The eye gaze tracker is designed to track both the 2D and 3D eye gaze of a surgeon. It interfaces with the da Vinci Surgical System through the da Vinci Research Kit (dVRK) and Robot Operating System (ROS) frameworks. The use of the eye gaze tracker is demonstrated in two applications. Firstly, a motor control framework is designed to aid surgeons in moving surgical tools towards their point of gaze. A haptic force is applied to the da Vinci master manipulators to pull the surgeon's hands towards where they are looking. This framework is demonstrated on a full da Vinci Surgical System on dry lab tasks. Secondly, eye gaze information is collected from 7 surgeons performing realistic clinical tasks with the da Vinci Surgical System. A prediction model using a random forest classifier is built based on the eye gaze information and tool kinematic information in order to predict how and when surgeons move their camera. This behavioural model has applications in both surgeon training and endoscope automation.
View record
Liver fibrosis is a largely prevalent concern in Canada and world-wide, due to high rates of Hepatitis, fatty liver disease, alcoholism, as well as several other possible causes. It is currently diagnosed and staged by performing biopsies or by tissue elasticity measurements referred to as elastography. Elastography methods are a relatively new means of measuring the mechanical properties of soft tissue non-invasively by measuring and processing the propagation of shear waves through the body. The Robotics and Control Laboratory at the University of British Columbia has developed an elastography technique, Vibro-elastography, that can quantitatively measure soft tissue stiffness in real time. It has previously been applied to prostate and breast pathologies. It is now being developed and optimized for liver applications.To validate Vibro-elastography as a new diagnostic tool, a comparison study should be performed on a clinical population. This work sets out to lay out the prerequisites needed to implement a full clinical study. It starts out with a repeatability study using a tissue phantom to ensure repeatable results and compare our results to manufacturer stiffness values. In this work, we compare the precision of several different implementations of Vibro-elastography including the placement of the excitation source, data acquisition techniques and single versus multi-frequency excitation. Most of the implementations resulted in good, repeatable results, regardless of exciter placement. The quality of wave propagation deteriorated with depth as expected, but elasticity results remained repeatable even at deeper regions of interest. The parameters are selected and designed for the use on the liver.Finally, a registration pipeline and initial case trial has been presented as a suggested means of comparing the elastography data obtained using Vibro-elastography and any elastography measures that can be obtained from a magnetic resonance system. Using manual fiducial vessel markers and applying an Iterative Closest Point registration process results in a quick alignment of the ultrasound and MRI volumes with registration error less than 20 mm.
View record
This thesis describes the development of a heart motion compensation system on the da Vinci Research Kit for coronary artery bypass surgery. With this teleoperation robotic platform, minimally invasive surgery on a beating heart could be performed on an already clinically prevalent system. Semi-automation of the slave manipulators of the robot is introduced as they track the surface of the beating heart. The surgeons' regular teleoperation commands are superimposed on the automated trajectory. To achieve a virtually stabilized environment, a novel concept of maintaining the camera fixed relative to the heart target is proposed. The preliminary research question is whether the robot is capable of tracking the highly dynamic heart motion. System identification was performed on the seven degree of freedom da Vinci slave manipulators, and an open loop controller was developed. The controller is based on spectral line decomposition and the assumption of a periodic trajectory. It successfully commanded the slave manipulators to track an actual three dimensional heart trajectory with submillimetre error. Experiments were conducted with expert robotic users to evaluate surgeons’ ability to perform tasks on a moving target emulating the beating heart, with very promising outcomes. The number of missed targets decreased from 37% to 13% when compensation was enabled, the number of hit targets increased from 26% to 41%, and completion time decreased. A second generation system was developed which includes real time motion measurement commanding the robot. Results from user studies with expert surgeons performing bimanual suturing on moving targets with the new system support the motion compensation. They also show the significance of motion measurement errors. As an added safety, a virtual fixture was implemented to protect the heart from accidental collisions with the instrument tips. User studies were conducted to validate the efficacy of the fixture. To expedite controller development, an interface was developed between Matlab Simulink and the C++ code that runs the da Vinci Research Kit. This allows on-the-fly testing of controllers which could be designed and developed in the convenient Simulink environment. Future work will be include closed loop control, improved experiment design, and the incorporation of electrocardiogram signals.
View record
This thesis presents an automatic pathology (AutoPath) approach to detect prostatic adenocarcinoma based on the morphological analysis of high resolution whole mount histopathology images of the prostate. We are proposing a novel technique of labeling individual glands as benign or malignant exploiting only gland specific features. Two new features, the Number of Nuclei Layers and the Epithelial Layer density are proposed here to label individual glands. To extract the features, individual gland and nuclei units are segmented automatically. The nuclei units are segmented by employing a marker-controlled watershed algorithm. The gland units are segmented by consolidating their lumina with the surrounding layers of epithelium and nuclei. The main advantage of this approach is that it can detect individual malignant gland units, irrespective of neighboring histology and/or the spatial extent of the cancer. Therefore, a more sensitive annotation of cancer can be achieved by the proposed AutoPath technique, in comparison to the current clinical protocol, where the cancer annotation is performed at the regional macro level instead of glandular level technique.We have also combined the proposed gland-based approach with a regional approach to perform automatic cancer annotation of the whole mount images. The proposed algorithm performs the task of cancer detection in two stages: at first with pre-screening of the whole mount images in a low resolution (5x), and then ii) a finer annotation of the cancerous regions by labeling individual glands at a higher magnification (20x). In the first stage, the probable cancerous regions are classified using a random forest classifier that exploits the regional features of the tissue. In the second stage, gland specific features are used to label individual gland units as benign or malignant. The strong agreement between the experimental results and the pathologist's annotation corroborates the effectiveness of the proposed technique. The algorithm has been tested on 70 images. In a 10-fold cross validation experiment it achieved average sensitivity of 88%, specificity of 94% and accuracy of 93%. This surpasses the accuracy of other methods reported to date.
View record
The use of registered intra-operative to pre-operative imaging has been proposed for many medical interventions, with the goal of providing more informed guidance to the physician. The registration may be difficult to carry out in real-time.Therefore, it is often necessary to track the motion of the anatomy of interest in order to maintain a registration.In this work, a surface based Extended Kalman Filter (EKF) framework is proposed to track a tissue surface based on temporal correspondences of 3D features extracted from the tissue surface. Specifically, an initial 3D surface feature map is generated based on stereo matched Scale Invariant Feature Transform (SIFT) feature pairs extracted from the targeted surface. For each consecutive frame, the proposed EKF framework is used to provide 2D temporal matching guidance in both stereo channels for each feature in the surface map. The 2D feature matching is carried out based on the Binary Robust Independent Elementary Feature (BRIEF) descriptor. If the temporal match is successful in both stereo channels, the stereo feature pair can be used to reconstruct the feature location in 3D. The newly measured 3D locations drive the EKF update to simultaneously estimate the current camera motion states and the feature locations of the 3D surface map. The framework is validated on ex vivo porcine tissue surface and in vivo prostate surface during a da Vinci radical prostatectomy. The peak and mean fiducial errors are 2.5 mm and 1.6 mm respectively.Compared to other methods, the surface based EKF framework can provide a reliable 2D feature matching guidance for each feature in the 3D surface map. This maintains a chance to relocate a feature that was lost for a significant period of time. Such a surface based framework provides persistent feature tracking over time, which is crucial to drift free surface tracking. With implementation on a Graphic Unit Processor (GPU), real time performance is achieved.
View record
Prostate cancer has been identified as a ubiquitous threat to the well being of NorthAmerican men living past their fourth decade. An accurate diagnosis, enablingthe selection of an appropriate treatment regime, is a key component to diseasemanagement. The current gold standard for diagnosis is the transrectal ultrasoundguided prostate biopsy procedure, where predefined templates are used to select tissuesample sites. Unfortunately, this procedure is incapable of producing reliableresults; causing multiple repeat biopsies to become common practice and motivatingmany men, in the face of uncertainty, to choose unnecessarily sever treatmentoptions with undesirable side effects.Elastography has shown great potential in its ability to detect cancerous tissue,and may enable for the current systematic sample site selection biopsy proceduretechnique to be replaced by a targeted biopsy approach. This transition promiseshigher cancer yields and improved diagnostic reliability, while at the same time decreasingprocedural side effects by reducing the number of required core samples.The objective of this thesis has been to integrate the elastography imagingmodality into a standard prostate biopsy system. This system may then act as aresearch platform for determining the feasibility of an elastography based, targetedprostate biopsy procedure.Realization of this objective required the development of four primary components.First, a tissue excitation mechanism, developed outside of this thesis, wasboth analyzed for performance capability and augmented in order to improve adesign limitation necessitating frequent maintenance. Second, a plastic bracket,enabling an unobtrusive rigid coupling between the excitation mechanism and ultrasoundprobe, was designed so that excitation forces may be transmitted into the prostate tissue during the standard biopsy procedure. Third, a sensor was designedwhich is capable of accurately detecting biopsy needle insertion depth by trackingbiopsy gun position using an optical absolute position sensor. And fourth, codewas developed for enabling an evaluation of system performance by performingelasticity measurements. This code was used to process elastography data, collectedfrom phantom and human subjects, in order to obtain a preliminary systemvalidation.
View record
Errors in seed placement during low dose rate prostate brachytherapy can resultin over-treating healthy tissue and/or under-treating cancer cells. In a standardtreatment procedure, seeds are implanted according to a planned seed distribution.This pre-operative plan (pre-plan) is created using an ultrasound volume scan takenabout two weeks earlier. Errors in seed placement can occur due to changes inprostate structure during those two weeks, and from seed displacement during andafter the actual operation.This thesis presents methods of seed localization that are suitable for both postoperativeand intra-operative use. The techniques can be applied to the imagingmodalities used in the current operation setup to implement a method of intraoperativeplanning. This involves using Transrectal Ultrasound (TRUS) and C-armX-ray Fluoroscopy (fluoro) data to monitor the seed positions relative to the currenttarget volume during an operation.Towards this goal, an automatic method of assigning seeds to their correspondinginsertion needle tracks has been developed to match seeds between modalitiesso that seed displacements can be computed. This method can be applied tomeasure intra-operative misplacement, by comparing the desired positions to theactual positions computed from fluoro data, or post-implant movement, comparingthe fluoro seed positions to those from post-implant Computed Tomography (CT)data. For the intra-operative and post-implant data, 99.31% and 99.41% of theseeds were correctly assigned, respectively. An average intra-operative seed displacementof 4.94±2.42 mm and a further 2.97±1.81 mm of post-implant movementis measured. This information reveals several directional trends and can beused to preemptively correct the pre-operative plan (pre-plan).
View record
This thesis describes a system for intraoperative transrectal ultrasound imaging inrobotic-assisted laparoscopic radical prostatectomy, and related image registrationwork.First, a novel method for registering three-dimensional ultrasound data to anexternal coordinate frame is presented. The method uses a registration tool pressedagainst an air-tissue boundary to provide common target points in the the ultrasound frame and the external frame. This method has two applications in our system: registering the ultrasound data captured by the system to a laparoscopic stereo camera to allow augmented-reality style overlays in laparoscopic or robotic surgery, and registering the system to the da Vinci Surgical System so the ultrasoundimaging arrays can automatically track the da Vinci tools during surgery. In an initial feasibility study, the method was used to register a mechanical three-dimensional ultrasound transducer to high-disparity stereo cameras through a tissue phantom. Average registration error was found to be 1.69 ± 0.60 mm. Accuracy of localizing ultrasound fiducials pressed against an air-tissue boundary was found to range from 0.54 mm to 1.04 mm. In a second study, the method was used to register three-dimensional transrectal ultrasound data to a da Vinci stereo endoscope. In this study, fiducials imaged at multiple registration tool positions were incorporated into a single registration. Registration error imaging through a tissue phantom ranged from 3.85 ± 1.76 mm using one registration tool position to 1.82 ± 1.03 mm using four positions. Registrationerror imaging through an ex-vivo porcine liver tissue sample ranged from 2.36 ± 1.01 mm using one registration tool position to 1.51 ± 0.70 mm using four positions.
View record
TransRectal Ultrasound (TRUS) is used for image guidance during prostate biopsy and for treatment planning of brachytherapy due to low cost and accessibility in operating room. However, tumors have better visibility in Magnetic Resonance (MR) images. The fusion of TRUS and MR images of the prostate can aid with the diagnosis and treatment planning for prostate cancer and with post-brachytheraphy quality assurance. We developed a 3D deformable registration method using the segmentations obtained from TRUS and MR images and a biomechanical model that employs stiffness values derived from elastography. The segmented source volume is meshed and a linear finite element model is created for it. This volume is deformed to the target image volume by applying surface forces computed by assuming a negative relative pressure between the non-overlapping and the overlapping regions of the volumes. This pressure drives the model to increase the volume overlap until the surfaces are aligned. We tested our algorithm on prostate surfaces extracted from postoperative MR and TRUS images for 14 patients and pre-operative MR and TRUS images for 4 patients, using a model with elasticity within the range reported in the literature for the prostate. We used three evaluation metrics for validation: the Dice Similarity Coefficient (DSC) (ideally equal to 1.0), the volume change in source surface during registration, and the Target Registration error (TRE) defined as the mean distance between landmarks such as urethrae and calcifications. For post-operative images, we obtained a DSC of 0.96±0.02 and a TRE of 1.5±1.4mm. The change in the volume of the source surface was 1.5±1.4%. For pre-operative images, we obtained the DSC of 0.96±0.01 and a TRE of 1.3±0.8mm. The change in the volume of the source surface was -0.9±0.2%. Our results show that this method is a promising tool for physically-based deformable surface registration. We also used our technique to register ultrasound strain images to free mount histo-pathology images with the goal of correlating cancer with areas of low strain. This was done using relative stiffness values derived from vibroelastography data. We also performed Computed Tomography (CT) and Ultrasound (US) kidney surface registration using this technique.
View record
In the past two decades, there has been considerable research interest in medical image registration during surgery. The overlay of medical images over the images from a surgical camera allows the surgeon to see sub-surface features such as tumor boundaries and vasculature. Ultrasound imaging is a prime candidate for medical image registration, as it is a real-time imaging modality and therefore is commonly-used for intraoperative surgical guidance. Prior technologies that attempted ultrasound-based registration have used external trackers in order to establish a geometric correspondence between the surgical cameras and the ultrasound probes; this requires probe and camera calibration, which is time-consuming, requires additional equipment, and adds additional sources of error to the registration.Another problem is how to maintain a registration between the ultrasound image and the underlying tissues, since tissues will move and deform from patient breathing and heartbeat, and from surgical instrument interaction with tissues. In order to overcome this, the underlying tissue should be tracked, and previously acquired ultrasound images should be registered and moved with the tracked tissue. Prior work has had limited success in providing a real-time solution for estimating local tissue deformation and movement; furthermore, there has been no work in estimating the accuracy of maintaining a registration --- that is, the accuracy of the registration after having been moved with the tracked tissue. In this work, we establish an image registration method between ultrasound images and endoscopic stereo-cameras using a novel registration tool; this method does not require external tracking or ultrasound probe calibration, thus providing a simple method for performing a registration. In order to maintain an image registration over time, we developed a tissue tracking framework. Its key innovation is in achieving real-time tracking of a dense tissue surface map. We use the STAR detector and Binary Robust Independent Elementary Features and compare their performance to prior tissue feature tracking methods, showing that they perform significantly faster while still managing to track the tissue at high densities. Experiments are performed on ex-vivo bovine heart, kidney, and porcine liver tissues, and initial results show that registrations can be maintained within 3 mm.
View record
Elastography is a method of imaging the viscoelastic properties of soft tissue and similar media. It can be used for identifying various forms of cancer and fibrosis in soft tissue. Elastography can thus aid in diagnosis of such diseases, and in treatment procedures, such a biopsy needle guidance. Acoustic radiation force (ARF) imaging is one form of elastography whereby the ARF of focused ultrasound displaces the medium at an internal, localised position. The same ultrasound system can be used to monitor the displacement, from which the viscoelastic properties can be recovered.This work presents a new ARF imaging method, called axial relaxation imaging, that uses transfer function techniques. The relaxation response at a point in the medium is monitored after a period of ARF application. By assuming the response corresponds to a negative step response of a linear system, the relative force-displacement transfer function is computed. This is then related to a Voigt model over a range of frequencies to obtain a relative elastic parameter and a frequency-dependent viscous parameter governed by a power law.The method was applied to homogeneous phantoms made with different gelatine concentrations. Relative elastic parameters of 1, 2.3 and 4.4 and relative viscous parameters of 1, 1.8 and 3.0 were obtained for 2, 3 and 4 wt% gelatine concentrations, respectively, with consistent results between phantoms of the same type and agreement with values from other estimation techniques. The viscous power law frequency dependencies were governed by flow index values of –0.10, –0.14 and –0.18, respectively. The good separation between parameters in the results shows the method holds potential for application to tissue characterisation.
View record
The conventional prostate brachytherapy approach is limited by needle positioning accuracy, needle trajectory option, and prostate motion and deformation between the pre-operative volume study and the seed implant procedure. These limitations increase the risks of post implant complications. In this thesis we develop a robotic needle guide to improve prostate brachytherapy needle placement accuracy and trajectory option as well as a pre-operative to intra-operative prostate volume registration algorithm to address the issue of prostate motion and deformation.Our four degrees of freedom robot provides X-Y axes translational accuracy of 0.12 and 0.1 mm compared to the 5 mm accuracy of the standard needle guide. The robot also provides yaw and pitch angulations with 0.05 degrees accuracy which can be used to reach prostate regions blocked by pubic arch interference. The robot is adaptable to conventional brachytherapy apparatus without adding the clinical procedure time and can be used manually in the case of electronic control failure.The registration approach is based on fitting prostate surfaces into ellipsoids. Pre-operative and intra-operative sagittal view-based volume data are contoured using a novel semi automatic sagittal view-based segmentation algorithm. The resulting contours are fit into ellipsoids whose parameters - centers, orientations, and radii - are used to calculate the registration matrix. The accuracy of the registration algorithm was compared with Optotrak measurement as the gold standard and with the Iterative Closest Point (ICP) algorithm. The result shows that the orientation of the ellipsoid fit is sensitive to user initialization points causing up to 5 mm translational errors and 5.5 degrees angular error. The comparison with ICP shows that the ellipsoid fitting based algorithm is faster but less accurate.
View record
If this is your researcher profile you can log in to the Faculty & Staff portal to update your details and provide recruitment preferences.