|
 
 |
| ORIGINAL ARTICLE |
|
| Year : 2016 | Volume
: 18
| Issue : 5 | Page : 732-734 |
|
Augmented reality assisted surgery: a urologic training tool
Ryan M Dickey1, Neel Srikishen1, Larry I Lipshultz1, Philippe E Spiess2, Rafael E Carrion2, Tariq S Hakky1
1 Scott Department of Urology, Baylor College of Medicine, Houston 77030, USA
2 University of South Florida, Department of Urology, Tampa 33606, USA
| Date of Submission | 07-May-2015 |
| Date of Decision | 30-Jul-2015 |
| Date of Acceptance | 10-Sep-2015 |
| Date of Web Publication | 27-Nov-2015 |
Correspondence Address:
Tariq S Hakky
Scott Department of Urology, Baylor College of Medicine, Houston 77030
USA
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/1008-682X.166436
Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass ® optical head-mounted display to train urology residents in how to place an inflatable penile prosthesis. We use the phrase Augmented Reality Assisted Surgery to describe this novel application of augmented reality in the setting of surgery. The application demonstrates the steps of the surgical procedure of inflatable penile prosthesis placement. It also contains software that allows for detection of interest points using a camera feed from the optical head-mounted display to enable faculty to interact with residents during placement of the penile prosthesis. Urology trainees and faculty who volunteered to take part in the study were given time to experience the technology in the operative or perioperative setting and asked to complete a feedback survey. From 30 total participants using a 10-point scale, educational usefulness was rated 8.6, ease of navigation was rated 7.6, likelihood to use was rated 7.4, and distraction in operating room was rated 4.9. When stratified between trainees and faculty, trainees found the technology more educationally useful, and less distracting. Overall, 81% of the participants want this technology in their residency program, and 93% see this technology in the operating room in the future. Further development of this technology is warranted before full release, and further studies are necessary to better characterize the effectiveness of Augmented Reality Assisted Surgery in urologic surgical training. Keywords: andrology; augmented reality assisted surgery; Google glass; inflatable penile prosthesis; optical head mounted display; surgical education
How to cite this article:
Dickey RM, Srikishen N, Lipshultz LI, Spiess PE, Carrion RE, Hakky TS. Augmented reality assisted surgery: a urologic training tool. Asian J Androl 2016;18:732-4 |
How to cite this URL:
Dickey RM, Srikishen N, Lipshultz LI, Spiess PE, Carrion RE, Hakky TS. Augmented reality assisted surgery: a urologic training tool. Asian J Androl [serial online] 2016 [cited 2021 Nov 28];18:732-4. Available from: https://www.ajandrology.com/text.asp?2016/18/5/732/166436 - DOI: 10.4103/1008-682X.166436 |
| Introduction | |  |
The earliest known surgical operation in humans was decompressive trepanation of the skull, performed in the Neolithic era. [1] Surgery continued to evolve with the advent of anesthesia, antiseptics, and X-ray in the 19 th century, allowing for more complex and precise procedures, as well as decreased mortality from sepsis. [2],[3] Novel use of technology has driven surgical advancement, from implantable technology such as the total artificial heart, cochlear implant, and inflatable penile prosthesis (IPP) to surgical techniques such as endoscopy and robotics. [4],[5],[6],[7],[8] The current state of endoscopic and robotic surgery has rendered many once open procedures to minimally invasive techniques. [4],[5] As the surgical evolution continues, new technologies will become more ubiquitous in the operating room, for both patient and surgeon.
Augmented reality (AR) is defined as a "technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view." [9] AR is widely used in aeronautics to assist pilots in takeoff, maneuvers, and landing; some have suggested a similar embrace of this technology in the field of surgery. [10] Recent use of AR in minimally invasive surgery has resulted in the creation of hybrid image-guided surgery using endoscopic and robotic video feeds. A dedicated institution has been developed around image-guided hybrid therapies. [10] However, the delivery of AR in open surgery will require an alternative technology for the surgeon's interface.
Optical head-mounted displays (OHMDs) and specifically Google Glass's® wearable technology (Google Inc., Mountain View, California) have shown to be an effective tool in both the clinical and operative setting and can be used as an AR interface for surgery. [11] OHMDs have demonstrated to help anesthesiologists maintain awareness of the patient's status in a distracting operating theater. [12],[13] AR is an emerging open surgical tool in education and is effective in orthopedic surgery in the creation of a composite view of student and teaching surgeon. [14] Google Glass has been studied in other surgical fields including plastic surgery and Mohs surgery. [15],[16] In urology, a primitive form of AR has been implemented in robotic partial nephrectomy. [17],[18] To date, there is no study of the use of AR or Google Glass in open urologic surgery. In this pilot study, our aim is to determine the feasibility of implanting AR in the setting of an OHMD that could be built for open urologic surgery not only as a surgical assistant but also as a surgical training tool in the placement of IPP. This is a pilot study to look at the use of Google Glass as an Andrologic training tool. We use the phrase Augmented Reality Assisted Surgery (ARAS) to describe this novel application of AR in the setting of urologic surgery.
| Materials and Methods | | |
After an IRB was obtained, Google Glass with software version XE 22.1 was purchased from Google during the beta testing phase. The device was loaded with a developed glass application, which contained a menu ([Figure 1]) with several options including a demonstration on how to place an IPP from start to finish in the penoscrotal approach, the most commonly used surgical approach for this procedure. [19] It also contained software that allowed for detection of interest points using an optical display camera from the OHMD output feeds to allow faculty to interact with residents during placement of the penile prosthesis ([Figure 2]). In the operating room, the OHMD is worn as a pair of glasses, with a monocular refractive lens over the right eye projecting input to the surgeon. The application is launched via voice commands or by tapping a menu display on the side of the Google glass. Once the application has loaded, the user is presented with a menu. Menu options include: how to use Google Glass, a step-by-step placement video of IPP placement in the penoscrotal approach, an interactive video display, and a section about the authors and build of the application. | Figure 1: ARAS menu screen shot from Google Glass application. ARAS: Augmented Reality Assisted Surgery.
Click here to view |
 | Figure 2: ARAS application screen shot demonstrating detection of interest points using an optical display camera from the OHMD output feeds with faculty interaction during IPP placement. ARAS: Augmented Reality Assisted Surgery; OHMD: optical head-mounted displays.
Click here to view |
In the first component of the application, video footage corresponding to the sequential steps of the IPP procedure was projected onto the live view of the patient in real time. This allowed trainees to visualize specific steps prior to completing them. The second component of the application included detection software that streamed live OR footage to a remote attending physician. The attending physician could then interact with the surgical trainee by moving a cursor highlighting areas of interest to explain details of the operation in depth. Consent was obtained to film sequences of the operations from patients undergoing IPP placement.
Study participants included urology residents and faculty at two institutions who volunteered to experience the technology and application and provide feedback. Participants were shown how to navigate Google Glass using either voice-command or tapping gestures and were given time to wear the Google Glass in either the operative or perioperative setting of IPP placement. Participants were then given a feedback survey to complete regarding the effectiveness and interest in the device and software. Survey questions included both quantitative and free text options ([Figure 3]). Quantitative questions asked participations to scale their perceptions to survey questions from 1 to 10, with 1 being "not at all" and 10 being "always". Other questions included a binary yes or no response with option to input free text. The surveys were then stratified into trainee and faculty categories and calculated to determine the mean rating for quantitative questions. Free text responses were analyzed for key words and grouped as closely as possible.
| Results | | |
A total of 30 participants volunteered to take part in the study. Of these, 10 were urology faculty members in academic urology programs, and 20 were either urology residents or fellows. Out of a 10-point rating scale, total participants rated the educational usefulness of the application as 8.6. Ease of navigation of the device and application was rated 7.6. The likeliness to use the technology if made available was rated 7.4. The technology was also rated if perceived as too distracting, with overall rating of 4.9 ([Table 1]). For the recommendation to implement the technology into respective training programs, a total of 26 responses were received (4 nonresponses), with 21/26 (81%) participants responding in the affirmative. For the belief that Google Glass has a role in the operating room, a total of 28 responses were received (2 nonresponses) with 26/28 (93%) responding in the affirmative. For the consideration of using hands-free Google Glass in future practice, a total of 28 responses were received (2 nonresponses) with 20/28 (71%) indicating the affirmative. For the belief that the application bettered understanding of the surgical procedure and anatomy during IPP placement, 16 responses were received with 9/17 (53%) responding in the affirmative ([Table 2]). | Table 1: Mean results of ARAS experience survey from urologic faculty and trainees using 10-point rating scale
Click here to view |
 | Table 2: Results of ARAS experience survey from urologic faculty and trainee respondents to yes-no questions
Click here to view |
When stratified between trainees and faculty, educational usefulness was found to be 8.7 versus 8.4, respectively. Ease of navigation was found to be 7.5 versus 7.9, respectively. Likelihood to use was found to be 7.4 versus 7.5, respectively. Distraction in the OR was found to be 4.7 versus 5.4, respectively ([Table 1]).
Specific comments given by participants recommended the inclusion of voice activation, patient and staff educational components for the procedure, video recording in the OR instead of streaming only, the ability to pull up patient imaging from while operating, and magnification.
Our novel study has several limitations. It is limited because the survey has not been validated and that we have a limited number of responses to the survey. Additionally, our surgical training tool was only useful in training one type of open urologic surgery. AR itself has limitations, as much of the surgical programming has yet to be done, and this technology is continuing to evolve. Another concern may be surgeon over reliance on AR, creating an intraoperative conflict as the surgeon's decision may not always be in line with the AR software. Limitations of the current Google Glass technology include limited battery life, overheating of the battery, and cumbersome software integration.
| Discussion | | |
As technology continues to permeate into every aspect of our lives, we look to integrate technology into urologic surgery and surgical training. Surgeons currently perform procedures after years of education and human surgical training to safely navigate through anatomical planes of the human body. The concept of AR, where a live view of the environment, in this case, a surgical operation, is supplemented by computer-generated sensory input using an OHMD, is the next evolutionary step in open urologic surgery and is an important training tool. Our goal is to introduce the concept of AR to enhance the surgeon's perception of reality during a procedure, and to enrich the learning process during a surgical procedure.
Here, we successfully demonstrate the first integration of Google Glass in the surgical training of urology residents and faculty along with the implementation of ARAS in feature detection during penile prosthesis surgery. The overwhelming response from faculty is to have a proctoring platform, while trainees slightly favored the implementation of ARAS over their training counterparts. As expected, we noticed that trainees, who are younger, rated the application more educationally useful and less distracting in the operating room than faculty, but were surprised to find faculty rating ease of navigation and likelihood to use in the future higher than trainees. This may suggest that though there is more acceptance of newer technology from trainees, there is also a higher expectation of available features and intuitive design than faculty. Suggested future capabilities include remote proctoring and a less cumbersome design. As ARAS evolves, it will continue to innovate surgical procedures by adding graphics, sounds, and haptic feedback. Advantages of ARAS include real-time feedback of residents during surgery, superior visibility, and interaction between faculty and residents. ARAS is part of the confluence of change that will be vital in the future armamentarium of surgical education.
| Conclusion | | |
ARAS is a novel and effective urologic surgical training tool in this pilot study. Both faculty and trainees view this new technology in a positive light. Augmented reality in surgery represents a paradigm shift within surgery, and urologic training may benefit from further exploration of this technology within residency programs and practices alike. As future platforms of OHMDs become available, this technology will continue to change how medicine is both practiced and taught.
| Author Contributions | | |
RMD wrote the manuscript, TSH developed the application and edited the manuscript, NS administered ARAS survey and collected data, LIL provided consultation, PES assisted with writing of the manuscript, REC developed the concept. All authors read and approved the final manuscript.
| Competing Interest | | |
All authors declared no competing financial interests.
| Acknowledgments | | |
The authors would like to acknowledge American Medical Systems for providing the funding for the development of the software application, and Guil Zriel for creating the Google Glass application.
| References | | |
| 1. | Bishop WJ. The Early History of Surgery. London: R. Hale; 1960. p. 192 p.  |
| 2. | Gawande A. Two hundred years of surgery. N Engl J Med 2012; 366: 1716-23. |
| 3. | Rontgen WC. On a new kind of rays. Science 1896; 3: 227-31. |
| 4. | Spaner SJ, Warnock GL. A brief history of endoscopy, laparoscopy, and laparoscopic surgery. J Laparoendosc Adv Surg Tech A 1997; 7: 369-73. |
| 5. | Pugin F, Bucher P, Morel P. History of robotic surgery: from AESOP(R) and ZEUS(R) to da Vinci(R). J Visc Surg 2011; 148: e3-8. |
| 6. | Mobley DF. Early history of inflatable penile prosthesis surgery: a view from someone who was there. Asian J Androl 2015; 17: 225-9. |
| 7. | Macherey O, Carlyon RP. Cochlear implants. Curr Biol 2014; 24: R878-84. |
| 8. | Cooley DA, Liotta D, Hallman GL, Bloodwell RD, Leachman RD, et al. Orthotopic cardiac prosthesis for two-staged cardiac replacement. Am J Cardiol 1969; 24: 723-30. [ PUBMED] |
| 9. | Stevenson A. Oxford Dictionary of English. New York: Oxford University Press; 2010. pxxii. 2069 p. |
| 10. | Marescaux J, Diana M. Next step in minimally invasive surgery: hybrid image-guided surgery. J Pediatr Surg 2015; 50: 30-6. |
| 11. | Muensterer OJ, Lacher M, Zoeller C, Bronstein M, Kubler J. Google Glass in pediatric surgery: an exploratory study. Int J Surg 2014; 12: 281-9. |
| 12. | Sanderson PM, Watson MO, Russell WJ, Jenkins S, Liu D, et al. Advanced auditory displays and head-mounted displays: advantages and disadvantages for monitoring by the distracted anesthesiologist. Anesth Analg 2008; 106: 1787-97. |
| 13. | Liu D, Jenkins SA, Sanderson PM, Watson MO, Leane T, et al. Monitoring with head-mounted displays: performance and safety in a full-scale simulator and part-task trainer. Anesth Analg 2009; 109: 1135-46. |
| 14. | Lindeque BG, Ponce BA, Menendez ME, Oladeji LO, Fryberger CT, et al. Emerging technology in surgical education: combining real-time augmented reality and wearable computing devices. Orthopedics 2014; 37: 751-7. |
| 15. | Davis CR, Rosenfield LK. Looking at plastic surgery through google glass: part 1. Systematic review of google glass evidence and the first plastic surgical procedures. Plast Reconst Surg 2015; 135: 918-28. |
| 16. | Kantor J. Application of google glass to mohs micrographic surgery: a pilot study in 120 patients. Dermatol Surg 2015; 41: 288-9. |
| 17. | Hughes-Hallett A, Mayer EK, Marcus HJ, Cundy TP, Pratt PJ, et al. Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology 2014; 83: 266-73. |
| 18. | Hung AJ, Shah SH, Dalag L, Shin D, Gill IS. Development and validation of a novel robotic procedure specific simulation platform: partial nephrectomy. J Urol 2015; 194: 520-6. |
| 19. | Simon R, Hakky TS, Henry G, Perito P, Martinez D, et al. Tips and tricks of inflatable penile prosthesis reservoir placement: a case presentation and discussion. J Sex Med 2014; 11: 1325-33. |
[Figure 1], [Figure 2], [Figure 3]
[Table 1], [Table 2]
| This article has been cited by | | 1 |
Validity and effectiveness of augmented reality in surgical education: A systematic review |
|
| Joshua G. Kovoor,Aashray K. Gupta,Marc A. Gladman | | Surgery. 2021; | | [Pubmed] | [DOI] | | | 2 |
Lessons Learned: Medical Library Pilot Testing of a Virtual Reality Anatomy Lab |
|
| Emily C. Weyant,Matthew Carroll,Rachel R. Walden | | Journal of Electronic Resources in Medical Libraries. 2021; : 1 | | [Pubmed] | [DOI] | | | 3 |
Augmenting prehospital care |
|
| Yuval Glick,B Avital,J Oppenheimer,D Nahman,L Wagnert-Avraham,A Eisenkraft,L Dym,D Levi,A Agur,B Gustus,A Furer | | BMJ Military Health. 2021; 167(3): 158 | | [Pubmed] | [DOI] | | | 4 |
A workflow to generate patient-specific three-dimensional augmented reality models from medical imaging data and example applications in urologic oncology |
|
| Nicole Wake, Andrew B. Rosenkrantz, William C. Huang, James S. Wysock, Samir S. Taneja, Daniel K. Sodickson, Hersh Chandarana | | 3D Printing in Medicine. 2021; 7(1) | | [Pubmed] | [DOI] | | | 5 |
Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review |
|
| Zahraa Bassyouni,Imad H. Elhajj | | Frontiers in Robotics and AI. 2021; 8 | | [Pubmed] | [DOI] | | | 6 |
Augmented Reality for Robotics: A Review |
|
| Zhanat Makhataeva,Huseyin Varol | | Robotics. 2020; 9(2): 21 | | [Pubmed] | [DOI] | | | 7 |
Head-Mounted Display Use in Surgery: A Systematic Review |
|
| Rafa Rahman,Matthew E. Wood,Long Qian,Carrie L. Price,Alex A. Johnson,Greg M. Osgood | | Surgical Innovation. 2020; 27(1): 88 | | [Pubmed] | [DOI] | | | 8 |
Augmented reality in surgical training: a systematic review |
|
| Matthew Adam Williams,James McVeigh,Ashok Inderraj Handa,Regent Lee | | Postgraduate Medical Journal. 2020; 96(1139): 537 | | [Pubmed] | [DOI] | | | 9 |
Google glass adoption in the educational environment: A case study in the Gulf area |
|
| Rana Saeed Al-Maroof,Aseel M. Alfaisal,Said A. Salloum | | Education and Information Technologies. 2020; | | [Pubmed] | [DOI] | | | 10 |
Integrating Eye-Tracking to Augmented Reality System for Surgical Training |
|
| Shang Lu,Yerly Paola Sanchez Perdomo,Xianta Jiang,Bin Zheng | | Journal of Medical Systems. 2020; 44(11) | | [Pubmed] | [DOI] | | | 11 |
Wave of Wearables |
|
| Jeffrey Tully,Christian Dameff,Christopher A. Longhurst | | Clinics in Laboratory Medicine. 2020; | | [Pubmed] | [DOI] | | | 12 |
Interface of mixed reality: from the past to the future |
|
| Steven Szu-Chi Chen,Henry Duh | | CCF Transactions on Pervasive Computing and Interaction. 2019; | | [Pubmed] | [DOI] | | | 13 |
Video Technologies for Recording Open Surgery: A Systematic Review |
|
| Tomas J. Saun,Kevin J. Zuo,Teodor P. Grantcharov | | Surgical Innovation. 2019; 26(5): 599 | | [Pubmed] | [DOI] | | | 14 |
Surgical Telementoring Without Encumbrance |
|
| Edgar Rojas-Muñoz,Maria Eugenia Cabrera,Daniel Andersen,Voicu Popescu,Sherri Marley,Brian Mullis,Ben Zarzaur,Juan Wachs | | Annals of Surgery. 2019; 270(2): 384 | | [Pubmed] | [DOI] | | | 15 |
A Systematic Review of the Use of Google Glass in Graduate Medical Education |
|
| Joseph F. Carrera,Connor C. Wang,William Clark,Andrew M. Southerland | | Journal of Graduate Medical Education. 2019; 11(6): 637 | | [Pubmed] | [DOI] | | | 16 |
Augmented Reality in Emergency Medicine: A Review (Preprint) |
|
| Brendan William Munzer,Mairaj Khan,Prashant Mahajan | | Journal of Medical Internet Research. 2018; | | [Pubmed] | [DOI] | | | 17 |
Using Google Glass in Surgical Settings: Systematic Review |
|
| Nancy J Wei,Bryn Dougherty,Aundria Myers,Sherif M Badawy | | JMIR mHealth and uHealth. 2018; 6(3): e54 | | [Pubmed] | [DOI] | | | 18 |
Touch interaction with Google Glass – Is it suitable for older adults? |
|
| Marten Haesner,Sebastian Wolf,Anika Steinert,Elisabeth Steinhagen-Thiessen | | International Journal of Human-Computer Studies. 2018; 110: 12 | | [Pubmed] | [DOI] | | | 19 |
Testing Smart Glasses in urology: Clinical and surgical potential applications |
|
| E. García-Cruz,A. Bretonnet,A. Alcaraz | | Actas Urológicas Españolas (English Edition). 2018; | | [Pubmed] | [DOI] | | | 20 |
Use of Augmented Reality and Virtual Reality Technologies in Endoscopic Training |
|
| Sheena Bhushan,Sharmila Anandasabapathy,Richa Shukla | | Clinical Gastroenterology and Hepatology. 2018; | | [Pubmed] | [DOI] | | | 21 |
Evaluación del uso de smart glasses en urología: potenciales aplicaciones en los ámbitos hospitalario y quirúrgico |
|
| E. García-Cruz,A. Bretonnet,A. Alcaraz | | Actas Urológicas Españolas. 2017; | | [Pubmed] | [DOI] | | | 22 |
Augmented reality: The use of the PicoLinker smart glasses improves wire insertion under fluoroscopy |
|
| Takafumi Hiranaka,Takaaki Fujishiro,Yuichi Hida,Yosaku Shibata,Masanori Tsubosaka,Yuta Nakanishi,Kenjiro Okimura,Harunobu Uemoto | | World Journal of Orthopedics. 2017; 8(12): 891 | | [Pubmed] | [DOI] | | | 23 |
The use of head-mounted display eyeglasses for teaching surgical skills: A prospective randomised study |
|
| Robert G. Peden,Rachel Mercer,Andrew J. Tatham | | International Journal of Surgery. 2016; 34: 169 | | [Pubmed] | [DOI] | | | 24 |
Feasibility and safety of augmented reality-assisted urological surgery using smartglass |
|
| H. Borgmann,M. Rodríguez Socarrás,J. Salem,I. Tsaur,J. Gomez Rivas,E. Barret,L. Tortolero | | World Journal of Urology. 2016; | | [Pubmed] | [DOI] | |
|
 |
|
|
|
Search Pubmed for