Malone researchers Swaroop Vedula, Vishal Patel, Shameema Sikder, and Masaru Ishii have received a four-year, nearly $1.2 million award from the National Science Foundation‘s highly competitive Smart Health and Biomedical Research in the Era of Artificial Intelligence and Advanced Data Science Program to develop artificial intelligence that is capable of giving surgeons expert feedback based on videos of their performance.
Surgical residents use feedback from their mentors to improve their skills and patient outcomes—but this constructive criticism isn’t so readily available after they complete their training and begin operating on their own.
“There are many incredible teaching strategies, as well as an equally diverse set of learning styles—but unfortunately, most training institutions lack access to the resources necessary to implement all teaching strategies and address all learning needs,” says Ishii, an associate professor in the Departments of Otolaryngology–Head and Neck Surgery and Neurology and Neurological Surgery and a co-investigator on the project.
“This is where AI comes in,” explains Vedula, an associate research professor at the Malone Center and the project’s principal investigator. “Recent advances have created a tremendous potential for technology that can provide surgeons with expert feedback on any surgery they perform. In this project, we plan to leverage current AI models and develop new ones that analyze videos of surgical procedures and generate tailored, natural-language feedback.”
Inspired by the Language of Surgery project pioneered by Hopkins researchers Gregory D. Hager, Sanjeev Khudanpur, and Rene Vidal, the research team will use its combined expertise in computer science, electrical and computer engineering, language and speech processing, and engineering for health care to solve a problem spanning multiple surgical disciplines: enabling surgeon-machine teaming at the speed of thought.
Focusing on cataract surgery as a prototype procedure, the team will first develop a unified framework of vision-language and large language models to generate expert analyses of surgical videos—including describing, interpreting, and reasoning about what the models observe in the videos and predicting how the procedures evolve over time. Based on this framework, the researchers will explore methods for continuous model improvement based on advanced AI training techniques.
Ultimately, the researchers want to train their AI without placing the burden on surgeons to give additional feedback and record new surgical videos. To this end, the team plans to use additional AI methods to synthesize new videos from preexisting analyses so that they can create datasets large enough for training.
The project’s end goals include shortening surgeons’ learning curves and reducing the variation in patient outcomes that results from different skill levels among surgeons, thus elevating surgical care for all patients.
“Our AI models will allow every surgeon to take an expert colleague into the operating room for every cataract surgery they perform,” says Vedula. “This will help them perform at their best—so every patient has the best possible outcome.”
“I’m so excited to have the prestigious support of the NSF for this paradigm-shifting project,” says Sikder, the L. Douglass Lee and Barbara Levinson-Lee Associate Professor of Ophthalmology, the director of the Center of Excellence for Ophthalmic Surgical Education and Training, and a co-investigator on the project. “Much like we have a hard time imagining driving without GPS now, one day surgeons will wonder how surgery could have taken place without the personalized technology that our team at Hopkins is creating.”
Other co-investigators on this project include Daniel Khashabi, an assistant professor of computer science, and Hanghang Wang, an assistant professor of cardiac surgery.