Though the field of oncology has some AI-driven tools, overall, physicians report the reality isn’t living up to the hype
Artificial intelligence (AI) has been heavily touted as the next big thing in healthcare for nearly a decade. Much ink has been devoted to the belief that AI would revolutionize how doctors treat patients. That it would bring about a new age of point-of-care clinical decision support tools and clinical laboratory diagnostic tests. And it would enable remote telemedicine to render distance between provider and patient inconsequential.
But nearly 10 years after IBM’s Watson defeated two human contestants on the game show Jeopardy, some experts believe AI has under-delivered on the promise of a brave new world in medicine, noted IEEE Spectrum, a website and magazine dedicated to applied sciences and engineering.
In the years since Watson’s victory on Jeopardy, IBM (NYSE:IBM) has announced almost 50 partnerships, collaborations, and projects intended to develop AI-enabled tools for medical purposes. Most of these projects did not bear fruit.
However, IBM’s most publicized medical partnerships revolved around the field of oncology and the expectation that Watson could analyze data and patients’ records and help oncologists devise personalized and effective cancer treatment plans. Success in helping physicians more accurately diagnosis different types of cancer would require anatomic pathologists to understand this new role for Watson and how the pathology profession should respond to it, strategically and tactically.
But Watson and other AI systems often struggled to understand the finer points of medical text. “The information that physicians extract from an article, that they use to change their care, may not be the major point of the study,” Mark Kris, MD, Medical Oncologist at Memorial Sloan Kettering Cancer Center, told IEEE Spectrum. “Watson’s thinking is based on statistics, so all it can do is gather statistics about main outcomes. But doctors don’t work that way.”
Ultimately, IEEE Spectrum reported, “even today’s best AI struggles to make sense of complex medical information.”
“Reputationally, I think they’re in some trouble,” Robert Wachter, MD, Professor and Chair, Department of Medicine, University of California, San Francisco, told IEEE Spectrum. “They came in with marketing first, product second, and got everybody excited. Then the rubber hit the road. This is an incredibly hard set of problems, and IBM, by being first out, has demonstrated that for everyone else.”
Over Promises and Under Deliveries
In 2016, MD Anderson Cancer Center canceled a project with IBM Watson after spending $62 million on it, Becker’s Hospital Review reported. That project was supposed to use natural language processing (NLP) to develop personalized treatment plans for cancer patients by comparing databases of treatment options with patients’ electronic health records.
“We’re doing incredibly better with NLP than we were five years ago, yet we’re still incredibly worse than humans,” Yoshua Bengio, PhD, Professor of Computer Science at the University of Montreal, told IEEE Spectrum.
The researchers hoped that Watson would be able to examine variables in patient records and keep current on new information by scanning and interpreting articles about new discoveries and clinical trials. But Watson was unable to interpret the data as humans can.
IEEE Spectrum reported that “The realization that Watson couldn’t independently extract insights from breaking news in the medical literature was just the first strike. Researchers also found that it couldn’t mine information from patients’ electronic health records as they’d expected.”
Researchers Lack Confidence in Watson’s Results
In 2018, the team at MD Anderson published a paper in The Oncologist outlining their experiences with Watson and cancer care. They found that their Watson-powered tool, called Oncology Expert Advisor, had “variable success in extracting information from text documents in medical records. It had accuracy scores ranging from 90% to 96% when dealing with clear concepts like diagnosis, but scores of only 63% to 65% for time-dependent information like therapy timelines.”
A team of researchers at the University of Nebraska Medical Center (UNMC) have experimented with Watson for genomic analytics and breast cancer patients. After treating the patients, scientists identify mutations using their own tools, then enter that data into Watson, which can quickly pick out some of the mutations that have drug treatments available.
“But the unknown thing here is how good are the results,” Babu Guda, PhD, Professor and Chief Bioinformatics and Research Computing Officer at UNMC, told Gizmodo. “There is no way to validate what we’re getting from IBM is accurate unless we test the real patients in an experiment.”
Guda added that IBM needs to publish the results of studies and tests performed on thousands of patients if they want scientists to have confidence in Watson tools.
“Otherwise it’s very difficult for researchers,” he said. “Without publications, we can’t trust anything.”
Computer Technology Evolving Faster than AI Can Utilize It
The inability of Watson to produce results for medical uses may be exacerbated by the fact that the cognitive computing technologies that were cutting edge back in 2011 aren’t as advanced today.
IEEE Spectrum noted that professionals in both computer science and medicine believe that AI has massive potential for improving and enhancing the field of medicine. To date, however, most of AI’s successes have occurred in controlled experiments with only a few AI-based medical tools being approved by regulators. IBM’s Watson has only had a few successful ventures and more research and testing is needed for Watson to prove its value to medical professionals.
“As a tool, Watson has extraordinary potential,” Kris told IEEE Spectrum. “I do hope that the people who have the brainpower and computer power stick with it. It’s a long haul, but it’s worth it.”
Meanwhile, the team at IBM Watson Health continues to forge ahead. In February 2019, Healthcare IT News interviewed Kyu Rhee, MD, Vice President and Chief Health Officer at IBM Corp. and IBM Watson Health. He outlined the directions IBM Watson Health would emphasize at the upcoming annual meeting of the Healthcare Information and Management Systems Society (HIMSS).
IBM Watson Health is “using our presence at HIMSS19 this year to formally unveil the work we’ve been doing over the past year to integrate AI technology and smart, user-friendly analytics into the provider workflow, with a particular focus on real-world solutions for providers to start tackling these types of challenges head-on,” stated Rhee. “We will tackle these challenges by focusing our offerings in three core areas. First, is management decision support. These are the back-office capabilities that improve operational decisions.”
Clinical laboratory leaders and anatomic pathologists may or may not agree about how Watson is able to support clinical care initiatives. But it’s important to note that, though AI’s progress toward its predicted potential has been slow, it continues nonetheless and is worth watching.
—JP Schlingman
Related Information:
How IBM Watson Overpromised and Underdelivered on AI Health Care
Why Everyone is Hating on IBM Watson – Including the People Who Helped Make It
Memorial Sloan Kettering Trains IBM Watson to Help Doctors Make Better Cancer Treatment Choices
4 Reasons MD Anderson Put IBM Watson On Hold
IBM Watson Health’s Chief Health Officer Talks Healthcare Challenges and AI
Applying Artificial Intelligence to Address the Knowledge Gaps in Cancer Care
