Might this be a sign that AI platforms like Watson still cannot diagnose the wide range of patients’ conditions as accurately as a board-certified clinical pathologist?
Computer technology evolves so quickly, products often become obsolete before fulfilling their expected potential. Such, apparently, is the case with Watson, the genius artificial intelligence (AI) brainchild of International Business Machines Corp. (IBM) which was going to revolutionize how healthcare providers diagnose disease. In some areas of healthcare, such as analyzing MRIs and X-rays, AI has been a boon. But from a business perspective, Watson has failed to turn a profit for IBM, so it has to go.
In February, The Wall Street Journal (WSJ) reported that IBM is looking to sell its Watson Health unit because it is not profitable, despite bringing in $1 billion annually in revenue. The sale of Watson Health, the article states, would be aligned with IBM’s goal of streamlining the company and focusing its energies on cloud computing and other AI functions. Because one goal of the Watson project was to give physicians a tool to help them diagnose patients more accurately and faster, the problems that prevented Watson from achieving that goal should be of interest to pathologists and clinical laboratory managers, who daily are on the front lines of helping doctors diagnose the most challenging cases.
In a follow-up article, titled, “Potential IBM Watson Health Sale Puts Focus on Data Challenges,” the WSJ wrote, “… some experts found that it can be difficult to apply AI to treating complex medical conditions. Having access to data that represents patient populations broadly has been a challenge, experts told the Journal, and gaps in knowledge about complex diseases may not be fully captured in clinical databases.”
“I believe that we’re many years away from AI products that really positively impact clinical care for many patients,” Bob Kocher, Partner at Venrock, a venture-capital firm that invests in healthcare IT and related services, told the WSJ.
IBM Watson was promoted as a major resource to help improve medical care and support doctors in making more accurate diagnoses. However, in “IBM’s Retreat from Watson Highlights Broader AI Struggles in Health,” the WSJ reported that “IBM spent several billion dollars on acquisitions to build up Watson [Health] … a unit whose marquee product was supposed to help doctors diagnose and cure cancer … A decade later, reality has fallen short of that promise.”
During the years following Watson’s Jeopardy win, Watson Health made some positive advances in the fields of healthcare data analytics, performance measurements, clinical trial recruitment, and healthcare information technology (HIT).
However, Watson Health also experienced some high-profile failures as well. One such failure involved a collaboration with MD Anderson Cancer Center, established in 2013, to help the health systems’ oncologists develop new tools to benefit cancer patients. MD Anderson ended the relationship in 2018 after spending more than $60 million on the project, citing “multiple examples of unsafe and incorrect treatment recommendations,” made by the Watson supercomputer, Healthcare IT News reported.
Watson Health later readjusted the development and sales of its AI drug discovery tools and altered its marketing strategy amid reports of disappointing sales and skepticism surrounding machine learning for medical applications.
Underestimating the Challenge of AI in Healthcare
Since its inception, Watson Health has achieved substantial growth, mainly through a series of acquisitions. Those targeted acquisitions include:
Merge Healthcare, a healthcare imaging software company that was purchased for $1 billion in 2015,
Phytel, a health management software company that was purchased for an undisclosed amount in 2015,
Explorys, a healthcare analytics company that was purchased for an undisclosed amount in 2015, and
Truven Health Analytics, a provider of cloud-based healthcare data, analytics, and insights that was purchased for $2.6 billion in 2016.
“IBM’s Watson Health business came together as a result of several acquisitions,” said Paddy Padmanabhan, founder and CEO of Damo Consulting, a firm that provides digital transformation strategy and advisory services for healthcare organizations. “The decision to sell the business may also have to do with the performance of those units on top of the core Watson platform’s struggles,” he told Healthcare IT News.
It should be noted that these acquisitions involved companies that already had a product in the market which was generating revenue. So, the proposed sale of Watson Health includes not just the original Watson AI product, but the other businesses that IBM put into its Watson Health business division.
Padmanabhan noted that there are many challenges for AI in healthcare and that “historical data is at best a limited guide to the future when diagnosing and treating complex conditions.” He pointed to the failure with MD Anderson (in the use of Watson Health as a resource or tool for diagnosing cancer) was a setback for both IBM and the use of AI in healthcare. However, Padmanabhan is optimistic regarding the future use of AI in healthcare.
“To use an oft-quoted analogy, AI’s performance in healthcare right now is more akin to that of the hedgehog than the fox. The hedgehog can solve for one problem at a time, especially when the problem follows familiar patterns discerned in narrow datasets,” he told Healthcare IT News. “The success stories in healthcare have been in specific areas such as sepsis and readmissions. Watson’s efforts to apply AI in areas such as cancer care may have underestimated the nuances of the challenge.”
Other experts agree that IBM was overly ambitious and overreached with Watson Health and ended up over-promising and under-delivering.
“IBM’s initial approach misfired due to how the solution AI was trained and developed,” Dan Olds, Principal Analyst with Gabriel Consulting Group, told EnterpriseAI. “It didn’t conform well to how doctors work in the real world and didn’t learn from its experiences with real doctors. It was primarily learning from synthetic cases, not real-life cases.”
Was Watson Already Obsolete?
Another issue with Watson was that IBM’s marketing campaign may have exceeded the product’s design capabilities. When Watson was developed, it was built with AI and information technologies (IT) that were already outdated and behind the newest generation of those technologies, noted Tech Republic.
“There were genuine AI innovation triggers at Watson Health in natural language processing and generation, knowledge extraction and management, and similarity analytics,” Jeff Cribbs, Research Vice President at Gartner Research, told Tech Republic. “The hype got ahead of the engineering, as the hype cycle says it almost always will, and some of those struggles became apparent.”
Can Artificial Intelligence Fulfill its Potential in Healthcare?
The fact that IBM is contemplating the sale of Watson Health is another illustration of how difficult it can be to navigate the healthcare industry in the US. It is probable that someday AI could make healthcare diagnostics more accurate and reduce overall costs, however, data challenges still exist and more research and exploration will be needed for AI to fulfill its potential.
“Today’s AI systems are great in beating you at chess or Jeopardy,” Kumar Srinivas, Chief Technology Officer, Health Plans, at NTT DATA Services told Forbes. “But there are major challenges when addressing practical clinical issues that need a bit of explanation as to ‘why.’ Doctors aren’t going to defer to AI-decisions or respond clinically to a list of potential cancer cases if it’s generated from a black box.”
And perhaps that is the biggest challenge of all. For doctors to entrust their patients’ lives to a supercomputer, it better be as good as the hype. But can AI in healthcare ever accomplish that feat?
“AI can work incredibly well when it’s applied to specific use cases,” gastroenterologist Nirav R. Shah, MD, Chief Medical Officer at Sharecare, told Forbes. “With regards to cancer, we’re talking about a constellation of thousands of diseases, even if the focus is on one type of cancer. What we call ‘breast cancer,’ for example, can be caused by many different underlying genetic mutations and shouldn’t really be lumped together under one heading. AI can work well when there is uniformity and large data sets around a simple correlation or association. By having many data points around a single question, neural networks can ‘learn.’ With cancer, we’re breaking several of these principles.”
So, in deciding to divest itself of Watson Health, IBM may simply be as prescient now as it was when it first embraced the concept of AI in healthcare. The tech giant may foresee that AI will likely never replace the human mind of a trained healthcare diagnostician.
If this proves true—at least for several more years—then board-certified clinical pathologists can continue to justifiably refer to themselves as “the doctor’s doctor” because of their skills in diagnosing difficult-to-diagnose patients, and because of their knowledge of which clinical laboratory tests to order and how to interpret those test results.
Since the early 1980s, UBC’s CMPT program, led by medical microbiologist Michael Noble, MD, has provided external quality assessment (EQA) for clinical microbiology and water testing laboratories. This includes providing biological samples related to:
“Typical of every jurisdiction in North America and probably around the world, BCCDC got swamped beyond swamped,” said Noble, the Clinical Microbiology Proficiency Testing (CMPT) program’s first and current Chair, in an exclusive interview with Dark Daily. “The increase was 10-fold, and they were unable to provide all the services they wanted to do. And since I was already running a proficiency testing program across the province, they asked if I would provide that service for COVID-19 for laboratories that were doing the testing.”
CMPT’s Proficiency Testing Serves Labs Worldwide
UBC’s CMPT external quality assessment (EQA) program serves all medical laboratories in British Columbia, as well as other labs in Canada, Europe, South America, and the Caribbean. Just over 200 laboratories currently participate in the program. More labs participated in past years, before lab consolidation affected CMPT and other programs as well, Noble said.
CMPT’s proficiency testing ensures that participant laboratories that have been provided with simulated samples can perform tests at the “level of quality and competence required,” notes UBC’s CMPT website.
“Samples are complex, highly realistic, and clinically relevant. CMPT samples contain host elements as well as targeted pathogens,” Noble explained on his blog, “Making Medical Laboratory Quality Relevant.”
COVID-19 Brings Non-Traditional ‘Laboratories’ to CMPT’s Proficiency Testing Program
UBC’s proficiency testing for SARS-CoV-2, the coronavirus that causes the COVID-19 infection, differs from other CMPT programs. That’s due to new participants that entered the laboratory testing program during the COVID-19 pandemic that are performing COVID-19 testing in non-traditional locations, Noble stated.
“In our proficiency programs, we had mainly been dealing with traditional clinical laboratories,” Noble explained. “But now, we find people doing COVID-19 testing—even though defined as medical laboratories—who are working in airports, or in tourism, or the movie industry, or forestry. They may never have worked in an actual clinical laboratory. So, it’s a very different style of proficiency testing. There has been a lot of handholding, teleconferences, discussions, and one-on-ones with that group,” Noble said.
Participant laboratories receive viral material that “simulates typical samples.” They need to demonstrate proficiency by performing the test and reporting it as positive, negative, or inconclusive.
“Our product is derived from a pure culture of a single strain of SARS-CoV-2, and it appears to be effective for all targets,” Noble stated.
Detecting COVID-19 by Gargling and Rinsing
UBC’s program typically offers simulated sampling for detection of SARS-CoV-2 in nasopharyngeal swabs. However, the BC Center for Disease Control’s (BCCDC) mouth rinse and gargle sample collection for diagnosis of COVID-19 also is available and widely used in Canada, Noble said.
In his career, Noble transitioned from medical microbiology to qualitology, which he describes as “the study of quality in the medical laboratory.”
In stressing the importance of laboratory quality testing, Noble describes the possibility of laboratory testing going awry and leading to a microbiological public health emergency.
“What happens if there’s a stool sample, and someone misses the presence of Campylobacteriosis in the stool? What happens if that’s part of a foodborne disease and there’s an outbreak in the city and samples are being missed? How many people will be impacted as a result of that error?” he asked.
University of British Columbia Endows a Chair for Laboratory Quality Management
Noble says UBC’s Program Office for Laboratory Quality Management (POLQM) has involved organizations worldwide and certified more than 500 people.
“The impact they have over their laboratories has been huge. Maybe that would have happened without us. But we were a part of that. And our impact is not one laboratory or one city or one province but widespread, and that’s a real and enriching experience to have,” he said.
But now it is time for him to move on. Noble secured (through UBC), a benefactor to establish the endowed Chair for Laboratory Quality Management. The family of the late Donald B. Rix, MD, a Canadian pathologist and philanthropist, gave $1.5 million (matched by the university) to create the Associate Professor (Grant Tenure) Donald B. Rix Professorship in Laboratory Quality at UBC, Department of Pathology and Laboratory Medicine.
Long-serving pathologists and medical laboratory professionals may remember that Rix was the founder and chair of MDS Metro Laboratory Services (now known as LifeLabs Medical Laboratory Services). It grew into the largest private medical laboratory in Western Canada.
Referring to this endowed new Chair for Laboratory Quality Management, Noble said, “I think this is the first named position of laboratory quality in North America.” UBC has commenced reviewing applications for the position, which is expected to be effective in January 2022. Pathologists and clinical laboratory scientists with appropriate qualifications and interest in this position should contact Dr. Noble’s office at the University of British Columbia Faculty of Medicine.
VCU scientists used the technique to measure mutations associated with acute myeloid leukemia, potentially offering an attractive alternative to DNA sequencing
More accurate but less-costly cancer diagnostics are the Holy Grail of cancer research. Now, research scientists at Virginia Commonwealth University (VCU) say they have developed a clinical laboratory diagnostic technique that could be far cheaper and more capable than standard DNA sequencing in diagnosing some diseases. Their method combines digital polymerase chain reaction (dPCR) technology with high-speed atomic force microscopy (HS-AFM) to generate nanoscale-resolution images of DNA.
The technique allows the researchers to measure polymorphisms—variations in gene lengths—that are associated with many cancers and neurological diseases. The VCU scientists say the new technique costs less than $1 to scan each dPCR reaction.
“We chose to focus on FLT3 mutations because they are difficult to [diagnose], and the standard assay is limited in capability,” said physicist Jason Reed, PhD, Assistant Professor in the Virginia Commonwealth University Department of Physics, in a VCU press release.
Reed is an expert in nanotechnology as it relates to biology and medicine. He led a team that included other researchers in VCU’s physics department as well as physicians from VCU Massey Cancer Center and the Department of Internal Medicine at VCU School of Medicine.
Validating the Clinical Laboratory Test
The physicists worked with two VCU physicians—hematologist/oncologist Amir Toor, MD, and hematopathologist Alden Chesney, MD—to compare the imaging technique to the LeukoStrat CDx FLT3 Mutation Assay, which they described as the “current gold standard test” for diagnosing FLT3 gene mutations.
The researchers said their technique matched the results of the LeukoStrat test in diagnosing the mutations. But unlike that test, the new technique also can measure variant allele frequency (VAL). This “can show whether the mutation is inherited and allows the detection of mutations that could potentially be missed by the current test,” states the VCU press release.
“We plan to continue developing and testing this technology in other diseases involving DNA structural mutations,” Reed said. “We hope it can be a powerful and cost-effective tool for doctors around the world treating cancer and other devastating diseases driven by DNA mutations.”
“In our approach we first used digital PCR, in which a mixed sample is diluted to less than one target molecule per aliquot and the aliquots are amplified to yield homogeneous populations of amplicons,” he said. “Then, we deposited each population onto an atomically-flat partitioned surface.”
The VCU researchers “scanned each partition with high-speed atomic force microscopy, in which an extremely sharp tip is rastered across the surface, returning a 3D map of the surface with nanoscale resolution,” he said. “We wrote code that traced the length of each imaged DNA molecule, and the distribution of lengths was used to determine whether the aliquot was a wild type [unmutated] or variant.”
In Diagnostics World, Reed said the method “doesn’t really have any more complexity than a PCR assay itself. It can easily be done by most lab technicians.”
Earlier Research
A VCU press release from 2017 noted that Reed’s research team had developed technology that uses optical lasers (similar to those in a DVD player) to accelerate the scanning. The researchers previously published a study about the technique in Nature Communications, and a patent is currently pending.
“DNA sequencing is a powerful tool, but it is still quite expensive and has several technological and functional limitations that make it difficult to map large areas of the genome efficiently and accurately,” Reed said in the 2017 VCU press release. “Our approach bridges the gap between DNA sequencing and other physical mapping techniques that lack resolution. It can be used as a stand-alone method or it can complement DNA sequencing by reducing complexity and error when piecing together the small bits of genome analyzed during the sequencing process.”
Using CRISPR technology, the team also developed what they described as a “chemical barcoding solution,” placing markers on DNA molecules to identify genetic mutations.
New DNA Clinical Laboratory Testing?
Cancer diagnostics are constantly evolving and improving. It is not clear how long it will be before VCU’s new technique will reach clinical laboratories that perform DNA testing, if at all. But VCU’s new technique is intriguing, and should it prove viable for clinical diagnostic use it could revolutionize cancer diagnosis. It is a development worth watching.
Physician use of genetic tests continues to grow at robust rates, even during the pandemic, but uncertainty about managed care reimbursement hangs over the market
It may surprise many pathologists and clinical laboratory managers to learn that the market for genetic testing is robust and growing swiftly, even in the midst of the COVID-19 pandemic. At the same time, the explosion in both the number of unique genetic tests available to physicians, and the willingness of doctors to order genetic tests for their patients, are creating major challenges for both government and private payers.
Moreover, how payers are attempting to gain control over this boom in genetic testing is creating serious problems for genetic testing companies seeking reimbursement for their test claims. This is because health insurers are taking aggressive steps to control their spending on genetic tests. Some of those steps include:
Prior-authorization requirements for an ever-larger number of genetic tests.
Reducing the prices paid for high-cost genetic tests.
Tough audits that use sampling and extrapolation and produce sizeable recoupment demands.
Unexpected Developments in Genetic Test Marketplace
These are reasons why clinical laboratories need to fully understand the state of the genetic testing market. Physicians are receptive to ordering genetic tests that will improve the care they provide their patients. But health insurers want better control over the unplanned and substantial increases in the total amount of money they pay out for the surging number of genetic test claims.
Collectively, these developments confront genetic testing companies with a mix of good news and bad news. The good news is that more physicians are using genetic tests in their daily medical practice. The bad news is that many payers are erecting ever-more restrictive hurdles that labs must overcome when submitting genetic test claims and seeking adequate payment.
Strategic Insights into What’s Changing with Genetic Testing
This webinar will be one of the most important strategic assessments of genetic testing presented to the clinical laboratory and diagnostics industries since the COVID-19 pandemic began last March. Your presenters are recognized thought-leaders in the genetic testing and laboratory medicine industries. Speaking in order are:
Bruce Quinn, MD, PhD, Principal, Bruce Quinn Associates LLC, Los Angeles: An expert in how Medicare and private payers establish coverage guidelines and prices for new genetic tests, Dr. Quinn will explain the key differences in how private payers are managing genetic test utilization and payment, compare to the federal Medicare program.
Heather Agostinelli, Asst. Vice President, Strategic Revenue Operations, XIFIN Inc., San Diego: Heather will provide a detailed perspective on the daily actions by payers as they process claims and issue payment for genetic tests. She will also present recommendations for how labs can optimize the number of clean genetic test claims, thus helping shorten payment times in ways that improve cash flow.
Rob Metcalf, CEO, Concert Genetics, Nashville, Tenn.: He will discuss the scope and scale of the explosion in the number of genetic test claims by sharing data, charts, and analyses usually only available to clients.
Your Chair and Moderator will be Robert L. Michel, Editor-in-Chief of The Dark Report.
The purpose of the upcoming webinar includes helping attendees with the following and more:
Learn why payers must now deal with more than 1,000 new genetic testing products launching every month and how that complicates claims processing.
Understand how the variation in CPT coding by different genetic testing labs complicates claims processing by payers.
Learn why “benefit investigation” is already a huge factor as consumers seek the lab with the cheapest genetic test price before they agree to be tested.
Master the art of working with prior authorization programs and know why having documents prior to authorization still does not necessarily mean the payer will reimburse for a genetic test claim.
Understand Medicare’s policy changes at the national level for genetic tests.
Know the core elements of the Medicare MolDx program that gov-erns genetic test claims across 28 states.
Valuable Information for Financial Analysis, Managed Care Executives
In addition to bringing clinical pathologists and directors/managers of clinical laboratories up to date on the genetic testing marketplace, this webinar will provide valuable insights into financial analysts’ tracking of genetic testing companies, managed care executives’ handling of genetic testing claims, genetic counselors, and others involved in managing clinical service lines that utilize genetic tests in patient care.
Medical technologists and clinical laboratory professionals are the unsung heroes of the COVID-19 pandemic and the public is beginning to notice
Medical technologists (MTs) and clinical laboratory scientists (CLSs) are the foundation of every successful clinical laboratory. But they seldom make the news. Therefore, it is worth noting, during this COVID-19 pandemic, when clinical laboratory professionals receive public recognition for the important role they play in fighting the disease.
A news story published by the Canadian Broadcasting Corporation (CBC), titled, “Lab Tech Who Found B.C.’s 1st Case of COVID-19 Recalls ‘Sheer Terror’ of Discovery,” describes a laboratory technologist’s experience in British Columbia when she discovered the Canadian province’s first positive case of COVID-19 in January of 2020.
Finding COVID-19 for the First Time
On January 27, 2020, Rebecca Hickman, Public Health Laboratory Technologist, Molecular Biology and Genomics at BC Centre for Disease Control (BCCDC) was carefully monitoring samples for COVID-19 and fearing a positive result for the SARS-CoV-2 coronavirus when her worst fear appeared before her eyes.
“I actually started to see it get positive within a few seconds,” Hickman recalled. “My first feeling was sheer terror, from a personal point of view.”
When Hickman realized a sample was going to test positive, she called Tracy Lee, Technical Coordinator at BC Centre for Disease Control and co-designer of the BCCDC’s COVID-19 test. Hickman had to interrupt Lee in a meeting, who then hurried to the lab to watch the test complete. It was a definite positive, the first confirmed case in British Columbia.
“To design, validate, and implement a molecular laboratory test usually takes months if not years, and so to do that in the span of days is a huge achievement,” Hickman told the CBC.
The following day, it was announced to the residents of BC that the COVID-19 coronavirus was in their province and that they needed to start taking necessary precautions. “This is the first time in my life I’ve ever found things out before I read it in the news,” Hickman said.
Supply Shortages Challenge British Columbia Clinical Laboratories
Hickman noted there have been several challenges in dealing with COVID-19 over the past year. “The instability and craziness of it all has been the hardest part,” she said. Last spring, the BC lab, like most labs, had to deal with a shortage of supplies and personal protective equipment.
According to BC Centre for Disease Control (BCCDC) data, as of March 2, 2021, there have been 81,367 confirmed cases of COVID-19 in the province of British Columbia. A total of 75,255 of those individuals have recovered from the coronavirus, more than 300 patients remain hospitalized, and 1,365 British Columbians have perished due to COVID-19. The population of the western Canadian province is approximately 5.1 million.
Today, Hickman, spends a majority of her time in the laboratory doing whole genome sequencing of confirmed COVID-19 cases. The data she collects is used for outbreak response and for tracking new variants of the SARS-CoV-2 coronavirus that are appearing in different parts of the world. “It has been easily the most difficult year of my life, but also the most fulfilling,” she told the CBC. “What we have achieved here over the last year is huge.”
Clinical Laboratories on the Front Lines
Last year, the American Society for Clinical Pathology (ASCP) produced a docuseries titled, “Laboratories on the Front Lines: Battling COVID-19” which highlighted the critical work clinical laboratories are doing to care for patients during the SARS-CoV-2 pandemic. The five-part series interviewed medical laboratory professionals across the US about their experiences during the pandemic.
In one episode, Stephanie Horiuchi, Clinical Microbiology Specialist at UCLA Health Systems, discussed how challenging and rewarding it has been working on the pandemic.
“Very long days. I’m not going to lie. Very, very long days, but it’s rewarding. I know the importance of what I am doing, and I know the importance of what needs to be done,” she said. “So, the time that I am here, it does go by very fast. You look up at the clock and you’re like oh, its 9pm. And then when I go home, it’s just eat and go to sleep and then rinse and repeat.
“I feel that this is a really important area of work that we all do as microbiologists,” Horiuchi continued. “And to just serve patients every day and to know that I am helping someone, it really warms my soul.”
In another episode of the docuseries, Professor of Pathology and Laboratory Medicine Alyssa Ziman, MD, Division Chief, Clinical Laboratory Medicine at UCLA Health, was interviewed regarding how they are coping with the increased demand for medical laboratory services.
“It’s been a really difficult and challenging time for our health system, for our laboratories, for our staff that are working through to provide the best possible patient care,” she said. Ziman is also Medical Director, Transfusion Medicine, at UCLA Health and Medical Director, Clinical Laboratories, at Ronald Reagan UCLA Medical Center. “Every day is a new challenge and a new way to adapt to changing rules from the CDC and from the LA County Public Health Department and to really evolve, so that we can continue to provide the testing that we have and continue to support our staff and our patients.”
Unsung Heroes of COVID-19
The COVID-19 pandemic has placed a strain on medical resources throughout the world. Clinical laboratory professionals are emerging as the unsung heroes of the crisis and the entire medical laboratory profession is receiving much deserved positive recognition for the crucial role laboratories are playing in fighting the pandemic.