Researchers intend their new AI image retrieval tool to help pathologists locate similar case images to reference for diagnostics, research, and education
Researchers at Stanford University turned to an unusual source—the X social media platform (formerly known as Twitter)—to train an artificial intelligence (AI) system that can look at clinical laboratory pathology images and then retrieve similar images from a database. This is an indication that pathologists are increasingly collecting and storing images of representative cases in their social media accounts. They then consult those libraries when working on new cases that have unusual or unfamiliar features.
The Stanford Medicine scientists trained their AI system—known as Pathology Language and Image Pretraining (PLIP)—on the OpenPath pathology dataset, which contains more than 200,000 images paired with natural language descriptions. The researchers collected most of the data by retrieving tweets in which pathologists posted images accompanied by comments.
“It might be surprising to some folks that there is actually a lot of high-quality medical knowledge that is shared on Twitter,” said researcher James Zou, PhD, Assistant Professor of Biomedical Data Science and senior author of the study, in a Stanford Medicine SCOPE blog post, which added that “the social media platform has become a popular forum for pathologists to share interesting images—so much so that the community has widely adopted a set of 32 hashtags to identify subspecialties.”
“It’s a very active community, which is why we were able to curate hundreds of thousands of these high-quality pathology discussions from Twitter,” Zou said.
“The main application is to help human pathologists look for similar cases to reference,” James Zou, PhD (above), Assistant Professor of Biomedical Data Science, senior author of the study, and his colleagues wrote in Nature Medicine. “Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to develop medical artificial intelligence for enhancing diagnosis, knowledge sharing, and education.” Leveraging pathologists’ use of social media to store case images for future reference has worked out well for the Stanford Medicine study. (Photo copyright: Stanford University.)
Retrieving Pathology Images from Tweets
“The lack of annotated publicly-available medical images is a major barrier for innovations,” the researchers wrote in Nature Medicine. “At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter.”
In this case, the goal “is to train a model that can understand both the visual image and the text description,” Zou said in the SCOPE blog post.
“Pathology is perhaps even more suited to Twitter than many other medical fields because for most pathologists, the bulk of our daily work revolves around the interpretation of images for the diagnosis of human disease,” wrote Jerad M. Gardner, MD, a dermatopathologist and section head of bone/soft tissue pathology at Geisinger Medical Center in Danville, Pa., in a blog post about the Pathology Hashtag Ontology project. “Twitter allows us to easily share images of amazing cases with one another, and we can also discuss new controversies, share links to the most cutting edge literature, and interact with and promote the cause of our pathology professional organizations.”
The researchers used the 32 subspecialty hashtags to retrieve English-language tweets posted from 2006 to 2022. Images in the tweets were “typically high-resolution views of cells or tissues stained with dye,” according to the SCOPE blog post.
The researchers collected a total of 232,067 tweets and 243,375 image-text pairs across the 32 subspecialties, they reported. They augmented this with 88,250 replies that received the highest number of likes and had at least one keyword from the ICD-11 codebook. The SCOPE blog post noted that the rankings by “likes” enabled the researchers to screen for high-quality replies.
They then refined the dataset by removing duplicates, retweets, non-pathology images, and tweets marked by Twitter as being “sensitive.” They also removed tweets containing question marks, as this was an indicator that the practitioner was asking a question about an image rather than providing a description, the researchers wrote in Nature Medicine.
They cleaned the text by removing hashtags, Twitter handles, HTML tags, emojis, and links to websites, the researchers noted.
The final OpenPath dataset included:
116,504 image-text pairs from Twitter posts,
59,869 from replies, and
32,041 image-text pairs scraped from the internet or obtained from the LAION dataset.
The latter is an open-source database from Germany that can be used to train text-to-image AI software such as Stable Diffusion.
Training the PLIP AI Platform
Once they had the dataset, the next step was to train the PLIP AI model. This required a technique known as contrastive learning, the researchers wrote, in which the AI learns to associate features from the images with portions of the text.
As explained in Baeldung, an online technology publication, contrastive learning is based on the idea that “it is easier for someone with no prior knowledge, like a kid, to learn new things by contrasting between similar and dissimilar things instead of learning to recognize them one by one.”
“The power of such a model is that we don’t tell it specifically what features to look for. It’s learning the relevant features by itself,” Zou said in the SCOPE blog post.
The resulting AI PLIP tool will enable “a clinician to input a new image or text description to search for similar annotated images in the database—a sort of Google Image search customized for pathologists,” SCOPE explained.
“Maybe a pathologist is looking at something that’s a bit unusual or ambiguous,” Zou told SCOPE. “They could use PLIP to retrieve similar images, then reference those cases to help them make their diagnoses.”
The Stanford University researchers continue to collect pathology images from X. “The more data you have, the more it will improve,” Zou said.
Pathologists will want to keep an eye on the Stanford Medicine research team’s progress. The PLIP AI tool may be a boon to diagnostics and improve patient outcomes and care.
Immunotherapy device could also enable clinical laboratories to receive in vivo biomarker data wirelessly
Researchers from Rice University in Houston and seven other states in the US are working on a new oncotherapy sense-and-respond implant that could dramatically improve cancer outcomes. Called Targeted Hybrid Oncotherapeutic Regulation (THOR), the technology is intended primarily for the delivery of therapeutic drugs by monitoring specific cancer biomarkers in vivo.
Through a $45 million federal grant from the Advanced Research Projects Agency for Health (ARPA-H), the researchers set out to develop an immunotherapy implantable device that monitors a patient’s cancer and adjusts antibody treatment dosages in real time in response to the biomarkers it measures.
It’s not a far stretch to envision future versions of the THOR platform also being used diagnostically to measure biomarker data and transmit it wirelessly to clinical laboratories and anatomic pathologists.
ARPH-A is a federal funding agency that was established in 2022 to support the development of high-impact research to drive biomedical and health breakthroughs. THOR is the second program to receive funding under its inaugural Open Broad Agency Announcement solicitation for research proposals.
“By integrating a self-regulated circuit, the THOR technology can adjust the dose of immunotherapy reagents based on a patient’s responses,” said Weiyi Peng, MD, PhD (above), Assistant Professor of Biology and Biochemistry at the University of Houston and co-principal investigator on the research, in a UH press release. “With this new feature, THOR is expected to achieve better efficacy and minimize immune-related toxicity. We hope this personalized immunotherapy will revolutionize treatments for patients with peritoneal cancers that affect the liver, lungs, and other organs.” If anatomic pathologists and clinical laboratories could receive biometric data from the THOR device, that would be a boon to cancer diagnostics. (Photo copyright: University of Houston.)
Antibody Therapy on Demand
Omid Veiseh, PhD, Associate Professor of Bioengineering at Rice University and principal investigator on the project, described the THOR device as a “living drug factory” inside the body. The device is a rod-like gadget that contains onboard electronics and a wireless rechargeable battery. It is three inches long and has a miniaturized bioreactor that contains human epithelial cells that have been engineered to produce immune modulating therapies.
“Instead of tethering patients to hospital beds, IV bags, and external monitors, we’ll use a minimally invasive procedure to implant a small device that continuously monitors their cancer and adjusts their immunotherapy dose in real time,” said Veiseh in a Rice University press release. “This kind of ‘closed-loop therapy’ has been used for managing diabetes, where you have a glucose monitor that continuously talks to an insulin pump.
But for cancer immunotherapy, it’s revolutionary.”
The team believes the THOR device will have the ability to monitor biomarkers and produce an antibody on demand that will trigger the immune system to fight cancer locally. They hope the sensor within THOR will be able to monitor biomarkers of toxicity for the purpose of fine-tuning therapies to a patient immediately in response to signals from a tumor.
“Today, cancer is treated a bit like a static disease, which it’s not,” Veiseh said. “Clinicians administer a therapy and then wait four to six weeks to do radiological measurements to see if the therapy is working. You lose quite a lot of time if it’s not the right therapy. The tumor may have evolved into a more aggressive form.”
The THOR device lasts 60 days and can be removed after that time. It is designed to educate the immune system to recognize a cancer and prevent it from recurring. If the cancer is not fully eradicated after the first implantation, the patient can be implanted with THOR again.
Use of AI in THOR Therapy
The researchers plan to spend the next two and a half years building prototypes of the THOR device, testing them in rodents, and refining the list of biomarkers to be utilized in the device. Then, they intend to take an additional year to establish protocols for the US Food and Drug Administration’s (FDA) good manufacturing practices requirements, and to test the final prototype on large animals. The researchers estimate the first human clinical trials for the device will begin in about four years.
“The first clinical trial will focus on refractory recurrent ovarian cancer, and the benefit of that is that we have an ongoing trial for ovarian cancer with our encapsulated cytokine ‘drug factory’ technology,” said Veiseh in the UH press release.
The group is starting with ovarian cancer because research in this area is lacking and it will provide the opportunity for THOR to activate the immune system against ovarian cancer, which is typically challenging to fight with immunotherapy approaches. If successful in ovarian cancer, the researchers hope to test THOR in other cancers that metastasize within the abdomen, such as:
All control and decision-making will initially be performed by a healthcare provider based on signals transmitted by THOR using a computer or smartphone. However, Veiseh sees the device ultimately being powered by artificial intelligence (AI) algorithms that could independently make therapeutic decisions.
“As we treat more and more patients [with THOR], the devices are going to learn what type of biomarker readout better predicts efficacy and toxicity and make adjustments based on that,” he predicted. “Between the information you have from the first patient versus the millionth patient you treat, the algorithm is just going to get better and better.”
Moving Forward
In addition to UH and Rice University, scientists working on the project come from several institutions, including:
More research and clinical trials are needed before THOR can be used in the clinical treatment of cancer patients. If the device reaches the commercialization stage, Veiseh plans to either form a new company or license the technology to an existing company for further development.
“We know that the further we advance it in terms of getting that human data, the more likely it is that this could then be transferred to another entity,” he told Precision Medicine Online.
Pathologists and clinical laboratories will want to monitor the progress of the THOR technology’s ability to sense changes in cancer biomarkers and deliver controlled dosages of antibiotic treatments.
According to an EADV press release, the AI software demonstrated a “100% (59/59 cases identified) sensitivity for detecting melanoma—the most serious form of skin cancer.” The AI software also “correctly detected 99.5% (189/190) of all skin cancers and 92.5% (541/585) of pre-cancerous lesions.”
“Of the basal cell carcinoma cases, a single case was missed out of 190, which was later identified at a second read by a dermatologist ‘safety net.’ This further demonstrates the need to have appropriate clinical oversight of the AI,” the press release noted.
AI is being utilized more frequently within the healthcare industry to diagnose and treat a plethora of illnesses. This recent study performed by scientists in the United Kingdom demonstrates that new AI models can be used to accurately diagnose some skin cancers, but that “AI should not be used as a standalone detection tool without the support of a consultant dermatologist,” the press release noted.
“The role of AI in dermatology and the most appropriate pathway are debated,” said Kashini Andrew, MBBS, MSc (above), Specialist Registrar at University Hospitals Birmingham NHS Foundation Trust. “Further research with appropriate clinical oversight may allow the deployment of AI as a triage tool. However, any pathway must demonstrate cost-effectiveness, and AI is currently not a stand-alone tool in dermatology. Our data shows the great promise of AI in future provision of healthcare.” Clinical laboratories and dermatopathologists in the United States will want to watch the further development of this AI application. (Photo copyright: LinkedIn.)
How the NHS Scientists Conducted Their Study
Researchers tested their algorithm for almost three years to determine its ability to detect cancerous and pre-cancerous growths. A group of dermatologists and medical photographers entered patient information into their algorithm and trained it how to detect abnormalities. The collected data came from 22,356 patients with suspected skin cancers and included photos of known cancers.
The scientists then repeatedly recalibrated the software to ensure it could distinguish between non-cancerous lesions and potential cancers or malignancies. Dermatologists then reviewed the final data from the algorithm and compared it to diagnoses from health professionals.
“This study has demonstrated how AI is rapidly improving and learning, with the high accuracy directly attributable to improvements in AI training techniques and the quality of data used to train the AI,” said Kashini Andrew, MBBS, MSc, Specialist Registrar at University Hospitals Birmingham NHS Foundation Trust, and co-author of the study, in EADV press release.
Freeing Up Physician Time
The EADV Congress where the NHS researchers presented their findings took place in October in Berlin. The first model of their AI software was tested in 2021 and that version was able to detect:
85.9% (195 out of 227) of melanoma cases,
83.8% (903 out of 1078) of all skin cancers, and
54.1% (496 out of 917) of pre-cancerous lesions.
After fine-tuning, the latest version of the algorithm was even more promising, with results that included the detection of:
100% (59 out of 59) cases of melanoma,
99.5% (189 out of 190) of all skin cancers, and
92.5% (541 out of 585) pre-cancerous lesions.
“The latest version of the software has saved over 1,000 face-to-face consultations in the secondary care setting between April 2022 and January 2023, freeing up more time for patients that need urgent attention,” Andrew said in the press release.
Still, the researchers admit that AI should not be used as the only detection method for skin cancers.
“We would like to stress that AI should not be used as a standalone tool in skin cancer detection and that AI is not a substitute for consultant dermatologists,” stated Irshad Zaki, B Med Sci (Hons), Consultant Dermatologist at University Hospitals Birmingham NHS Foundation Trust and one of the authors of the study, in the press release.
“The role of AI in dermatology and the most appropriate pathway are debated. Further research with appropriate clinical oversight may allow the deployment of AI as a triage tool,” said Andrew in the press release. “However, any pathway must demonstrate cost-effectiveness, and AI is currently not a stand-alone tool in dermatology. Our data shows the great promise of AI in future provision of healthcare.”
Two People in the US Die of Skin Cancer Every Hour
According to the Skin Cancer Foundation, skin cancer is the most common cancer in the United States as well as the rest of the world. More people in the US are diagnosed with skin cancer every year than all other cancers combined.
When detected early, the five-year survival rate for melanoma is 99%, but more than two people in the US die of skin cancer every hour. At least one in five Americans will develop skin cancer by the age of 70 and more than 9,500 people are diagnosed with the disease every day in the US.
The annual cost of treating skin cancers in the United States is estimated at $8.1 billion annually, with approximately $3.3 billion of that amount being for melanoma and the remaining $4.8 billion for non-melanoma skin cancers.
More research is needed before University Hospitals Birmingham’s new AI model can be used clinically in the diagnoses of skin cancers. However, its level of accuracy is unprecedented in AI diagnostics. This is a noteworthy step forward in the field of AI for diagnostic purposes that can be used by clinical laboratories and dermatopathologists.
Clinical laboratories and pathology groups should be on the alert to this new digital threat; telehealth sessions and video conferencing calls particularly vulnerable to acoustic AI attacks
Banks may be the first to get hit by a new form of hacking because of all the money they hold in deposit accounts, but experts say healthcare providers—including medical laboratories—are comparably lucrative targets because of the value of patient data. The point of this hacking spear is artificial intelligence (AI) with increased capabilities to penetrate digital defenses.
AI is developing rapidly. Are healthcare organizations keeping up? The hackers sure are. An article from GoBankingRates titled, “How Hackers Are Using AI to Steal Your Bank Account Password,” reveals startling new AI capabilities that could enable bad actors to compromise information technology (IT) security and steal from customers’ accounts.
Though the article covers how the AI could conduct cyberattacks on bank information, similar techniques can be employed to gain access to patients’ protected health information (PHI) and clinical laboratory databases as well, putting all healthcare consumers at risk.
The new AI cyberattack employs an acoustic Side Channel Attack (SCA). An SCA is an attack enabled by leakage of information from a physical computer system. The “acoustic” SCA listens to keystrokes through a computer’s microphone to guess a password with 95% accuracy.
“With recent developments in deep learning, the ubiquity of microphones and the rise in online services via personal devices, acoustic side channel attacks present a greater threat to keyboards than ever,” wrote UK study authors Joshua Harrison, MEng, Durham University; Ehsan Toreini, University of Surrey; and Maryam Mehrnezhad, PhD, University of London.
Hackers could be recording keystrokes during video conferencing calls as well, where an accuracy of 93% is achievable, the authors added.
This nefarious technological advance could spell trouble for healthcare security. Using acoustic SCA attacks, busy healthcare facilities, clinical laboratories, and telehealth appointments could all be potentially compromised.
“The ubiquity of keyboard acoustic emanations makes them not only a readily available attack vector, but also prompts victims to underestimate (and therefore not try to hide) their output,” wrote Joshua Harrison, MEng (above), and his team in their IEEE Xplore paper. “For example, when typing a password, people will regularly hide their screen but will do little to obfuscate their keyboard’s sound.” Since computer keyboards and microphones in healthcare settings like hospitals and clinical laboratories are completely ubiquitous, the risk that this AI technology will be used to invade and steal patients’ protected health information is high. (Photo copyright: CNBC.)
Why Do Hackers Target Healthcare?
Ransomware attacks in healthcare are costly and dangerous. According to InstaMed, a healthcare payments and billing company owned by J.P. Morgan, healthcare data breaches increased to 29.5% in 2021 costing over $9 million. And beyond the financial implications, these attacks put sensitive patient data at risk.
Healthcare can be seen as one of the most desirable markets for hackers seeking sensitive information. As InstaMed points out, credit card hacks are usually quickly figured out and stopped. However, “medical records can contain multiple pieces of personally identifiable information. Additionally, breaches that expose this type of data typically take longer to uncover and are harder for an organization to determine in magnitude.”
With AI advancing at such a high rate, healthcare organizations may be unable to adapt older network systems quickly—leaving them vulnerable.
“Legacy devices have been an issue for a while now,” Alexandra Murdoch, medical data analyst at GlobalData PLC, told Medical Device Network, “Usually big medical devices, such as imaging equipment or MRI machines are really expensive and so hospitals do not replace them often. So as a result, we have in the network these old devices that can’t really be updated, and because they can’t be updated, they can’t be protected.”
But telehealth, according to the UK researchers, may also be one way hackers get past safeguards and into critical hospital systems.
“When trained on keystrokes recorded using the video-conferencing software Zoom, an accuracy of 93% was achieved, a new best for the medium. Our results prove the practicality of these side channel attacks via off-the-shelf equipment and algorithms,” the UK researchers wrote in IEEE Xplore.
“[AI] has worrying implications for the medical industry, as more and more appointments go virtual, the implications of deepfakes is a bit concerning if you only interact with a doctor over a Teams or a Zoom call,” David Higgins, Senior Director at information security company CyberArk, told Medical Device Network.
Higgins elaborated on why healthcare is a highly targeted industry for hackers.
“For a credit card record, you are looking at a cost of one to two dollars, but for a medical record, you are talking much more information because the gain for the purposes of social engineering becomes very lucrative. It’s so much easier to launch a ransomware attack, you don’t even need to be a coder, you can just buy ransomware off of the dark web and use it.”
Steps Healthcare Organizations Should Take to Prevent Cyberattacks
Hackers will do whatever they can to get their hands on medical records because stealing them is so lucrative. And this may only be the beginning, Higgins noted.
“I don’t think we are going to see a slowdown in attacks. What we are starting to see is that techniques to make that initial intrusion are becoming more sophisticated and more targeted,” he told Medical Device Network. “Now with things like AI coming into the mix, it’s going to become much harder for the day-to-day individual to spot a malicious email. Generative AI is going to fuel more of that ransomware and sadly it’s going to make it easier for more people to get past that first intrusion stage.”
To combat these attacks patient data needs to be encrypted, devices updated, and medical staff well-trained to spot cyberattacks before they get out of hand. These SCA attacks on bank accounts could be easily transferable to attacks on healthcare organizations’ patient records.
Clinical laboratories, anatomic pathology groups, and other healthcare facilities would be wise to invest in cybersecurity, training for workers, and updated technology. The hackers are going to stay on top of the technology, healthcare leaders need to be one step ahead of them.
Clinical laboratory managers should note that this company’s new diagnostic offering involving screening embryos for specific genetic conditions is not without controversy
Is the world ready for whole genome sequencing (WGS) of preimplantation embryos to help couples undergoing in vitro fertilization (IVF) treatments know if their embryos have potential genetic health problems? Orchid Health, a clinical preimplantation genetic testing (PGT) laboratory that conducts genetic screening in San Francisco, believes the answer is yes! But the cost is high, and the process is not without controversy.
According to an article in Science, Orchid’s service—a sequencings of the whole human genome of preimplantation embryos at $2,500 per embryo tested—“will look not just for single-gene mutations that cause disorders such as cystic fibrosis, but also more extensively for medleys of common and rare gene variants known to predispose people to neurodevelopmental disorders, severe obesity, and certain psychiatric conditions such as schizophrenia.”
However, Science also noted that some genomics researchers “claim the company inappropriately uses their data to generate some of its risk estimates,” adding that the “Psychiatric Genomics Consortium (PGC), an international group of more than 800 researchers working to decode the genetic and molecular underpinnings of mental health conditions, says Orchid’s new test relies on data [PGC] produced over the past decade, and that the company has violated restrictions against the data’s use for embryo screening.”
There are some who assert that a whole genome sequence of an embryo—given today’s state of genetic technology and knowledge—could generate information that cannot be interpreted accurately in ways that help parents and doctors make informed prenatal testing decisions. At the same time, criticisms expressed by the PGC raise reasonable points.
Perhaps this is a sign of the times. Orchid Health is the latest genetic testing company that is looking to get ahead of genetic testing competitors with its diagnostics offerings. Meanwhile, knowledgeable and credible experts question the appropriateness of this testing, given the genetic knowledge that exists today.
“This is a major advance in the amount of information parents can have,” Orchid’s founder and CEO Noor Siddiqui (above) told CNBC. “The way that you can use that information is really up to you, but it gives a lot more control and confidence into a process that, for all of history, has just been totally left to chance.” Should Orchid Health’s analysis prove useful, pediatricians could order further clinical laboratory prenatal testing to confirm and diagnose potential genetic diseases for parents. (Photo copyright: General Assembly.)
Orchid Receives World-class Support
Regardless of the pushback from some genetic researchers, Orchid has attracted several world-class geneticists and genetics investors to its board of advisors. They include:
Jacques Cohen, PhD, embryologist, co-founder and former director for genetics laboratory Reprogenetics LLC (now CooperGenomics).
Anne Wojcicki, co-founder and CEO of direct-to-consumer genetic testing company 23andMe.
and others.
The WGS test, according to Orchid, detects genetic errors in embryos that are linked to severe illnesses before a pregnancy even begins. And by sequencing 99% of an embryo’s DNA, the test can spot potential health risks that could affect a future baby.
According to its website, the PGT lab company uses the WGS data to identify both monogenic (single-gene) and polygenic (multiple-gene) diseases, including:
Orchid is not without its critics. Knowledgeable, credible experts have questioned the appropriateness of this type of genetic testing. They fear it could become a modern-day form of eugenics.
Andrew McQuillin, PhD, Professor of Molecular Psychiatry at University College London, has concerns about Orchid’s preimplantation genetic testing. He maintains that it is difficult to control how such data is used, and that even the most accurate sequencing techniques do not predict disease risk very well.
“[Polygenic risk scores are] useful in the research context, but at the individual level, they’re not actually terribly useful to predict who’s going to develop schizophrenia or not,” McQuillin told Science. “We can come up with guidance on how these things should be used. The difficulty is that official guidance like that doesn’t feature anywhere in the marketing from these companies.”
McQuillin also stated that researchers must have an extensive discussion regarding the implications of this type of embryo screening.
“We need to take a look at whether this is really something we should be doing. It’s the type of thing that, if it becomes widespread, in 40 years’ time, we will ask, ‘What on Earth have we done?’” McQuillin emphasized.
Redefining Reproduction
It takes about three weeks for couples to receive their report back from Orchid after completing the whole genome sequence of a preimplantation embryo. A board-certified genetic counselor then consults with the parents to help them understand the results.
Founder and CEO Noor Siddiqui hopes Orchid will be able to scale up its operations and introduce more automation to the testing process to the cost per embryo.
“We want to make this something that’s accessible to everyone,” she told CNBC.
“I think this has the potential to totally redefine reproduction,” she added. “I just think that’s really exciting to be able to make people more confident about one of the most important decisions of their life, and to give them a little bit more control.”
Clinical laboratories have long been involved in prenatal screening to gain insight into risk levels associated with certain genetic disorders. Even some of that testing comes with controversy and ambiguous findings. Whether Orchid Health’s PGT process delivers accurate, reliable diagnostic insights regarding preimplantation embryos remains to be seen.
