AI Providing Mixed Results For Medical Information Inquiries

The study led by the University of California, Riverside found that both AI internet information-gathering services have some strengths and weaknesses for those seeking information about Alzheimer’s disease and other forms of dementia. Dementia and related conditions such as Alzheimer's disease were chosen to be the topics of focus as over 6 million Americans are impacted by these diseases making it a popular topic of research by both families and patients. 

"Research also shows that caregivers of people living with dementia are among the most engaged stakeholders in pursuing health information, since they often are tasked with making decisions for their loved one's care," said co-author Nicole Ruggiano, a professor of social work at the University of Alabama.

The researchers submitted 60 inquiries to Google and ChatGPT that represented typically questions that might be submitted from people living with dementia and/or their families and caregivers. Half of the questions sought information about the disease process and the other half sought information about services that could help patients and their families. The results were mixed. 

"If you pick the best features of both, you can build a better system, and I think that this is what will happen in the next couple of years," said Vagelis Hristidis, a professor of computer science and engineering in UCR's Bourns College of Engineering.

"Google has more up-to-date information, and covers everything," Hristidis said. "Whereas ChatGPT is trained every few months. So, it is behind. Let's say there's some new medicine that just came out last week, you will not find it on ChatGPT."

While outdated, the researchers suggest that ChatGPT provided more reliable and accurate information than Google, because it is trained better with the more reliable websites through computationally intensive training, while Google has reliability issues because it uses everything from reliable sources to advertisements, including those that pay Google for their website to appear at the top of search results. 

"Although there is no cure for the disease, many clinical trials are underway and recently a promising treatment for early-stage Alzheimer's disease was approved by the FDA Therefore, up-to-date information is important for families looking to learn about recent discoveries and available treatments,” Co-author Ellen Brown, an associate professor of nursing at the Florida International University.

The authors note that "the addition of both the source and the date of health-related information and availability in other languages may increase the value of these platforms for both non-medical and medical professionals." 

The authors also noted that both platforms scored very low for readability scores, making it difficult for people with lower levels of education and those with low medical/health literacy and low comprehension skills to be able to understand and utilize. 

"My prediction is that the readability is the easier thing to improve because there are already some tools, some AI methods, that can read and paraphrase text," Hristidis said. "In terms of improving reliability, accuracy, and so on, that's much harder. Don't forget that it took scientists many decades of AI research to build ChatGPT. It is going to be slow improvements from where we are now."