Google’s Medical LLM Is Proving More Accurate

A study carried out by Google researchers and published in Nature reveals that tech giant’s generative AI technology, Med-PaLM, provided detailed answers aligned with scientific consensus on 92.6% of questions, on par with clinician-generated answers at 92.9% .

Med-PaLM is a generative AI technology that uses Google LLMs to answer medical questions.

The researchers used MultiMedQA, a standard combining six existing medical question datasets covering the scope of research, occupational medicine, and consumer queries, and HealthSearchQA, a frequently searched medical question dataset.

MultiMedQA questions were asked via PaLM, a 540 billion parameter LLM, and Flan-PaLM, its instruction-tuned variant.

Responses were then subjected to human ratings to assess understanding, reasoning, factuality, and possible prejudice and bias.

Using various prompting strategies, Flan-PaLM was found to be accurate in answering the MultiMedQA dataset, with an accuracy of 67.6% on US Medical Licensing Exam-type questions, exceeding accuracy levels previous 17%. Still, researchers have noted significant shortcomings in its responses to consumer medical questions.

Therefore, the researchers introduced Quick Instruction Tuning, an efficient data and parameter alignment technique, resulting in Med-PaLM, which revealed significantly more accurate responses (92.9%) than Flan-PaLM ( 61.9%).

Flan-PaLM responses were also rated as potentially causing adverse outcomes 29.7% of the time, compared to 5.9% of the time for Med-PaLM. The accuracy of clinician-generated responses was similar to Med-PaLM at 5.7%.

The researchers acknowledged that many limitations still need to be overcome before the models are viable for clinical use, and further evaluation is needed, particularly with respect to safety, bias, and fairness.

“We hope that LLM systems such as Med-PaLM, which are designed for medical applications where safety is paramount, will democratize access to high-quality medical information, especially in geographies with a limited number of medical professionals. healthcare,” Vivek Natarajan, an AI researcher at Google and one of the study’s researchers, said on LinkedIn.

“And ultimately, with further development, rigorous validation of safety and efficacy, we hope that Med-PaLM will find wide adoption in direct care pathways, thereby increasing our clinicians, reducing their administrative burden, helping to clinical decision-making, giving them more time to focus on patients and overall making health care more accessible, equitable, safer and more humane.”

THE GREAT TREND

In March, the technology company’s Med-PaLM 2 tested on US Medical Licensing Examination-style questions, performing at an “expert” test level with over 85% accuracy. He also received a passing grade on the MedMCQA dataseta multiple-choice data set designed to answer actual medical entrance exam questions.

A month later, the company announced Med-PaLM 2 will be available to select Google Cloud customers in the coming weeks to share feedback, explore use cases, and for limited testing.

The company also announced a new AI-powered claims acceleration suite, created to facilitate the pre-authorization process and processing of health insurance claims. The Suite converts unstructured data (data sets not organized in a predefined way) into structured data (highly organized and easily decipherable data sets).