While artificial intelligence, or AI, tools like ChatGPT might be great for helping you pick where to go for dinner or which TV show to binge watch, would you trust it to make decisions about your medical care or finances?
AI tools like ChatGPT and Gemini include a disclaimer that the information they find scanning the internet may not always be accurate. If someone was researching a topic that they didn’t know anything about, how would they know how to confirm the information as truth? As AI tools become smarter and gain more widespread use in daily life, so do the stakes for the accuracy and dependability of using this evolving technology.
Michigan State University researchers aim to increase the reliability of AI information. To do this, they have developed a new method that acts like a trust meter and reports the accuracy of information produced from AI large language models, or LLMs.
Reza Khan Mohammadi, a doctoral student in MSU’s College of Engineering, and Mohammad Ghassemi, an assistant professor in the Department of Computer Science and Engineering, collaborated with researchers from Henry Ford Health and JPMorganChase Artificial Intelligence Research on this work.
“As more people rely on LLMs in their daily work, there’s a fundamental question of trust that lingers in the back of our minds: Is the information we’re getting actually correct?” said Khan Mohammadi. “Our goal was to create a practical ‘trust meter’ that could give users a clear signal of the model’s true confidence, especially in high-stakes domains where an error can have serious consequences.”
Though a person can repeatedly ask an AI tool the same question to check for consistency — a slow and energy costly process — the MSU-led team developed a more efficient internal approach. The new method called Calibrating LLM Confidence by Probing Perturbed Representation Stability, or CCPS, applies tiny nudges to an LLM’s internal state while it’s forming an answer. These nudges “poke” at the foundation of the answer to see if the answer is strong and stable or weak and unreliable.
“The idea is simple but powerful, and if small internal changes cause the model’s potential answer to shift, it probably wasn’t very confident to begin with,” said Ghassemi. “A genuinely confident decision should be stable and resilient, like a well-built bridge. We essentially test that bridge’s integrity.”
The researchers have found that their method is significantly better at predicting when an LLM is correct. Compared to the strongest prior techniques, the CCPS method cuts the calibration error — the gap between an AI’s expressed confidence and its actual accuracy — by more than half on average.
“The CCPS method has profound clinical implications because it addresses the primary safety barrier for LLMs in medicine, which is their tendency to state errors with high confidence,” said Kundan Thind, co-author on the paper and division head of radiation oncology physics with Henry Ford Cancer Institute. “This method improves an LLM’s internal confidence calibration, enabling the model to reliably ‘know when it doesn’t know’ and defer to human expert judgment.”
This breakthrough has been tested on high-stakes examples in medical and financial question-answering, and its potential to enhance safety and trust in AI is vast.
This research was recently presented at the Conference on Empirical Methods in Natural Language Processing in China, where it was nominated for an Outstanding Paper Award — a distinction placing the work in the top 0.4% of more than 8,000 submissions to the conference.
Research funding was provided by the Henry Ford Health + Michigan State University Health Sciences Cancer Seed Funding Program and by the JPMorganChase Artificial Intelligence Research Faculty Research Award.
