Not a Famous Fact Checker with High Accuracy

It doesn't take much to get ChatGPT to make a factual mistake. My son is doing a report on U.S. presidents, so I figured I'd help him out by looking up a few biographies. I tried asking for a list of books about Abraham Lincoln and it did a pretty good job: A reasonable list of books about Lincoln. Screen capture by Jonathan May/CC BY-ND · Number 4 isn't right. Garry Wills famously wrote "Lincoln at Gettysburg," and Lincoln himself wrote the Emancipation Proclamation, of course, but it's not a bad start.

Our test results reveal that AI is not yet a reliable fact-checking tool. It lacks contemporary knowledge, it loses perspective, and tells you what you want to hear and not always the truth.

LibGuides: Student Guide to ChatGPT: Fact checking is always needed

As large language models (LLMs) like ChatGPT become more relevant and advanced, many writers and editors may be concerned about how these models will affect their careers. One area of particular interest is determining if ChatGPT could be used to detect the accuracy of information. The following study set out to determine whether writers and editors—the traditional fact-checkers in publishing—could be replaced by emerging LLMs when it comes to verifying facts.

While the fantastical possibilities of generative AI can be awe-inspiring, it is important to not become passive while using it. When using systems like ChatGPT, try to remember the saying "trust, but verify". This saying simply means that you should always do your due diligence to fact check the system and ask it to cite it's sources.

According to OpenAI, it's "40% more likely to produce factual responses than GPT-3.5 on our internal evaluations." But it's still not perfect. So verification of the output is still needed. ChatGPT often makes up fictional sources One area where ChatGPT usually gives fictional answers is when asked to create a list of sources.

If you are using the free version of ChatGPT, which does not link to sources, it’s important to do a quick web search to verify the information. Look for more than one source to check the validity of what the AI is saying. Resources like Wikipedia or mainstream news sites that employ fact-che...

All the experts agreed that ChatGPT is not yet reliable or accurate enough to be used as a fact-checker. The technology is improving, but Corney said it is “extremely challenging” to get an AI to reliably determine or recognize the truth.

ChatGPT scores 88.7% on the MMLU general-knowledge benchmark, but its factual accuracy on short Q&A drops to 49% with a 51% hallucination rate, according to OpenAI's own system card.