New Data on LLM Accuracy

Print Friendly, PDF & Email

Juan Sequeda, Principal Scientist at, recently published a research paper, “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases.”

He and his co-authors benchmarked LLM accuracy in answering questions over real business data (from an insurance co.) and found that responses for basic queries were accurate only 22% of the time. With intermediate/expert-level queries, accuracy dropped to 0%.

Looking closer, the results become even more shocking. The LLM answered “What are all the premiums that have been paid by policyholders?” wrong 74.5% of the time. An even simpler question like “How many policies do we have?” had an incorrect response rate of 62.6%.

As enterprises invest in LLMs, these findings paint a bleak picture. Poor accuracy could render LLMs obsolete or worse – lead to erroneous analytics, forecasts, strategic business decisions, etc. There’s a lot of data to explore here.

Sign up for the free insideBIGDATA newsletter.

Join us on Twitter:

Join us on LinkedIn:

Join us on Facebook:

Speak Your Mind