Market Research Society Report: Inaccuracies of Generative AI-based Search Tools for Extracting Data

Market Research Society Report: Inaccuracies of Generative AI-based Search Tools for Extracting Data

Most current large language models failed to accurately answer a question focused on UK statistics, in an experiment published by the MRS Census and GeoDemographics Group (CGG).

Given their current trajectory, chatbots/Large Language Models (LLMs) are well on their way to becoming the go-to tool for summarizing information on the web. LLMs are trained to answer natural language questions with well-summarized and confidently expressed text. This makes trusting their output very compelling.

However, due to how current models are built, and even with the supporting information systems surrounding them at runtime, the accuracy of their output should still not be blindly trusted. The authors of this study demonstrated through a simple experiment how most current LLMs failed to answer a UK statistics-focused question accurately.

LLM predictive accuracy is primarily determined by the quality of the training data. Therefore, it stands to reason that data accuracy from an LLM answering questions will be greatly increased if the model is looking at text that contains the answer the user wants answered.

This is the approach the IMF, in collaboration with EPAM, has taken. They released StatGPT, a GenAI-powered platform that significantly improves the way users access the world’s global economic data from trusted high-quality sources such as the IMF, World Bank, Eurostat and National Statistics services.

The main goal of StatGPT is to simplify access to statistical data, ensuring it is easily accessible not only for data analysts and statisticians but also for individuals seeking a deeper comprehension of diverse aspects like economy, population, migration and more.

StatGPT acts as an intermediary between the user and the data provider, translating a data query written in the user’s native language into an SDMX data query. To do this, it uses SDMX-compliant data queries to source its RAG data. Created in the early 2000s, SDMX standardizes statistical data queries across data providers. StatGPT’s process ensures that the model is looking at accurate data from trusted sources rather than relying on a web search algorithm to return a relevant page.

Part of the reason StatGPT achieved this level of reliability and speed lies in its foundation. It was developed on top of DIAL, an open-source enterprise platform that provides the underlying components for model orchestration, monitoring, and reproducibility, as well as building blocks to agentic orchestration. This allowed the IMF and EPAM teams to focus on the statistical logic and data alignment, while DIAL ensured that every interaction remains deterministic, traceable, auditable, and consistent across sessions. The result is a system that combines the accessibility of conversational AI with the determinism expected from official statistical tools.

Read the full study findings here.

Learn more about EPAM DIAL at: https://epam-rail.com/platform.

Discover how EPAM is delivering real value for businesses through GenAI: https://www.epam.com/services/artificial-intelligence/generative-ai