In our article “Building Faster Isn’t the Same as Building Trust Faster, ” we wrote that a good answer isn’t always the right answer, and that a black box with no visible path to the outcome leaves no room for verification. Here, we take that point a step further: what makes an error invisible, and what does that mean for how you design verification?
Statistical models have had this problem for much longer than the current generation of language models. Any model that makes predictions based on patterns can produce an outcome that appears plausible even when the underlying assumptions are no longer valid. What is new, however, is the scale and speed at which these types of models are now being used in decision-making. This increases the likelihood of an unnoticed deviation.
The tricky part is that an incorrect result often looks just as convincing as a correct one. A figure or recommendation may seem plausible, even though the underlying reasoning has strayed somewhere from what you intended. Was the correct problem solved, or a similar problem that just happened to yield the same result? Without insight into the process, you cannot answer that question.
Consider two errors that happen to cancel each other out in the current situation, or a false assumption that happens to have no visible effect. This kind of coincidence isn’t the most common error, but precisely because of its rarity, it’s rarely investigated. With some of these errors, circumstances change at some point, and the result suddenly becomes visibly incorrect. With others, this never happens: the error remains hidden forever in a result that looked good enough to never be questioned. The latter is no consolation. In fact, it is a stronger argument for verification than the scenario in which an error eventually comes to light.
This means that a system not only provides an answer but also the steps leading up to it: what data was used, what assumptions were applied. In practice, this boils down to a few existing practices. Evals—structured test sets used to periodically measure quality—provide a baseline level of confidence before something goes into production. Human-in-the-loop reviews of decisions with financial or operational impact ensure that an expert reviews the decision before an outcome is implemented. Drift monitoring flags when the data on which a model runs begins to gradually deviate from the training data.
This takes time, which conflicts with the speed of development that makes AI so appealing. The trade-off, therefore, is not always between verifying everything versus never verifying anything, but rather scaling the verification effort in proportion to the impact of the decision. AI is a tool that recognizes patterns, not an entity that knows the truth. Whether an outcome is trustworthy remains a question that humans must be able to answer.
An organization that looks only at the outcome is taking a gamble and calling it a decision. An organization that can also trace the path leading to that outcome makes a decision and can justify it.
Do you want to know if your teams are sufficiently equipped to critically evaluate AI output, rather than relying on the assumption that the system is usually right? We help organizations not only build a solid data platform but also learn to better manage their data and the tools they use. In this context, AI is a tool, not an oracle. Schedule a 30-minute, no-obligation intake session, and we’ll work with you to find the best solution.