Value Driven Data Science

[00:00:00] Dr Genevieve Hayes: Hello, and welcome back to Value-Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and I'm here again with Jia Huang, lead research engineer at Astar and author of Designing AI Agents. Last week, Jia and I discussed how data scientists need to rewire their thinking to succeed in the agentic AI era.
[00:00:29] Today, in this Value Boost episode, we're exploring why silent correctness is the most dangerous failure mode in agentic AI systems and what you can do to avoid it. Welcome back, Jia.
[00:00:44] Jia Huang: Hey, thank you Genevieve. Yeah, silent correctness is one of the failure mode I worry about most. I'm glad you brought it up. So let's start with a simple situation. Where a credit card holder ask his agent, "I'm traveling to China. Which airport lounge I can use today?"
[00:01:06] The agent checks the profile and says, "Yes, you are platinum card holder. You are eligible. Go Singapore Changi Airport." Confident. Then the traveler heads to the airport. But the agent missed three things they hadn't enrolled in the offer. The benefit only applies in certain countries, and that lounge wasn't in the current network.
[00:01:32] The card tier is true, but the eligibility decision was wrong. That is the dangerous kind. The agent didn't hallucinate. It used a real fact and missed the constraints that make the answer false. A partly correct answer produced a completely wrong decision
[00:01:54] Dr Genevieve Hayes: I remember back when I was writing deterministic programs to create business intelligence reports. One of the things that used to frighten me the most, it wasn't the loud catastrophic errors that crashed the entire system and got a phone call from the system administrator.
[00:02:14] They were the ones where everything would run perfectly correctly, and the numbers would even come out being something that looked close enough to the truth so that it didn't freak me out. And then you'd realize six months later that you'd been reporting absolute rubbish and you had to go and have a talk to one of the executives and explain why your report was rubbish.
[00:02:36] This sounds like the agentic AI equivalent to that.
[00:02:40] Jia Huang: Yes, exactly. Load failures are easy. The pipeline crashed, the query returns null. You immediately know where is the problem, then you fix it. The silence failures are different. , In data science field, the analysis run fine, the chart looks fine, the report even looks very professional, but the denominator is wrong or a joint is missing.
[00:03:08] Nothing kills you, but data scientists, you can catch it. That's why we are again, very familiar with this kind of a situation. The report looks all good, very polished, but how about the real content? That is the thing. As a programmer, a system engineer, I don't care. My database, no mistake. Online, it's okay.
[00:03:35] But as a data scientist, we know what is your real analysis? That's the thing. Again, we are very familiar with the silent failure. Now, in the agent era, I call it silent correctness because it's up one level, up to the whole system level already. It's not just a data analysis report.
[00:04:01] It's the whole system. It's "I'm going to airport. Can you tell me which air ticket I'm going to buy?" This kind of failure result disaster. That's why I call it the most dangerous failure mode
[00:04:17] Dr Genevieve Hayes: Why is this more dangerous than a hallucination?
[00:04:20] Jia Huang: Maybe let me first give you another few ex-examples, then we will talk about more. Let's say a agent write a weekly summary. So great news, churn dropped from eight percent to five percent. We have beautiful chart, confident recommendations, then we keep the current strategy. But a data scientist checks the denominator, then he found the agent counts free trial users as customers, and then the marketing just add a lot of free trial users last week.
[00:05:00] So the base got bigger, the paid churn didn't actually improved, it's just looked better. Nothing crashed. The language was also professional. The business conclusion was simply wrong. This is silent correctness. Confident output, plausible format, but you have a wrong denominator, and that is the hard part.
[00:05:25] The agent is right most of the time, so people relax and stop checking. Then the one wrong denominator slips through. Auditing this kind of a system is harder because auditing a polished answer with hidden assumption is harder than doing the analysis yourself. As data scientist, I generate the report from the beginning.
[00:05:53] I don't want to see something like, looks very smooth, but I need to check where the data is getting from, then why today the agent check this and not that. To find a error is difficult, especially when this error looks correct
[00:06:11] Dr Genevieve Hayes: Yes, that was what I was gonna ask. In the old deterministic programming world, I know how I would fix that dodgy denominator problem. That's just a matter of me fixing my logic in the code, and I can easily check that by checking the calculation formulae that are embedded within the code. But if the agent itself is creating that formula to calculate whatever it is with that wrong denominator in it
[00:06:40] How do you stop it from getting there in the first place, and how do you pick it up once it ends up there?
[00:06:45] Jia Huang: Yes. I think three things we have to paying attention here is, first of all, I think we stop treating confidence as correctness because the agent, when we talk to Claude, ChatGPT they always very confident. The agent tone is not evidence. I think now we can understand this part. Then the second is we have to sample and audit.
[00:07:09] If we never sample, we will never see any mistakes. The third one is we have to demand provenance. Every important output should trace back to its source, assumptions and steps. Did this number come from? Which rule? Which version? Everything should be tracked again, these three things are the basic principles, and there are some more techniques, like in the agent design pattern, we can have self-reflex pattern.
[00:07:46] So means a agent generate a result, but that's not end of the game. You have to think again and check whether you are right or wrong. This is one pattern. Another pattern could be engage another agent. Means we use the ChatGPT to do one conclusion, then maybe we will engage Claude to actually challenge ChatGPT's answer to see whether there's a consistent result or there should be some discrepancy the other agent can find.
[00:08:18] This kind of a design pattern will help a little bit.
[00:08:21] Dr Genevieve Hayes: And there's probably still a place for the old style checking , if I was doing this in the old pre-AI days I would literally perform a couple of sample calculations by hand and then check to make sure my code was outputting what I'd calculated, just with a pen and calculator. If you were doing just a few sample calculations by hand, you could pick up some of these errors because you'd be able to say, "Hang on, this isn't doing what I think it should be doing."
[00:08:51] Jia Huang: Yes. , It comes back to our profession. Our experience in data science is never obsolete. We need all this knowledge and experience to control the agentic system. The goal isn't a perfect agents, is how we as system designer to make important design and decision to that the failures visible.
[00:09:21] It's our job to have these techniques to catch the failures. I can give you four techniques here.
[00:09:28] Dr Genevieve Hayes: That would be good
[00:09:29] Jia Huang: First, we can have independent oracles, so we run a second method or maybe even certain situation, we can just run another deterministic rule, and that the disagreement be the signal. Agent can give us something, but we have our own rules.
[00:09:49] Is that really correct? If the agent says eligible, but the rule engine say no, then the gap trigger a human review. Then the second is property testing. Don't test whether the answer looks good. Test whether the important metrics hold. Is the denominator paid users or all users?
[00:10:11] Was the right policy version used? These are invariance checks. The data scientists already think that way. The third method is continuous evaluation in production system. Silent failures show up when the input shifts, a new region, a new format, a new rule coming.
[00:10:32] So when the situation changed, the environment changed. The evaluation has to keep running after launch, not just before go online. Fourth is preserve provenance. When you do catch a failure, you need to replay the session, what this agent saw, what it retrieved, and where the wrong belief entered. This is how an invisible failure becomes a debuggable one
[00:11:04] Dr Genevieve Hayes: If you could design one safeguard into every agentic AI system, what would it be?
[00:11:10] Jia Huang: I think the only one thing important is full reasoning provenance end to end from every important output. Not because it prevents failures, it doesn't, because it make them recoverable. When something breaks six weeks later, we can replay what's the agent retrieved, which source it trusted, and which wrong belief that got in,
[00:11:35] so my final message is this: make provenance non-negotiable. The way reproducibility is non-negotiable in research, that is the line between impressive AI and trustworthy AI
[00:11:53] Dr Genevieve Hayes: And that's it for today's conversation with Jia. If you haven't already, listen to our previous episode where we discussed the impact of the AI paradigm shift on how data scientists should think about their work. Thanks for joining me again, Jia
[00:12:09] Jia Huang: Hey, thank you very much, Genevieve. It's so nice talking to you
[00:12:13] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.