Value Driven Data Science

[00:00:00] Dr Genevieve Hayes: Hello, and welcome to Value-Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and today I'm joined by Jia Huang. Jia is a lead research engineer at A*STAR, Singapore's Agency for Science, Technology and Research, and is the author of multiple books on AI engineering and agent design, including Designing AI Agents and RAG from First Principles.
[00:00:34] His work focuses on turning agentic AI from impressive demos into reliable, auditable, and value-producing engineering systems. In this episode, we'll explore how data scientists need to rewire their thinking if they want to succeed in the AI era, and what that means for how you think about risk, trust, and accountability in your work.
[00:01:00] Jia, welcome to the show.
[00:01:02] Jia Huang: Hi. Hi. Thank you, Genevieve
[00:01:04] Dr Genevieve Hayes: I once worked in an organization where every member of the data science team was assigned an $8,000 laptop. It sounds way cooler than it was. We're not talking about the Rolls-Royce of MacBooks here. Rather, it was a massive brick of a device that ran Ubuntu Linux and was so heavy it rarely left my desk.
[00:01:26] My team somehow managed to convince our employer to buy them, because fitting the machine learning models my employer needed required powerful GPUs, and with security concerns limiting our cloud access, this was considered the only way. At that time, and in that context, compute power was the bottleneck to what our team could deliver, and over time, the relationship between compute and model capacity became a fundamental truth to me, as reliable as gravity.
[00:01:56] Reading about AI agents recently, however, I found myself suddenly questioning this truth. When dealing with agents, it turns out, compute is no longer the bottleneck. Context is. The rules of the game had suddenly changed. Now, data scientists are better placed than most to thrive in the AI era. But to do so, we need to learn the new rules and unlearn some of the old ones.
[00:02:25] And Jia, while most data professionals haven't even begun thinking this way, through your writing, you've been mapping the fundamental differences created by the shift to the new AI paradigm and what these mean for how data scientists and software developers work. Your background is in software engineering, where deterministic programming has historically been the norm.
[00:02:50] But your focus has now shifted to agentic AI design. What has the paradigm shift implied by that transition looked like for you?
[00:03:00] Jia Huang: Yeah, that is very great question, Genevieve. I want to start with something counterintuitive. Data scientists are one of the professions best prepared for the agentic AI era, not because they know every framework, because they have spent years working with uncertainty. Here is the part I really want to land.
[00:03:24] A data scientist and an agent come from the same family. Both are probabilistic systems. Both reason under uncertainty. So this isn't a foreign field data scientists have to enter. It's their own field with one new piece added. In the old-style programming, we write rules. Input goes in, fixed logic runs, output comes out.
[00:03:53] In agentic AI, we design a system that read context, reasons through the steps, call tools, and sometimes decides what to do next. A-agentic AI is not ordinary software. Agentic AI is not just a model API call also. It is a software with a thinking process inside. So for a data scientist, the shift is this: we move from predicting a outcome to designing a decision process.
[00:04:27] From tuning one accuracy number to managing uncertainty across a whole system. So my message is very simple. Nothing we know is obsolete. Everything we know is one layer more valuable
[00:04:43] Dr Genevieve Hayes: Okay. There's a lot to unpack there
[00:04:45] Jia Huang: Yeah. So maybe I give you a example to explain what I mean.
[00:04:49] Dr Genevieve Hayes: Yes, that would be very helpful.
[00:04:51] Jia Huang: Yeah, maybe a quick example from my own work. Let's say I build a agent that read a family pedigree and write some clean analysis, real looking citations, confident conclusion, but this agent treat siblings and cousins as independent samples.
[00:05:13] They are not. We know related people are correlated, so the real sample size is much smaller. A data scientist catches that in a second. This is not my agent problem. It's not agentic AI problem. It's the old statistic problem hiding inside a new system. So that is the point. The expert doesn't disappear.
[00:05:41] They move up one level from doing the analysis yourself to designing the checks around the agent
[00:05:49] Dr Genevieve Hayes: Okay. So under the old paradigm, you would have a data scientist who was literally doing that analysis testing for things like correlations, understanding probability distributions, developing models that are fundamentally probabilistic in nature. Under this new paradigm,
[00:06:11] A lot of that probabilistic thinking has now become folded into the agent design. So someone like me as a data scientist who is working with an AI agent would then have to understand how to check to make sure the probabilistic logic is being properly implemented.
[00:06:37] Jia Huang: Exactly. Yeah, you make the point. So the day-to-day work changed to system design. Maybe I just give more examples on how this changed to our day-to-day work So the old workflow is operations ask, why yesterday's order dropped. You pull the data, check a few things, write a short analysis, then your deliverable is a notebook or a report, submit to your boss.
[00:07:05] Agentic version is the team wants a agent that answer the same question every morning on its own. Reads the data, checks the unusual suspects, explain what's changed, suggest what to look at next. So your new job now is different. You are not producing one analysis every morning. You are designing a system that keeps making decisions every day, live, keep making these analysis-like decisions.
[00:07:39] This change our work in three ways. First, evaluation should move online. Before, you test a model once, then you go ahead. Now the system face new cases every day, a delayed pipeline or a changed log format or slightly different question, so evaluation cannot just stop at launch. The sample reviews and your watch also drift.
[00:08:08] Is it still making good call in the real workflow? So that's why we actually need to evaluate the agents from time to time. This is the first thing changed. Then the specification become the deliverable. Before we do the analysis by ourself, we can clarify, we talk with our colleague, and then we deliver it.
[00:08:32] But with a agent, you specify its behavior upfront, what it should do, what it must never do, when it stops, when it escalate to a human. All these are specifications. That spec is not a meeting notes anymore. It's part of your product. So the second key change here is the specification. Then the third one is the observability.
[00:09:01] The observability also becomes part of the system. If someone question your notebook, we can open it and show our steps, but now the agent need give you a solution. He needs the same thing where your result come from. So every important retrieval should be logged. Every tool the agent call should be traceable, and every key decision should be replayable.
[00:09:28] So six weeks later, someone ask why did this agent recommend this product A but not product B? And where did the wrong assumption get in? This is the shift from doing the analysis to designing a solution under which analysis can be repeatedly safely. That is the data science becoming system design
[00:09:53] Dr Genevieve Hayes: So basically what you're designing is a data science employee, and the data scientist now moves up one step in the ladder and becomes the manager of that data science employee. So just like you would with a human data science employee, you have to be able to monitor their work, and you have to have enough observability so that you can do the equivalent of if something goes wrong, literally go and say to that agent, "Can you tell me what you did?"
[00:10:25] And that's in the form of doing things like those logs and everything you just described, so that it's not just some sort of black box
[00:10:36] Jia Huang: Yeah. Agree. One of the main change in my mind is in the agentic era, everybody, including the programmers like me, we have to be one level up. Before AI comes into the picture, I just write a program, I submit to my team. Now I need to make sure the agent writes the correct code.
[00:11:01] So the work of me is actually coordinating, designing, and collaborating and make sure there should be no disaster happen, that there should be enough testing case ready before I can release this piece of code generated by AI agent. I think same thing happened to data scientist, but we are more used to it because we are from , like I mentioned, the probability world.
[00:11:30] Dr Genevieve Hayes: How do you do this efficiently? Because when you're writing deterministic code, you can just create the unit tests that are things like show that X is equivalent to Y. But when you've got non-deterministic outputs, you can't just say, "In this case, the output will be exactly this text." I'm really struggling to get my mind around what does a test look like in the non-deterministic world?
[00:11:58] Jia Huang: So we have to first understand where the uncertainties come from that's why we split this uncertainties into three type of uncertainties. Let me still use some examples to explain. A agent answers every morning, "Why did orders drop yesterday? And what should we check?" Sound simple, but uncertainties show up right away.
[00:12:27] First, the reasoning itself. Run it three times, you can get three different explanations from the same data. Could be payment failure, inventory, could be a traffic drop. That is cognitive uncertainty. It is not just the final answer, it is in the past. A small difference at step two changes the recommendation at step 10.
[00:12:53] As a designer, we have to understand the system first. We need to know the failure could come from payment, could come from inventory, could come from traffic drop. The design part never replaced by agent. We are still in the center, so we have to monitor the different source to avoid this cognitive uncertainty
[00:13:20] Dr Genevieve Hayes: So that's monitoring the inputs to the system
[00:13:22] Jia Huang: Correct. We will come to this later in the context engineering part. . So then the second uncertainty is the tool uncertainty. It also worse than a normal pipeline. So normal pipelines, I call database A, then I call database B, then I run some report to get the inventory. That is the fixed pipeline.
[00:13:47] So because the agent also now choose which tool to call the failure can be wrong tools, wrong source, and wrong reading. So in this process, we also need to clearly tracking which tool a agent is calling. Then some validation and the process workflow should be there to prevent the agent calling the wrong tool.
[00:14:13] Again, this is a human being's design. It's not like I use ChatGPT and CloudCode, just generate some PPT for me, or just generate a report for me. It's a no design. You are just put everything to the agent to do, but the real system is not like that. We are still in the center. We need to carefully design what tools I allow this agent to call, what tools the agent cannot call.
[00:14:45] So everything is actually in control. Otherwise, it's a very dangerous situation. So this is the second uncertainty, the tool cause. The third one is the environmental uncertainty. This is like distribution shift. But now it's at the level of a whole system. So where a agent is wrong, the first question should not be was the model wrong?
[00:15:11] Can we change to a smarter model? No. It should be where did the uncertainty get in? The reasoning, the tool, or the environment, what changed? Until the system can answer that question, we don't really have accountability. We have hope
[00:15:31] Dr Genevieve Hayes: The environmental shift, sounds a lot like data drift that you'd get in machine learning models.
[00:15:37] Jia Huang: Yes
[00:15:37] Dr Genevieve Hayes: I would say a lot of the techniques that data scientists have previously used for dealing with data model-modeling can be transposed over for checking for environmental uncertainty. But I would say that in particular that uncertainty around the tools, that's something that none of us would have ever dealt with previously
[00:16:03] Jia Huang: Yeah. Exactly. That's why now I want to emphasize on observability. So say six weeks later, someone ask why did the agent blame payment failure? The final answer is not enough. We have to replay the past. So what data the agent saw, which log the agent used, whether the table was complete or not, or what the agent ignored.
[00:16:33] All these question needs to be answered six weeks later, so I will build observability on these things, a trace of the important reasoning steps to understand how the agent thinks. Not every model, I think, have this tech called thinking mode, or you can know what is the thinking logic behind it.
[00:16:58] Then the provenance of every fact it used, so where it came from and when. And also we need to have the replayab- replayability, so we can reconstruct the whole session later. Currently most systems log the happy path only, then it lose everything else. I think that is the big gap. I will call it observability.
[00:17:26] Once we have that one, we will use the traditional way we are familiar to analyze the reason of right or wrong
[00:17:35] Dr Genevieve Hayes: With the replayability, does that involve setting some sort of random seed?
[00:17:40] Jia Huang: Yes, the random seed is always there, it's by nature, right? That's maybe the difficult part of the large language model, because human being is not able to fix large language models thinking, which is impossible. Because I'm a data scientist myself, I know in some of the machine learning model, if I set the random seed as for example, forty-two, then every time I run it, I could have the same result.
[00:18:12] So I reproduce my analysis. But now it's getting much harder. The large language model can really give you a different answer, even you give them the same context. So how do I control? So the whole session we are talking about this, is you need to use the guardrail, the design.
[00:18:36] You need to actually think for them what could be wrong, A, B, C, D, E, F. You need to make sure the system didn't go haywire. So you have to give the agent some freedom to move freely, and then you control. Okay, the bottom line is you have to check again when you are not sure. The bottom line is you should really care about the documents version.
[00:19:06] Don't give me the wrong version. So we still need to go to different system and different scenario to design one by one. There is no silver bullet to say, yeah, I give you this idea that every problem resolved. Not that. But the idea still be be aware of all these uncertainties will come from the real scenario, and use enough guardrails to protect your system, and also use the observability to make the failure clearer than before.
[00:19:41] Dr Genevieve Hayes: So it's not so much about ensuring that the output is word for word identical in a text output it's making sure that you get the same essential result or the same record or the same numerical answer if you're doing something mathematical, for example.
[00:19:57] Jia Huang: Yes. At least you know why the system got wrong, why the agent think this way. You have the trace of what you put in and how it thinks, and then which tool it calls and how this environment change comes to a wrong condition.
[00:20:17] Dr Genevieve Hayes: I'm just thinking about how much text gets output whenever I have a conversation with Claude or ChatGPT. Aren't you gonna end up with massive trace files going down this path?
[00:20:29] Jia Huang: Yeah. Actually, in ChatGPT and Claude, sometimes we couldn't have enough trace in their user because they are basically a product, they are agent, but the product is going to show you the actual answer. It's not show everybody the tracing and the logic they are doing the thinking behind.
[00:20:56] So that's also the dangerous part, when we design a system, we need exactly to do the the other way. We are showing a result, but we are not satisfied with only show customer a result. We need to provide the result and the log and the whole traceability behind the scene. So only with that, our system is complete
[00:21:25] Dr Genevieve Hayes: That would be a file saying things like, "At step one, the agent does blah, blah, blah. At step two, it does blah, blah, blah and gets this sub-answer, which feeds into step three." And so you'd have... You'd basically be able to understand the input and output to each step in the process
[00:21:43] Jia Huang: You get it, yes. When you talk with ChatGPT, you don't want to see that, right? We just want to see a clear answer. However, I think one of my friends give me a very good analogy. See it's something like you have a Rolex a nice watch. Yeah, Every day you just look at the time.
[00:22:05] That is the facade. So it's easy for you to get the time at this moment. But do you want to see , what happening really inside? Then you turn the watch around. So sometime there's , transparent glass, you can see every small part is actually coordinating, working together with each other.
[00:22:27] Then you know, oh, now I know how the seconds, the hours, the minutes runs. Oh, it's this way. Now I really understand how the system design. He said, "Our agent system should be like this." People just want to see quick answer. Okay, I give you a quick answer, but do you want to see details? You have to.
[00:22:48] Without that one, you are only halfway done, and it is dangerous
[00:22:54] Dr Genevieve Hayes: Yep, got it. And yeah, I think you've just defined the two types of people in the world, those who wanna see the guts of a watch and those who don't. Returning back to something we touched on at the start of the episode I was talking about how when dealing with agentic AI systems, the bottleneck for data scientists has now shifted away from compute to context.
[00:23:16] Why are things suddenly different in that respect?
[00:23:19] Jia Huang: Yeah, that's also a good question. I'd like to say the context engineering is new feature engineering. Twenty years ago, we learned that more feature aren't always better. If you have more features, you have more noise, overfitting, leakage. The agent have the same property, so more context is not always better.
[00:23:45] Quick example is there's a support team wants this agent to answer why this customer not eligible for a refund. The naive move is to hand it everything, all the policies, all the ticket, all the history. It feels very helpful. Usually, it will make the agent worse because it could read a outdated policy or miss the customer's country or confuse the old exception with the current rule.
[00:24:19] So a better system design is like this: select first which customer, which order, which policy version was active on this purchase date. So same model, you got better answer. It sound like very similar to the feature selection, right? That's why I say the context is the bottleneck, not compute. Most teams think they need a bigger model, but usually what they need is a better context selection.
[00:24:54] A good agent is not a system that can think, it's actually the one knows what to look at before it thinks
[00:25:03] Dr Genevieve Hayes: Okay so the way I'm seeing it is with your average LLM, so say the Claude model, you've got a context window which places a hard limit on the amount of information you can put in. So that's placing a hard limit on the dimensionality of your data. Most people would just try and cram in as much information as possible which I have probably done in the past myself, and so that would create a bottleneck.
[00:25:33] But what you're saying is that's the wrong approach. You're gonna be contending with the curse of dimensionality there. What you should be doing is thinking about this like a data scientist would, which is what are the features that have the greatest predictive power, and just focus on giving that information to the AI agent so that you can get the most effective answer given the context window that you've been provided with.
[00:26:02] Jia Huang: Exactly. Yeah, that's why a data scientist already have the instinct for it. This is really a feature selection wearing new clothes. We shouldn't dump everything into the context window. We should select the feature first. Yes, you are right
[00:26:22] Dr Genevieve Hayes: So the message that I'm getting from this whole conversation is data scientists, because we've been dealing with machine learning models for our entire careers, and an LLM at the end of the day is just another machine learning model, we have all of the building blocks necessary to understand agentic AI.
[00:26:41] It's just taking these building blocks and translating them into this new context. And once we can translate them into that new context, which is a form of rewiring our thinking, then we're already ahead of people who haven't been thinking this way for the past 10 plus years.
[00:27:00] Jia Huang: Exactly. Yeah, I like your summary. It make a lot of sense, yeah
[00:27:05] Dr Genevieve Hayes: So what's the one mindset shift you would want every data scientist listening to this to walk away with today?
[00:27:13] Jia Huang: I think it's just this one sentence. Nothing we know is obsolete. Everything we know is one layer more valuable. We are not being replaced. We are being placed at the center in the AI era
[00:27:29] Dr Genevieve Hayes: I love that message. For listeners who wanna get in contact with you, Jia, what can they do?
[00:27:34] Jia Huang: The best place to find me is LinkedIn under Jia Huang. I also write a Substack, called Agent Patterns. For more structured framework, my manual book, Designing AI Agents, is in early access now. I think it's quite a structured way , to explain the whole framework.
[00:27:55] And I'd like to mention another thing called ADPS at adpsagents.com. This is a community effort around agent design patterns, collecting real cases and publishing pattern whitepapers. The goal is a shared vocabulary for designing, evaluating, and governing agents.
[00:28:18] So my broader goal is simple: help the field move from demos to agent systems that actually working in production.
[00:28:27] Dr Genevieve Hayes: And I will just say that your Substack is one of the best that I've seen. I really enjoy it when it pops into my inbox, and you have really changed the way I think about a lot of the things we just discussed today
[00:28:40] Jia Huang: Thank you so much, Genevieve. Yeah
[00:28:43] Dr Genevieve Hayes: And that's it for today's episode of Value-Driven Data Science. But if you want more from Jia, next week you can catch our Value Boost episode where we explore why silent correctness is the most dangerous failure mode in agentic AI systems and what you can do to avoid it. And if you found today's episode useful and think others would benefit, please leave us a rating and review on your favorite podcast platform.
[00:29:11] That way, we'll be able to reach more data scientists just like you. Thanks for joining us today, Jia
[00:29:18] Jia Huang: Thank you, Genevieve
[00:29:20] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I'm Dr. Genevieve Hayes, and this has been Value Driven Data Science.