Value Driven Data Science

[00:00:00] Dr Genevieve Hayes: Hello, and welcome back to Value-Driven Data Science, where data professionals become strategic experts. I'm Dr. Genevieve Hayes, and I'm here again with Dr. Imran Ahmad, author of 30 Agents Every AI Engineer Must Build and 50 Algorithms Every Programmer Should Know. Last week, Imran and I discussed the evolution of the data science toolkit from algorithms to AI agents.
[00:00:30] Today, in this Value Boost episode, we'll explore what memory actually means in AI agents and why getting it wrong could prove disastrous for those building them. Welcome back, Imran.
[00:00:43] Dr Imran Ahmad: Oh, thank you very much. Nice to be here
[00:00:46] Dr Genevieve Hayes: One of the most frustrating things about working with AI is when you get to the point where it seems like Claude or ChatGPT or your AI tool of choice really understands you and the way you work, and then all of a sudden it doesn't. It's like training a new graduate to be useful at their job, only to be told they've moved to another role and you have to start again training someone else.
[00:01:12] This is annoying enough when dealing with AI chatbots, but can be catastrophic when building AI systems. So this is something data scientists really need to understand in order to thrive in the AI era. Imran, this is a topic you've written about in your book. Why is it that LLMs can seemingly remember things one day and then suddenly forget about them the next?
[00:01:33] Dr Imran Ahmad: Yes. Yeah first of all, as in life, there is a text and there is a context, and context changes everything. Now the thing is that when we talk about the context basically it gives the perspective to your AI solution. And let's talk about memory, and let's be tool agnostic, and let's think from an algorithmic point of view.
[00:01:54] Now when you're working with an AI solution, you need more than one type of memories. . First of all, a memory that is always there is the working memory, acts like a cache.
[00:02:04] It's the active session where it stores whatever was asked in the previous turn or a turn before that. That's the simplest memory. That is called the working memory. Working memory is nothing new. In software systems, we always build those cache and that speed up things that removes the for-forgetfulness of our software, you know. so this is one thing. But in addition to that, we have two more types of memories that are optional but are absolutely necessary if you want to build efficient systems. And let me expand on this, by a simple example, So let's assume it's not basically an assumption, that's a fact, that my son, he really love eating eggs. He's doing some bodybuilding and he wants enough protein, so he eats lots of eggs. And for the last two years my son actually is eating three, four eggs. And if you want to collect the data, so we can say that January thirteenth, he ate four eggs, and they were bought from Walmart, and this was the price.
[00:03:09] They were brown, and he fried them and ate them. And then, June seventh, he did that, June eighth, and so on. So last two years of the data in each and every detail that is stored. That is called episodic memory. There's an episode where each of the event and the data point in its fi-finest granularity is stored there with timestamp.
[00:03:32] But Perhaps one sentence is enough that can summarize all this, that my son loves eggs. That's it. So it means that all those two years of data, one sentence can pick the essence of that, the meaning of that, and can summarize it for the model that my son love eggs.
[00:03:52] We don't need the, all , that data because that is not relevant, So this semantic memory needs to be built constantly. It means that episodic memory needs to store everything, and from that, what sort of meaning is being extracted, perhaps one of the meanings could be that eggs have got expensive, and that's a fact here.
[00:04:13] The last couple of years, in fact, there was some political backlash and all those things that were the news due to eggs last year, and especially in US. So that can be something that has come from that episodic memory or Walmart is our favorite store. But all of these are the meanings that have been extracted from the episodic memory, now, this simple example can be applied to different use cases. It means that first of all, there's working memory, no intelligence, just simple cache, the second one is episodic memory. We are storing everything in its detail, and it needs to be persisted or it need to be flushed out. It depends on the problem that we're trying to solve.
[00:04:51] But the actual thing, the differentiating thing, the intelligent thing is something that is extracting meaning from that and then passing it to a genetic system, and those meanings keep on changing. So if you have these three types of memories, working memory that is in the cache, episodic memory is usually vectorized, but semantic memory almost always vectorized and exist in the semantic conceptual neighborhoods where the meanings can be extracted and they are available in a semi-digested form. Agent can just pick this up and consume it
[00:05:24] Dr Genevieve Hayes: So what does this mean for data scientists who are building AI systems? For example, do they need to include all three memory types, or are there situations where you might want to only include one or two?
[00:05:37] Dr Imran Ahmad: Okay first of all, let's talk about the context so context is a memory as well, so , where the context fits in whole thing now, context is something that is attached in one turn, so with each turn, you send something that is the context,
[00:05:52] memory has essentially, a longer lifetime it is multi-turn, multi-days, So this is the difference between a context and a memory. So there are four things: context, then there is working memory then you have episodic memory, and then you have semantic memory,
[00:06:08] so in each AI system, you always has a context, it's going nowhere. And if you don't use working memory, your cost will be higher. Claude, in fact, will suggest you that why don't you turn your cache on? It will reduce the cost, so it will suggest you turn that on.
[00:06:22] So that is an easy thing to do. So it means that now you have a context, and now you have a working memory as well. Now, third thing is that how you're going to build the next two, we talked about what needs to be done, but we did not talk about how it needs to be done.
[00:06:36] Let's tackle that part as well. Now, for episodic memory, it means that whatever is going on, it needs to be recorded somewhere. And the user should not be able to delete that. And usually it's a vector DB where you actually record all this,
[00:06:51] and if you look into the semantic memory, usually there is a background process. There should be a LLM that is working and that extract the meanings and keeps updating the semantic memory. It can be a process that runs once an hour or once a day. It looks into what is there or it depends on.
[00:07:10] AI engineer needs to look into these different parts. Is it a cloud-based solution? What is the architecture? What type of database will be able to do the work that you're doing? And it also depends on the use case and the type of the data that person is dealing.
[00:07:24] Is it financial data? Is it all text data? Is this something sentiment analysis or what type of problem and they are trying to solve?
[00:07:33] Dr Genevieve Hayes: Okay so basically you need to have your context and your working memory, but with the episodic and semantic you can get away without them, but you'll get better performance if you include them as well.
[00:07:46] Dr Imran Ahmad: Yeah, that is the bottom line. Yes
[00:07:48] Dr Genevieve Hayes: What's the first decision a data scientist should make about memory whenever they start designing an agentic AI system?
[00:07:55] Dr Imran Ahmad: So what are the requirements? Are there any regulatory requirements for it ? Number one. The second thing is that if you build no memory, your system will be expensive, it will be less performant, it will be less accurate as well.
[00:08:09] It will be expensive, and also it will be slow. So it reminds me of the story of the invention of a shoe, horseshoe was invented.
[00:08:16] So there was a king, and wherever the king will go, they will lay out lavish carpet for that king so that his feet will be protected. He will go to streets, and the lavish red carpets will be there. So one day someone suggested that why don't you cut a piece of carpet and attach it to your foot, and wherever you will go, you will have the carpet. And he liked it that much, and that, as the story tells, was the invention of the shoe, . Now, what I'm saying here is that you can live with the context as well.
[00:08:46] Each time what you're doing is that you are providing all the context needed to solve this problem. Essentially what you're doing is that, yes, you have one million token window in many cases, but you are actually laying out the carpet for the king everywhere, so these memories, when you build this, actually provide the pointed solution. That is the shoe for the king. Because there's a concept of context rotting. There's a concept of attention. LLM doesn't pay attention to all the context.
[00:09:15] The studies say, when you provide context, LLM only pays attention to the first part, and then it start losing attention. The later part, it just skim through as uh, context become larger. When you provide these memories, you can be sure that the portions that needs to be combined to get the right answer, they are actually utilized,
[00:09:37] and you do not provide unnecessary information. It keeps the cost down and just fundamentally change the accuracy of your whole AI solution, so this is something that you can build, as we said, level one architecture in which there is no memory. Then you live level two architectures where you have one type of memory.
[00:09:55] Then the sophistication level becomes level three, where you have all these different types of memories working together to create something that is much more performant and lower cost as well.
[00:10:06] Dr Genevieve Hayes: So it sounds like the big decision is what are the most important things to tell your LLM So that the LLM can do its work efficiently and cheaply?
[00:10:16] Dr Imran Ahmad: That is correct. And I will give you an example. I was just preparing for an exam to get myself certified as an anthropic instructor. And one of the questions that was there, you will be surprised that it's quite relevant to what we are saying here, that if you have a long context, very long context, where you should put the actual instruction, in the beginning or at the end?
[00:10:39] So you have two things: context and instructions, and the context is huge. There is no memory, you just want to pass it on. So what is the optimal place to put the instruction, at the beginning or at the end? And the right answer is at the end. And there's a reason for it.
[00:10:53] The reason is that as context becomes large, LLM starts looking at in detail, and it gets almost bored. It's a machine it d-doesn't get bored but I can give it a persona. So it starts from the beginning, and it gets bored, and starts skimming through it, but if you put instructions at the end, it cannot afford to get bored because instructions have not come. It needs to look into why this context is there. So it needs to pay attention till it meets that end. And there's a art and science. You try this out and you will be surprised how much difference , does it make, and you can verify that using Claude documentation,
[00:11:29] Dr Genevieve Hayes: Yeah I was thinking I'm gonna try this out as soon as this episode's over, but anyway, keep going
[00:11:34] Dr Imran Ahmad: So this gives you an idea that when you give a large context, you are not in control of how it is processed. This is what I'm trying to say. You give the context and it's your best effort. You just keep praying that, okay, the LLM will read all of that. That's not going to happen. But when you give these memories, then there's a guarantee that LLM, because you are only pointing to the portion that is relevant, LLM will not get bored.
[00:12:02] Dr Genevieve Hayes: That's it for today's conversation with Imran. If you haven't already, listen to our previous episode where we discussed how AI agents are fundamentally changing the data science toolkit and what it means for how data scientists solve real business problems. Thanks for joining me again, Imran.
[00:12:22] Dr Imran Ahmad: It was really nice talking to you. Thank you very much.
[00:12:24] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I'm Dr. Genevieve Hayes, and this has been Value-Driven Data Science.