Why You Can Remember Pythagoras' Theorem, But Not What Your Wife Said Yesterday

Memory Engineering, Part 1 — Building Another You

A² + B² = C².

You learned that when you were twelve. It's been roughly two decades since you've calculated the hypotenuse of anything in real life. And yet, if I asked you to recite Pythagoras' Theorem right now, you could.

Now: what did your wife tell you to pick up from the supermarket on the way home yesterday?

…

If you confidently said "milk," I admire the optimism. If you went quiet and stared at the ceiling for a moment, I think our wives can be good friends.

This is the first post of a series I'm writing about agent engineering — the discipline of building AI systems that can reason, remember, and act on your behalf. The pitch I'm going to make across the whole series is almost embarrassingly simple:

The easiest way to understand how to build an agent is to think about how you'd build another version of yourself.

I’m going to start with what I’m personally weakest at: memory. Specifically, why my brain refuses to let go of Pythagoras but happily loses track of "milk and eggs, please."

A 60-year-old model still does most of the work

In 1968, two psychologists named Richard Atkinson and Richard Shiffrin proposed what became the foundational model of human memory: The human memory is not a single entity, but made of at least two things, working in concert.

There's short-term memory: the stuff you're holding in your head right now, this second. The number you're about to dial. The point you're trying to make at the end of the sentence you're currently in the middle of. The grocery list your wife just sent you out for.

And there's long-term memory: the stuff that's been filed away. Pythagoras. Your mother's birthday. How to ride a bike. The smell of your grandmother's kitchen.

Most things you experience pass through short-term memory and get forgotten. A small fraction gets stashed into long-term memory and sticks around for years. Sometimes decades.

This is, more or less, the same architecture an AI agent has.

Short-term memory: the kitchen table

Imagine your short-term memory as a small kitchen table. You can put things on it, move them around, and work with them. But the table is small. And every time you put something new on it, something else falls off the edge.

Your agent has a kitchen table too. We just call it a context window.

In practice, agents juggle three different kinds of things on that kitchen table. They look different in the code, but they're all the same flavour of memory.

1. The scratchpad

Your wife sends you to the supermarket. You do the responsible-husband thing and write it down: milk, if there are eggs, buy a dozen.

That piece of paper is a scratchpad. It's a temporary working system that exists to help you finish exactly one task (although context switching allows you to have more than one scratchpad running at the same time).

Agents do exactly this. When they're working on a problem, they write things down. Their inner thoughts, intermediate plans, todo list. The scratchpad helps the agent reason without having to hold everything in its head at once. When the task is done, the scratchpad is thrown away.

2. The conversational history

You and your wife have a Telegram channel. There are, conservatively, fourteen thousand messages in it. She sent you a message yesterday afternoon that contained important information about Sunday's plans.

Quick Test: Without referring to your phone, tell me what it is?

You could remember. The information is right there, in the channel. You'd just need to scroll up. But it's not in your head, because the channel is long and your kitchen table is small. So you live with the trade-off: you have a complete record, but only the last few messages (hopefully) are actually residing in your mind.

Agents have the same problem. The "conversation history" is the running transcript of what you and the agent have said to each other. However, these context windows have limits. When the conversation gets too long, older messages have to be:

• dropped

• summarised; or

• stashed somewhere else.

The kitchen table isn't bigger; it just seems bigger because we're getting cleverer about what we put on it.

3. Semantic caching

This one's subtler. You're at work. You pull up an email about the Q3 budget. You read it. A colleague then walks over and asks you a question about the budget.

You don't pull the email up again.

Instead, you reach into your short-term memory and answer from the impression you have. In this case, you are retrieving the context you derived from the document a moment ago.

That's semantic caching.

Likewise, for an agent, instead of re-fetching and re-parsing the same document seven times in a single conversation, the agent stashes a compressed, queryable representation of what it learned. The next time someone asks about the doc, the agent answers from that cached understanding. Faster, cheaper, but occasionally wrong because the impression has drifted from the source.

That's short-term memory, in three flavours. Now for the part where you stop being a goldfish.

Long-term memory: where you live

In 1972, Canadian psychologist Endel Tulving, looked at long-term memory and said: this isn't one thing either.

Tulving carved long-term memory into three buckets.

Episodic memory: things that happened to you

Episodic memories are experiences. They have a time stamp, even if a fuzzy one. Your first kiss. The morning your daughter was born. That Valentine’s last year that ended at the food court, because you forgot to book a restaurant.

Episodic memory is what makes you feel like a continuous person. You are a story, told to yourself in a loose chronological order, with you as the protagonist (or antagonist).

For an agent, episodic memory is the log of interactions:

• The meeting it scheduled for last Tuesday;

• The email it drafted on your behalf last month;

• The customer it helped two weeks ago.

Each entry is a little time-stamped story: on this date, in this conversation, this is what I did and what happened next. Without episodic memory, an agent has no continuity.

Semantic memory: things that are true

Semantic memory is facts. You don't remember when or where you learned that Paris is the capital of France, but you do remember it.

Here's the interesting bit: some facts decay. Or rather, the truth they describe decays.

Your wife's nails were red last July. Then she had a manicure. Then they were brown. Then, for some reason, red again, and then in October, she went through that nail-art phase no one talks about anymore.

The fact "my wife's nails are red" was true on July 14th. It was untrue on August 3rd. It was true again on September 2nd. It is now mid-November, and you do not, in fact, know what colour her nails are at this exact moment.

Semantic memory in agents needs the same property: facts deserve a "valid from" and "valid to". A naive agent that learns "the user's nails are red" and stores that fact forever could be wrong roughly six weeks later. A smart one stores something closer to:

user’s nail colour is red
valid from: 2025-07-14
valid to: 2025-08-01

More specifically, we refer to these as Temporal Semantic Memory or Temporal Hierarchical Memory. More of that in the future!

Procedural memory: things you know how to do

To a human, procedural memory is your skills. How to ride a bike. How to drive. How to navigate your local supermarket without consulting the floor plan. How to cook the four meals you've been on rotation for the last year.

You can’t easily explain procedural memory. You can't write down "how to ride a bike" in a way that would let someone who's never ridden one actually do it. You can demonstrate, hint, correct, but the skill itself lives somewhere wordless.

For an agent, procedural memory is workflows. The repeatable sequence of steps that gets a particular outcome. To onboard a new customer, do A, then B, then check C, then if D, escalate. These are the recipes the agent reaches for when it recognises a familiar situation.

Procedural memory is what turns a competent agent into a fluent one. The agent that knows the steps doesn't have to figure them out from first principles every single time.

So why does Pythagoras stick and the grocery list doesn't?

Because Pythagoras lives in your semantic long-term memory. He's been rehearsed thousands of times with every triangle, every right-angle in your geometry homework.

The grocery list, on the other hand, lived briefly on your kitchen table and was never promoted. Why would it be? Tomorrow's list will be different. Storing it long-term would be a waste of biological storage. So your brain, ever the efficient little organ, lets it slip off the edge.

Agents face the same decision a million times a day: should I remember this, and for how long? Get that decision wrong in one direction, and the agent is a goldfish. Get it wrong in the other, and the agent is a stalker who knows what your nails looked like two summers ago.

There's an art to it. There's also, increasingly, an engineering discipline to it. That's what this series is about.

What's next

We've now mapped the territory: short-term memory in three flavours (scratchpad, conversational history, semantic cache) and long-term memory in three more (episodic, semantic, procedural). All informally called "memory" by the engineers building today's agents.

Over the next few posts, we're going to walk through each of these and look at how they're actually implemented. How does an agent decide what to keep on the kitchen table and what to file away? How does it find a memory when it needs one? How does it forget?

Next post: The Elephant vs the Goldfish. We zoom in on short-term memory — and the surprising fact that most agents today are goldfish, and that's often the right design call. Follow for more