Durable Agents: Surviving Restarts, Crashes, and a Week of Idle

The agent loop in every tutorial holds the whole conversation in memory, runs to completion, and exits. It is a beautiful thing for about thirty seconds. Then someone deploys it, the process gets recycled at 2am during a routine restart, and the agent that was three steps into a seven-step collections workflow simply ceases to exist. No error. No half-finished state. Nothing. The task evaporated.

That gap, between an agent that runs and an agent that survives, is where most of the real engineering lives. A demo agent is a function call. A durable agent is closer to a database transaction that happens to think.

What “durable” actually has to mean

When I say an agent is durable, I mean it passes three tests that the in-memory loop fails:

• It survives a restart mid-task. The box reboots, the container redeploys, the process OOMs. When it comes back, the agent resumes from the last completed step, not from zero, and not from a corrupted middle.
• It survives a crash inside a step. A tool call to a payments API times out. The service the agent was hitting returns a 500. The agent does not lose the work it already did and does not redo the parts that already succeeded.
• It survives a week of idle. A task says “follow up in seven days.” The agent has to actually wake up in seven days and remember, in full, what it was doing and why. Holding a Python coroutine alive for a week is not a plan.

None of those are AI problems. They are workflow-durability problems that the LLM industry rediscovered the hard way, the same problems job-queue and saga people solved years ago. The agent crowd keeps reinventing them badly because the demo never exercises them.

State has to live outside the process

The core move is simple to say and annoying to do: the agent’s state cannot live in the process running the agent. It has to live in durable storage, and the process has to be a disposable worker that picks up state, advances it by one step, writes it back, and is free to die at any instant.

I model this as an explicit run record. Every agent task in the AI-native ERP I’m building (Vontra) is a row, not a coroutine:

AgentRun
  id
  goal               # what it is trying to do
  status             # pending | running | waiting | done | failed
  cursor             # which step it is on
  scratchpad         # accumulated reasoning + tool results
  wake_at            # null, or a timestamp to resume at
  envelope_id        # what it is allowed to do
  updated_at

The worker does one thing in a loop: claim a run that is pending or whose wake_at has passed, load its full state, execute exactly one step, persist the new state, release it. If the worker dies between claiming and persisting, a lease timeout returns the run to the pool and another worker picks it up. The agent does not know or care which physical process is carrying it from one step to the next.

This is the durable-step pattern, and it is the single most important architectural decision in any agent meant to run unattended. Everything else is detail.

Steps must be idempotent, because they will run twice

Here is the part people get wrong even after they externalize state. If a worker executes a step, performs a side effect, and then crashes before it writes the new cursor, the next worker will re-run that step. So every step that touches the outside world will, eventually, run more than once. That is not a rare edge case. At any real volume it is a Tuesday.

So every side-effecting step needs an idempotency key. When the agent records a payment or sends an email, the key is derived from the run id and the step index, and the downstream system is asked to dedupe on it.

def record_payment(run_id, step, invoice, amount):
    key = f"{run_id}:{step}:record_payment"
    return ledger.record(
        invoice=invoice, amount=amount,
        idempotency_key=key,   # second run is a no-op, not a double charge
    )

Without that key, durability becomes a liability. An agent that reliably resumes is also an agent that reliably double-charges, double-emails, and double-books. The first durable agent I shipped did exactly this: a retry after a transient failure sent the same overdue-invoice reminder twice to the same customer. Not catastrophic, but it taught me that resume and idempotency are the same feature wearing two hats.

The scratchpad is the memory, and it is expensive

The agent’s accumulated context, its reasoning and tool results, is the scratchpad. On a long task it grows, and every step re-feeds it to the model. A seven-step task where the scratchpad balloons to 40K tokens means you are paying to re-read 40K tokens on every single step. That adds up fast, both in latency and in dollars.

Durable agents therefore need a memory discipline, not just memory. A few things that earned their keep:

• Summarize completed phases. Once a sub-goal is done, collapse its raw tool spew into a short factual summary and drop the raw transcript from the active context. The agent rarely needs the 600-line API response again; it needs the two facts it extracted.
• Keep durable facts separate from working context. Things the agent learned that matter beyond this step (an invoice id, a confirmed amount) get written to structured fields, not left to drift in prose where a later summarization might lose them.
• Cap and checkpoint. If the scratchpad crosses a threshold, force a summarization step before continuing. A task that never compresses is a task whose token cost grows without bound.

Waking up after a week

The idle case is where the externalized model pays off most. An agent that needs to act in seven days does not stay running. It sets wake_at, sets status to waiting, and persists. The worker pool ignores it entirely until that timestamp passes. No process is held open. No memory is consumed. The “agent” between now and then is just a row in a table with a future timestamp.

This is also where event-driven and scheduled triggers meet durability. A waiting run can be woken by the clock, or it can be woken early by an event: the customer paid, so the planned follow-up is now moot and the run should resume immediately to cancel itself. That coupling, durable state plus event wakeups, is what lets an agent both wait patiently and react instantly, which sounds contradictory until the state lives in a place that both the clock and the event bus can poke.

An idle durable agent should cost you exactly one database row and zero running processes. If it costs you a live coroutine, you have built a demo with a long timeout.

The unglamorous conclusion

Nobody puts durability in the demo because it does not show. The agent that resumes cleanly after a crash looks identical to the agent that never crashed. The work is invisible right up until the night a deploy goes out mid-task and one system loses 200 in-flight workflows while the other one shrugs, reboots, and continues from the last committed step.

I have come to treat durability as the dividing line between a toy and a tool. Anyone can write the loop that runs once. The agents I am willing to leave unattended overnight are the ones where I genuinely do not care if the process dies, because the process was never where the agent lived. The agent lives in the storage. The process is just the muscle that moves it forward one step at a time, and muscle is cheap and replaceable. That is what lets the thing wake up tomorrow and remember exactly where it was.