How to recover a multi-step AI pipeline after a crash

A checkpoint is a saved record of which steps of a pipeline have finished and the data they produced, written after each step succeeds, so that after a crash the pipeline resumes from the last good step instead of starting over.

Anyone with a multi-step job where the steps are slow or cost money: document pipelines that parse, summarize and classify; agents that plan, research and write; data imports with many stages; anything that calls a model several times per item.

Long jobs fail partway through. A deploy restarts the worker, a provider times out, the machine runs out of memory. Without saved progress, the only way to recover is to run everything again.

That repeats every model call you already paid for, can give different answers the second time, and repeats side effects such as emails or database writes from steps that had already finished. The longer the pipeline, the more a late failure costs.

1. Give each run an ID that stays the same across restarts.
2. After each step succeeds, save a checkpoint: the names of finished steps and the data later steps need, as plain JSON, in durable storage.
3. On start, load the checkpoint for that run ID and skip the steps that already finished.
4. Make each step safe to run twice. If the process dies after a step did its work but before its checkpoint was saved, that step runs again on resume. Give it an idempotency key made from the run ID and the step name, and pass the key to any API that changes something.

export interface Checkpoint<C> {
  completed: string[] // names of steps that finished, in order
  context: C // everything later steps need, as plain JSON
}

export interface CheckpointStore<C> {
  load(runId: string): Promise<Checkpoint<C> | undefined>
  save(runId: string, checkpoint: Checkpoint<C>): Promise<void>
}

export interface Step<C> {
  name: string
  /**
   * May run more than once: a crash after the work but before the save
   * repeats it. Pass `idempotencyKey` to any API that changes something.
   */
  run(context: C, idempotencyKey: string): Promise<C>
}

/** Run the steps in order, skipping any that a previous attempt finished. */
export async function runPipeline<C>(
  runId: string,
  steps: Step<C>[],
  input: C,
  store: CheckpointStore<C>,
): Promise<C> {
  const saved = await store.load(runId)
  let context = saved?.context ?? input
  const completed = saved?.completed ?? []

  for (const step of steps) {
    if (completed.includes(step.name)) continue // finished before the crash
    context = await step.run(context, `${runId}:${step.name}`)
    completed.push(step.name)
    await store.save(runId, { completed: [...completed], context })
  }
  return context
}

The last point is the one people skip. Temporal, a workflow engine built around this idea, says the same about its steps: an activity may be retried, so it may run more than once, and Temporal recommends making activities idempotent. The key it suggests, the workflow run ID plus the activity ID, is the same idea as `${runId}:${step.name}` in the example.

Keep checkpoints small. Store results or references to them (a file path, a row ID), not whole documents, and keep them in a database or object store, never only in the process’s memory, which is the thing that just crashed.

Ask for “a pipeline that summarizes, tags and publishes each document” and the natural result is three calls in a row inside one try block. It works until the first crash in the middle, and then reruns everything. Recovery is behavior you only see when something fails, so it has to be asked for and tested on purpose: stop the run after step two and check that step one doesn’t run again.

• Rerun everything. The simplest option, and fine when the pipeline is short, cheap and has no side effects.
• A workflow engine. Temporal keeps an event history for each workflow and replays it after a failure, so the workflow picks up where the last recorded event left off. AWS Step Functions can redrive a failed Standard Workflow execution from the step that failed, without rerunning the steps that succeeded, for up to 14 days. You get durable state and retries without writing them; you still need idempotent steps.
• A framework checkpointer. LangGraph saves a snapshot of graph state at each step, and if one node fails, it keeps the writes of the nodes that finished in that step so they aren’t rerun on resume. Its MemorySaver keeps checkpoints in RAM, so they’re lost when the process restarts; use a database-backed checkpointer such as PostgresSaver in production.
• One queue job per step. Each step becomes its own job, so a crash only loses the step in progress. You then need a plan for steps that never succeed (see what to do with records your pipeline can never process).

1. Give every run an ID and store it with the run’s input.
2. Add a checkpoint table: run ID, finished steps, context as JSON, time of the last update.
3. Run the steps through one function like runPipeline.
4. Give every step with side effects an idempotency key built from the run ID and the step name.
5. Test recovery: throw in the middle step, run again, and assert the earlier steps ran once.

What to tell your coding agent. Paste this into Claude Code, Codex or Antigravity:

Find every multi-step job or agent pipeline in this project. Give each run an ID that survives restarts. After each step succeeds, save a checkpoint to the database with the run ID, the names of the finished steps and the data later steps need (plain JSON, or references to stored files). On start, load the checkpoint and skip finished steps. Give every step that changes something an idempotency key made from the run ID and the step name, and pass it to the APIs it calls. Add a test that crashes the run in the middle, resumes it and asserts the earlier steps ran only once.

To make it a standing rule instead of a one-off request, put the same text in the instructions file your agent reads at the start of a session: CLAUDE.md for Claude Code, AGENTS.md for Codex, and AGENTS.md, GEMINI.md or a file in .agents/rules/ for Antigravity. ArcLab doesn’t connect to any of these tools; you paste the text yourself.

The Checkpoint Snapshots & Recovery Boundaries module in the Graph Engineering track starts from an example incident, stage 12 of a 14-stage ingestion pipeline timing out during a three-hour run. It covers where to cut checkpoints in a graph, immutable step outputs and rollback boundaries, and ends with a checkpoint and recovery spec for your own pipeline.

The official documentation this guide was checked against:

• Persistence, LangGraph docs, LangChain: MemorySaver is lost on restart; PostgresSaver
• Checkpointers, LangGraph docs, LangChain: a snapshot per super-step; pending writes
• Workflow Execution, Temporal docs: event history and replay
• Activity Definition: idempotency, Temporal docs: activities may run more than once; run ID plus activity ID as a key
• Restarting state machine executions with redrive, AWS Step Functions Developer Guide: resume from the failed step; 14-day window
• How Claude remembers your project, Claude Code docs, Anthropic: CLAUDE.md is read at the start of every session
• AGENTS.md, Codex docs, OpenAI: Codex reads AGENTS.md before doing any work
• Rules, Google Antigravity docs: workspace rules in AGENTS.md, GEMINI.md or .agents/rules/