What to do with records your pipeline can never process

A dead-letter queue (DLQ) is a separate queue where a message goes after it has failed a set number of processing attempts, so it stops being retried, the rest of the work keeps moving, and someone can see why it failed and send it back once the cause is fixed.

Quarantine is the same idea for records that aren’t messages: a table or bucket where rejected items wait with the reason they were rejected, instead of being retried forever or thrown away.

Anyone with a queue, a background job runner or a pipeline that processes records one at a time: file uploads, webhook events, emails to sort, documents to summarize. Sooner or later one record can never succeed: a PDF the parser can’t read, an event about a user who was deleted, an input the model mishandles every time.

A record that can never succeed meets a system built to retry. With no limit, it’s retried forever, using worker time and, in an AI pipeline, a paid model call on every attempt. Depending on how the queue orders work, it can also hold up the records behind it.

The opposite reflex, catching the error and dropping the record, is worse: the work quietly disappears and nobody can say what happened to it.

1. Count attempts. Every time a message is received, its receive count goes up. In Amazon SQS the limit is the maxReceiveCount in the queue’s redrive policy: the number of times a consumer can receive a message before SQS moves it to the dead-letter queue.
2. Move it after the limit. When the count reaches the limit, the message goes to the dead-letter queue instead of back to the main one. Set the limit high enough to ride out a short outage: AWS points out that with a value of 1, a single failure sends the message to the DLQ.
3. Keep the reason. Store the last error, the time and the number of attempts with the message. The SQS docs suggest checking your logs for the exceptions behind dead-lettered messages; keeping the reason with the record saves that search.
4. Skip retries for permanent failures. Invalid input fails the same way every time, so send it to the dead-letter queue on the first attempt.

export interface Message<T> {
  id: string
  body: T
  receiveCount: number
}

export interface DeadLetter<T> {
  message: Message<T>
  reason: string // the last error, so nobody has to dig through logs
  deadAt: string
}

export type Outcome = 'processed' | 'retry' | 'dead-lettered' | 'empty'

/** Throw this for failures a retry can never fix, such as invalid input. */
export class PermanentError extends Error {}

export function createQueue<T>(maxReceiveCount: number) {
  const ready: Message<T>[] = []
  const dead: DeadLetter<T>[] = []

  return {
    ready,
    dead,
    send(id: string, body: T) {
      ready.push({ id, body, receiveCount: 0 })
    },

    async processNext(handler: (body: T) => Promise<void>): Promise<Outcome> {
      const message = ready.shift()
      if (!message) return 'empty'
      message.receiveCount++
      try {
        await handler(message.body)
        return 'processed'
      } catch (error) {
        const reason = error instanceof Error ? error.message : String(error)
        const permanent = error instanceof PermanentError
        if (permanent || message.receiveCount >= maxReceiveCount) {
          dead.push({ message, reason, deadAt: new Date().toISOString() })
          return 'dead-lettered'
        }
        ready.push(message) // try again later
        return 'retry'
      }
    },

    /** After fixing the cause, send dead letters back with a fresh receive count. */
    redrive() {
      for (const { message } of dead.splice(0)) {
        ready.push({ ...message, receiveCount: 0 })
      }
    },
  }
}

Once the cause is fixed, redrive: move the messages back to be processed again. SQS has this built in. Redriven messages are treated as new messages, with a new message ID and enqueue time, and SQS can’t filter or change messages while moving them, so fix the code or the data first.

Give the dead-letter queue a longer retention period than the main queue. In SQS standard queues, a message’s expiry is based on when it was first sent, and moving it to the DLQ doesn’t reset that, so a short DLQ retention can delete messages before anyone looks at them.

A worker written quickly for “process each uploaded file” tends to end in one of two ways: a retry with no limit, or a catch that logs the error and moves on. Both look fine in development, where every test file is valid.

AI pipelines also add a new kind of poison message: an input the model handles badly every single time, where each retry pays for the same wrong answer again. When a validation gate rejects model output, the record belongs in quarantine with the reason (see how to validate LLM output before it reaches your database), not in an endless repair loop.

• Retry forever. Never loses a message, but one bad record costs attempts forever.
• Drop on failure. Cheap, and loses data without a trace.
• Amazon SQS. A DLQ set by a redrive policy with maxReceiveCount, in the same account and Region as the source queue, plus a redrive allow policy for which queues may use it and built-in redrive. AWS advises against a DLQ on a FIFO queue when breaking the exact order of messages isn’t acceptable.
• Google Cloud Pub/Sub. Dead-letter topics with a maximum number of delivery attempts between 5 and 100 (5 by default). The count is approximate, because Pub/Sub forwards undeliverable messages on a best-effort basis.
• RabbitMQ. Dead-letter exchanges. A message is dead-lettered when it’s rejected without requeue, expires, overflows a queue length limit or, on quorum queues, passes the delivery limit, and RabbitMQ records why in the message’s x-death header.
• A database table. Status, attempts and last error columns on the jobs table. Fine for small systems; it’s the same pattern without a broker.

1. For each queue or job type, choose a maximum number of attempts that covers a short outage.
2. Separate permanent errors (bad input) from temporary ones (timeouts, 5xx responses), and dead-letter permanent ones straight away.
3. Store the reason, the attempts, the time and the original message with every dead letter.
4. Alert when the dead-letter queue isn’t empty. On AWS, the SQS docs show how to set a CloudWatch alarm for this.
5. After fixing the cause, redrive, slowly at first. AWS recommends starting SQS redrive at a low rate and raising it while you watch the source queue.

What to tell your coding agent. Paste this into Claude Code, Codex or Antigravity:

Find every queue consumer, background job and pipeline step in this project. Give each one a maximum number of attempts. When a message fails that many times, or fails with an error that can never succeed (such as invalid input), move it to a dead-letter queue or table together with the original message, the last error, the number of attempts and the time. Make sure one bad message never blocks the ones behind it. Add an alert when the dead-letter queue isn’t empty and a way to redrive messages after a fix. Add tests for a temporary failure, a permanent failure and a redrive.

To make it a standing rule instead of a one-off request, put the same text in the instructions file your agent reads at the start of a session: CLAUDE.md for Claude Code, AGENTS.md for Codex, and AGENTS.md, GEMINI.md or a file in .agents/rules/ for Antigravity. ArcLab doesn’t connect to any of these tools; you paste the text yourself.

The Dead-Letter Queues & Quarantine Routing module in the Graph Engineering track starts from an example incident, a malformed PDF the parser could never read. It covers bounded retries before quarantine, the context a dead letter should carry, keeping the pipeline moving and routing each failure to the right fix, and ends with a dead-letter and quarantine spec for your own pipeline.

The official documentation this guide was checked against:

• Using dead-letter queues in Amazon SQS, Amazon SQS Developer Guide: maxReceiveCount, redrive allow policy, FIFO caveat, retention
• Configure a dead-letter queue redrive in Amazon SQS, Amazon SQS Developer Guide: redrive creates new messages; no filtering; start slow
• Handle message failures: dead-letter topics, Google Cloud Pub/Sub docs: 5 to 100 delivery attempts; best effort
• Dead Letter Exchanges, RabbitMQ docs: what triggers dead-lettering; x-death header
• How Claude remembers your project, Claude Code docs, Anthropic: CLAUDE.md is read at the start of every session
• AGENTS.md, Codex docs, OpenAI: Codex reads AGENTS.md before doing any work
• Rules, Google Antigravity docs: workspace rules in AGENTS.md, GEMINI.md or .agents/rules/