Designing Dead-Letter Queues for Document Pipelines

When a change order fails validation or a submittal parse throws, the pipeline has two honest choices: park the document somewhere it can be recovered, or lose it. A dead-letter queue is the parking, and its design decides whether “recover it” is a real operation or a hopeful one. This page solves one precise problem: how to design a dead-letter queue that retains the original payload, the validation output, and the exception context together, supports replay after the underlying fix, and makes that replay idempotent by keying on a document hash — so a corrected document reprocesses exactly once and a fixed batch never double-posts. The dead-letter queue is the backstop of the error handling protocols that guard automated document ingestion: every quarantined record, every exhausted retry, and every sub-threshold match lands here rather than vanishing. It targets Python automation builders who need a recovery path with an audit trail, not a graveyard of unparseable files.

Key Rules and Specification

A dead-letter queue is only useful if what it stores is enough to fix the problem and replay safely. The rules below separate a recoverable envelope from a graveyard entry.

• Retain the original payload, unmodified. The raw bytes as received — the PDF, the spreadsheet row, the API JSON — are stored immutably. Replay reprocesses the original, not a partially-transformed intermediate, so a fix to the parser is genuinely re-exercised against the real input.
• Retain the validation output beside it. The envelope records which rule failed and what the validator produced, not just that something failed. An operator fixing a schema or a mapping needs to see the specific rejection, and an auditor needs the decision that parked the record.
• Retain the exception context. The exception type, message, and traceback are captured so a transient fault (a timeout) is distinguishable from a permanent one (a malformed document). The disposition — retry versus park-for-fix — depends on that distinction.
• Key everything on a document hash. A stable content hash (SHA-256 of the original payload) is the envelope’s identity and the idempotency key. Replay commits through that key, so a corrected document reprocesses exactly once even if the same batch is replayed twice.
• Replay after fix, do not blind-retry. Automatic retry with backoff handles transient faults upstream; the dead-letter queue is for records that need a change — a new mapping, a schema correction, a locale rule — before they can succeed. Replay is a deliberate operation, taken after the fix lands.
• Escalation lives elsewhere. The queue parks and preserves; deciding who gets paged and when an SLA is breached is handed to fallback alert routing, so escalation policy stays in one place. Records that arrive here below the canonical 0.75 confidence band are the routing quarantine, distinct from an exception.

Production Code Example

The envelope is a typed Pydantic contract so a malformed dead-letter entry is itself a caught error rather than a second silent failure. The document hash is computed from the original bytes and is the record’s identity. Structured diagnostics are written per the Python Logging HOWTO so every park and replay is traceable.

from __future__ import annotations

import hashlib
import logging
import traceback
from datetime import datetime, timezone
from decimal import Decimal
from typing import Literal

from pydantic import BaseModel, ConfigDict, Field

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)-8s | %(message)s")
logger = logging.getLogger("dlq")

DeadLetterReason = Literal["validation", "exception", "low_confidence"]

QUARANTINE_FLOOR = Decimal("0.75")  # site-canonical: below this a record dead-letters


def document_hash(payload: bytes) -> str:
    """Stable content identity; the idempotency key for replay."""
    return hashlib.sha256(payload).hexdigest()


class DeadLetter(BaseModel):
    """Everything needed to diagnose, fix, and safely replay one failed document."""
    model_config = ConfigDict(frozen=True)

    doc_hash: str = Field(min_length=64, max_length=64)
    reason: DeadLetterReason
    payload: bytes                                   # original, immutable
    validation_output: str = Field(default="")       # which rule failed
    exception_context: str = Field(default="")        # type · message · traceback
    attempts: int = Field(default=1, ge=1)
    first_seen: datetime
    confidence: Decimal | None = Field(default=None, ge=0, le=1)

    def escalation_payload(self) -> dict[str, str]:
        # Handed to the fallback alert router; carries identity, not the raw bytes.
        return {"doc_hash": self.doc_hash, "reason": self.reason,
                "attempts": str(self.attempts), "first_seen": self.first_seen.isoformat()}


def to_dead_letter(
    payload: bytes,
    reason: DeadLetterReason,
    *,
    validation_output: str = "",
    exc: BaseException | None = None,
    confidence: Decimal | None = None,
) -> DeadLetter:
    """Build an envelope from a failure, capturing full exception context."""
    ctx = ""
    if exc is not None:
        ctx = "".join(traceback.format_exception(type(exc), exc, exc.__traceback__))
    dl = DeadLetter(
        doc_hash=document_hash(payload),
        reason=reason,
        payload=payload,
        validation_output=validation_output,
        exception_context=ctx,
        first_seen=datetime.now(timezone.utc),
        confidence=confidence,
    )
    logger.warning("DEAD-LETTER | hash=%s reason=%s | %s",
                   dl.doc_hash[:12], reason, validation_output or (exc and str(exc)) or "")
    return dl

Replay is idempotent by construction: the guard checks a durable committed-set keyed on the document hash, so replaying a corrected batch reprocesses each document at most once. The process callback is the same pipeline entry point that originally failed; after the fix lands it now succeeds, and its result commits under the hash.

from typing import Callable

# Stand-ins for durable stores (a DB unique index / Redis set in production).
_COMMITTED: set[str] = set()
_DEAD_LETTERS: dict[str, DeadLetter] = {}


def park(dl: DeadLetter) -> None:
    """Store one envelope per document hash; a re-park just bumps the attempt count."""
    existing = _DEAD_LETTERS.get(dl.doc_hash)
    if existing is not None:
        dl = dl.model_copy(update={"attempts": existing.attempts + 1,
                                   "first_seen": existing.first_seen})
    _DEAD_LETTERS[dl.doc_hash] = dl


def replay(doc_hash: str, process: Callable[[bytes], str]) -> str:
    """Replay one parked document after the underlying fix. Idempotent on doc_hash."""
    dl = _DEAD_LETTERS.get(doc_hash)
    if dl is None:
        return "not_found"
    if dl.doc_hash in _COMMITTED:            # already succeeded on an earlier replay
        logger.info("replay skip (committed) %s", doc_hash[:12])
        return "skipped"
    try:
        outcome = process(dl.payload)         # re-exercise the corrected pipeline
    except Exception as exc:                   # still broken -> re-park with fresh context
        park(to_dead_letter(dl.payload, dl.reason, exc=exc))
        return "still_failing"
    _COMMITTED.add(dl.doc_hash)                # commit under the idempotency key
    del _DEAD_LETTERS[dl.doc_hash]
    logger.info("replay ok %s -> %s", doc_hash[:12], outcome)
    return outcome


if __name__ == "__main__":
    bad = b'{"change_order": "CO-014", "amount": "1.234,56"}'  # locale bug, will be fixed
    dl = to_dead_letter(bad, "validation", validation_output="amount not canonical Decimal")
    park(dl)
    # After deploying the European-decimal fix, replay is safe to run repeatedly:
    print(replay(dl.doc_hash, lambda p: "auto_route"))  # -> auto_route
    print(replay(dl.doc_hash, lambda p: "auto_route"))  # -> skipped (idempotent)

Common Mistakes and Gotchas

Storing the error but not the payload. A dead-letter queue that logs the exception and drops the original document is a graveyard: you know something failed, but you cannot replay it because the input is gone. Retain the raw bytes immutably. Replay must reprocess the original through the fixed pipeline, and that is impossible if only a stack trace survived.

Blind-retrying what needs a fix. Retrying a document that fails because of a bad mapping or an unhandled locale just burns attempts and, with acks_late, can loop forever. Separate transient faults — handled by retry logic with backoff upstream — from permanent ones that belong in the dead-letter queue awaiting a code or configuration change. The exception_context is what lets an operator tell them apart.

Replaying without an idempotency key. Re-running a fixed batch of change orders without a commit guard double-posts every one that had actually succeeded on a partial earlier replay. Key the commit on the document hash and check a durable committed-set before reprocessing, so replaying the same batch twice is safe. Idempotency is not a nicety here; it is the difference between a recovery tool and a way to inflate committed cost.

Where This Fits in the Pipeline

The dead-letter queue is the backstop of the error handling protocols that wrap every ingestion stage. It receives three kinds of record: exceptions thrown during parsing, failures from the schema validation rules at the boundary, and low-confidence results below the canonical 0.75 band that were quarantined rather than committed. During a bid-period spike, the durable-queue model behind async batching workflows feeds its exhausted tasks here, and the queue’s own escalation is delegated to fallback alert routing so the on-call estimator is paged from one place. Because the confidence bands and the idempotency key are the same ones used everywhere on the site, a document parked here and replayed after a fix rejoins the pipeline exactly as if it had never failed — once, and with its full audit trail intact.

Frequently Asked Questions

Related

• Error Handling Protocols
• Async Batching Workflows
• Fallback Alert Routing
• Schema Validation Rules
• PDF/Excel Sync Pipelines

← Back to Error Handling Protocols