Celery vs asyncio for Construction Document Bursts

Construction document volume is not steady — it spikes. A bid period, an addendum drop, or a milestone submission lands hundreds of RFIs, submittals, and change orders in an hour, and the concurrency model that carries that burst decides whether the pipeline stays durable or quietly drops work. This page answers one precise question: for a bid-period document burst, when does Celery’s distributed task queue earn its operational weight, and when is a single-process asyncio bounded-gather the right tool instead? Choosing wrong is expensive in opposite directions — Celery over a trivial workload buys you a broker to babysit, while asyncio over a durable workload loses in-flight documents on the first crash. Both sit at the same point in the pipeline, behind automated document ingestion and inside the async batching workflows that absorb the spike, and both must hand quarantined work to the same downstream. This page is for Python automation builders who have to size the concurrency model to a real burst rather than a benchmark.

Key Rules and Specification

The two models solve the same throughput problem with different guarantees. The rules below separate the properties that matter for construction document work — where a lost change order is a lost cost event — from the ones that only matter at a benchmark.

• Durability is Celery’s defining property. A Celery task is persisted in the broker before a worker touches it, so a worker crash mid-document redelivers the task rather than dropping it. asyncio holds the entire burst in one process’s memory; if that process dies, every in-flight and queued coroutine is gone.
• Retries and dead-lettering are first-class in Celery. acks_late with a bounded max_retries and exponential backoff gives automatic redelivery, and exhausted tasks park in a dead-letter queue for replay. asyncio has no built-in retry or dead-letter concept — you build both by hand, and they live only as long as the process does.
• Back-pressure differs in mechanism. asyncio bounds concurrency with a Semaphore(N) inside a single gather, so the loop never opens more than N connections at once. Celery bounds it with worker concurrency and broker prefetch across processes and hosts, which scales past what one event loop can drive.
• Throughput ceilings are set by the workload shape. For I/O-bound work — network document pulls, API calls — one asyncio loop saturates a host cheaply. For CPU-bound parsing or work that must span multiple machines, Celery’s process pool is the only one that scales horizontally.
• Operational cost is real and asymmetric. Celery adds a broker, worker supervision, and monitoring to run and page on. asyncio adds nothing beyond the Python process, which is its main advantage for a modest, self-contained burst.
• Both must be idempotent and Decimal-safe. Whichever model runs, a redelivered or retried task must produce the identical result, so every task is keyed on a document identity and every money field in the payload is a Decimal, never a float. Quarantine decisions use the canonical bands: 0.92 and above auto-routes, 0.75–0.92 reviews, below 0.75 dead-letters.

Decision guide

Which to choose: let durability requirements decide, not raw speed. If losing an in-flight document is unacceptable — the usual case for change orders and submittals that carry cost — choose Celery, because the broker persists the work and the dead-letter queue guarantees nothing vanishes on a crash. If the burst is I/O-bound (pulling documents from a portal API), fits on one host, and any lost work can simply be re-fetched, choose asyncio for its near-zero operational cost. A common and correct hybrid is asyncio inside a Celery task: Celery owns durability and retries across the burst, while a bounded-gather drives the I/O-heavy fan-out within each task. Reach for that when the fetch is I/O-bound but the outcome must not be lost.

Production Code Example

Both sketches process the same unit of work — a document reference resolved to a typed, Decimal-safe result — and both are idempotent: the task key is the document hash, so a redelivery or a retry is a no-op. The shared payload and the idempotency guard come first.

from __future__ import annotations

import logging
from decimal import Decimal
from typing import Literal

from pydantic import BaseModel, ConfigDict, Field

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)-8s | %(message)s")
logger = logging.getLogger("burst")

RouteState = Literal["auto_route", "human_review", "quarantine"]

AUTO_ROUTE = Decimal("0.92")
HUMAN_REVIEW = Decimal("0.75")


class DocResult(BaseModel):
    """Decimal-safe outcome of processing one construction document."""
    model_config = ConfigDict(frozen=True)
    doc_hash: str = Field(min_length=64, max_length=64)  # sha-256 hex; the idempotency key
    cost_impact: Decimal = Field(ge=0, decimal_places=2)  # Decimal, never float
    confidence: Decimal = Field(ge=0, le=1)

    @property
    def route_state(self) -> RouteState:
        if self.confidence >= AUTO_ROUTE:
            return "auto_route"
        if self.confidence >= HUMAN_REVIEW:
            return "human_review"
        return "quarantine"


_SEEN: set[str] = set()  # stand-in for a durable idempotency store (Redis SETNX, unique index)


def already_done(doc_hash: str) -> bool:
    """Idempotency guard: a replayed document is committed at most once."""
    if doc_hash in _SEEN:
        logger.info("skip replay %s", doc_hash[:12])
        return True
    _SEEN.add(doc_hash)
    return False


def process_document(doc_hash: str, raw_cost: str, confidence: str) -> DocResult:
    """Pure, deterministic core. Same input -> same DocResult, so retries are safe."""
    return DocResult(
        doc_hash=doc_hash,
        cost_impact=Decimal(raw_cost).quantize(Decimal("0.01")),
        confidence=Decimal(confidence),
    )

The Celery task wraps that core with durable retry and dead-lettering. acks_late means the broker only drops the task after it succeeds, so a worker crash redelivers it; an exhausted retry raises into the dead-letter queue rather than vanishing.

from celery import Celery
from celery.exceptions import MaxRetriesExceededError

app = Celery("ingest", broker="redis://localhost:6379/0")


@app.task(bind=True, acks_late=True, max_retries=5, default_retry_delay=10)
def ingest_document(self, doc_hash: str, raw_cost: str, confidence: str) -> str:
    """Durable, idempotent task. Retries with backoff; dead-letters when exhausted."""
    if already_done(doc_hash):
        return "duplicate"
    try:
        result = process_document(doc_hash, raw_cost, confidence)
    except Exception as exc:  # transient pull/parse fault -> retry with backoff
        try:
            raise self.retry(exc=exc, countdown=10 * (2 ** self.request.retries))
        except MaxRetriesExceededError:
            logger.error("dead-letter %s after retries: %s", doc_hash[:12], exc)
            app.send_task("ingest.dead_letter", args=[doc_hash, str(exc)])
            return "dead_letter"
    logger.info("celery %s -> %s", doc_hash[:12], result.route_state)
    return result.route_state

The asyncio equivalent handles the whole burst in one process, bounding concurrency with a semaphore so the fan-out never opens more than limit connections at once. It is idempotent and Decimal-safe by reusing the same core, but note what it does not have: no broker, so a crash loses the in-flight batch.

import asyncio


async def _pull_and_process(sem: asyncio.Semaphore, ref: tuple[str, str, str]) -> DocResult | None:
    doc_hash, raw_cost, confidence = ref
    async with sem:  # back-pressure: at most `limit` concurrent pulls
        if already_done(doc_hash):
            return None
        await asyncio.sleep(0)  # stand-in for an awaited I/O-bound document pull
        result = process_document(doc_hash, raw_cost, confidence)
        logger.info("asyncio %s -> %s", doc_hash[:12], result.route_state)
        return result


async def run_burst(refs: list[tuple[str, str, str]], limit: int = 20) -> list[DocResult]:
    """Bounded-gather over a burst. return_exceptions keeps one bad doc from killing the run."""
    sem = asyncio.Semaphore(limit)
    outcomes = await asyncio.gather(
        *(_pull_and_process(sem, r) for r in refs), return_exceptions=True
    )
    results: list[DocResult] = []
    for outcome in outcomes:
        if isinstance(outcome, Exception):
            logger.error("burst item failed, would dead-letter: %s", outcome)
        elif outcome is not None:
            results.append(outcome)
    return results


if __name__ == "__main__":
    batch = [(f"{i:064x}", "7500.00", "0.94") for i in range(3)]
    for r in asyncio.run(run_burst(batch)):
        print(r.doc_hash[:12], r.cost_impact, r.route_state)

Common Mistakes and Gotchas

Choosing asyncio for durable work because it benchmarks faster. A bounded-gather clears an I/O-bound burst quickly, and that speed is seductive — until the process is redeployed mid-burst and every in-flight change order is gone with no redelivery. Speed is the wrong axis for durable cost documents. If losing a document is unacceptable, the broker is not overhead, it is the requirement, and Celery is the answer even though its steady-state throughput per host may be lower.

Skipping the idempotency key on retries. Both models retry, and a retry that is not idempotent double-posts. A Celery task redelivered after acks_late, or an asyncio coroutine re-run after a caught exception, must be keyed on the document hash so the second run is a no-op. Without the guard, one addendum burst can draft the same change order twice and inflate committed cost.

Running an unbounded gather over the whole burst. Firing asyncio.gather over hundreds of document pulls with no semaphore opens hundreds of simultaneous connections, exhausts the source system’s limits, and turns a burst into an outage. The Semaphore(N) is not optional back-pressure; it is what keeps the single-process model from becoming a self-inflicted denial of service against the portal you are pulling from.

Where This Fits in the Pipeline

This decision lives inside the async batching workflows that absorb a bid-period spike before any document reaches validation. Whichever model runs, an exhausted or low-confidence document is handed to the error handling protocols, whose dead-letter queue retains the payload for replay, and SLA-breaching escalations are routed by fallback alert routing so the on-call estimator is paged from one place. The priced tables the burst produces reconcile against the spreadsheet side through the PDF/Excel sync pipelines. The confidence bands used to decide auto-route versus quarantine here are the same 0.92 / 0.75 thresholds applied everywhere on the site, so a document parked during a burst behaves identically to one parked during a quiet afternoon.

Frequently Asked Questions

Related

• Async Batching Workflows
• Error Handling Protocols
• Fallback Alert Routing
• PDF/Excel Sync Pipelines
• Schema Validation Rules

← Back to Async Batching Workflows