Construction Automation CA

Implementing retry logic for failed API document pulls

Construction ingestion pipelines pull submittals, RFIs, change orders, and takeoff spreadsheets from vendor platforms such as Procore, Autodesk Construction Cloud, and Bluebeam on a schedule that never matches the vendor’s uptime. Network instability, aggressive per-tenant rate limits, and transient 5xx faults interrupt those pulls constantly, leaving estimators with incomplete cost data and project managers with stale revision histories. This page covers exactly one slice of the pipeline: how to wrap an outbound document GET so that a transient failure becomes a recoverable, automatically-retried event, while a permanent failure fails fast and lands in a review queue — without ever amplifying vendor throttling or silently fetching a duplicate change-order revision. Getting this boundary deterministic is what lets the rest of an automated document ingestion and parsing workflow trust that a missing record means “genuinely absent,” not “the network blinked.” It is the network-resilience primitive the broader error handling protocols layer depends on.

Key rules and specification

A production retry boundary is governed by a few non-negotiable constraints. Encode each as an explicit branch or typed field — never as an implicit “just retry everything” loop, which is how pipelines get themselves rate-limited into a death spiral.

Rule Specification
Retryable statuses Retry only on 408, 429, 500, 502, 503, 504 and on connection/read timeouts
Non-retryable statuses Fail fast on 400, 401, 403, 404, 422 — malformed request, dead credential, missing resource, or validation failure that repetition cannot fix
Backoff Honor a vendor Retry-After header first; otherwise exponential backoff with randomized jitter to avoid synchronized retry storms
Attempt cap Hard cap (e.g. 5 attempts) and a per-delay ceiling (300 s) so a worker never blocks indefinitely
Idempotency Versioned pulls (change orders, drawing sets) must send If-Match / If-None-Match so a retry cannot ingest a duplicate revision
Timeouts Explicit (connect, read) tuple; large binaries get a long read timeout but never an infinite one
Exhaustion Exhausted retries route to a dead-letter queue with the last exception, never a swallowed None

The non-retryable set matters most. A 401 means the OAuth token expired and belongs in security boundary configuration for rotation, not in a retry loop; a 422 means the payload failed the vendor’s contract and belongs in the same schema validation rules remediation path used elsewhere in the pipeline. Retrying either one only burns quota and delays throughput.

Typed retry policy

Express the policy as a Pydantic v2 model so the constraints above are validated at construction time rather than discovered at 2 a.m. during a bid-period burst. Tagging each pull with the construction document_type lets you tune attempt counts per artifact — a pay application is worth more retries than a thumbnail preview.

from typing import Literal

from pydantic import BaseModel, Field, field_validator

DocumentType = Literal["RFI", "SUBMITTAL", "CHANGE_ORDER", "PAY_APP", "DRAWING_SET"]

# Transient faults worth retrying; everything else fails fast.
RETRYABLE_STATUS_CODES: frozenset[int] = frozenset({408, 429, 500, 502, 503, 504})
PERMANENT_STATUS_CODES: frozenset[int] = frozenset({400, 401, 403, 404, 422})


class RetryPolicy(BaseModel):
    """Validated, per-document retry configuration."""

    document_type: DocumentType
    max_attempts: int = Field(default=5, ge=1, le=10)
    connect_timeout: float = Field(default=10.0, gt=0)
    read_timeout: float = Field(default=300.0, gt=0)   # large binaries need headroom
    max_backoff_seconds: float = Field(default=300.0, gt=0)  # ceiling per delay
    require_idempotency: bool = True

    @field_validator("read_timeout")
    @classmethod
    def read_must_exceed_connect(cls, v: float, info) -> float:
        connect = info.data.get("connect_timeout", 0.0)
        if v <= connect:
            raise ValueError("read_timeout must exceed connect_timeout")
        return v

Production retry implementation

The implementation below uses tenacity for declarative retry control and requests for transport. It distinguishes transient from permanent failures, respects Retry-After, streams large PDF/Excel binaries without exhausting worker memory, and routes exhausted pulls to a dead-letter callback. See the tenacity documentation for the async-compatible decorators referenced in the gotchas below.

import logging
import random
from typing import Callable, Optional

import requests
from requests.exceptions import ConnectionError, HTTPError, Timeout
from tenacity import (
    before_sleep_log,
    retry,
    retry_if_exception,
    stop_after_attempt,
    RetryError,
)

logger = logging.getLogger("construction.doc_ingestion")


def is_retryable(exc: BaseException) -> bool:
    """True only for transient faults — never for permanent client errors."""
    if isinstance(exc, (ConnectionError, Timeout)):
        return True
    if isinstance(exc, HTTPError) and exc.response is not None:
        return exc.response.status_code in RETRYABLE_STATUS_CODES
    return False


def wait_with_retry_after(retry_state) -> float:
    """Respect the vendor Retry-After header; else exponential backoff + jitter."""
    exc = retry_state.outcome.exception()
    if isinstance(exc, HTTPError) and exc.response is not None:
        header = exc.response.headers.get("Retry-After")
        if header and header.isdigit():
            return min(int(header), 300)  # cap to prevent worker starvation
    # Fallback: exponential backoff with randomized jitter to avoid retry storms.
    base = min(2 ** retry_state.attempt_number, 60)
    return base + random.uniform(0, base * 0.5)


def fetch_construction_document(
    url: str,
    headers: dict[str, str],
    policy: RetryPolicy,
) -> bytes:
    """Retrieve a versioned construction document with deterministic retries."""

    @retry(
        retry=retry_if_exception(is_retryable),
        wait=wait_with_retry_after,
        stop=stop_after_attempt(policy.max_attempts),
        before_sleep=before_sleep_log(logger, logging.WARNING),
        reraise=True,
    )
    def _do_fetch() -> bytes:
        # Idempotency: If-None-Match prevents re-ingesting a revision a retry
        # already pulled. The caller seeds headers with the stored ETag.
        if policy.require_idempotency and "If-None-Match" not in headers:
            logger.warning("Idempotent pull requested without ETag for %s", url)
        resp = requests.get(
            url,
            headers=headers,
            timeout=(policy.connect_timeout, policy.read_timeout),
            stream=True,
        )
        resp.raise_for_status()
        # Stream in chunks: a 500 MB drawing set must not be loaded whole.
        payload = bytearray()
        for chunk in resp.iter_content(chunk_size=8192):
            if chunk:
                payload.extend(chunk)
        return bytes(payload)

    return _do_fetch()


def pull_with_dlq_routing(
    url: str,
    headers: dict[str, str],
    policy: RetryPolicy,
    dlq_callback: Callable[[str, BaseException], None],
) -> Optional[bytes]:
    """Catch exhausted/permanent failures and route them to a dead-letter queue."""
    try:
        return fetch_construction_document(url, headers, policy)
    except RetryError as e:
        last = e.last_attempt.exception()
        logger.error("Retries exhausted for %s: %s", url, last)
        dlq_callback(url, last)
    except HTTPError as e:
        # Permanent client error (401/404/422) — fail fast, no retries spent.
        logger.error("Permanent failure for %s: %s", url, e)
        dlq_callback(url, e)
    except Exception as e:  # noqa: BLE001 - last-resort isolation per worker task
        logger.critical("Unexpected failure for %s: %s", url, e)
        dlq_callback(url, e)
    return None

A pull that exhausts its retries is not lost — it lands in the dead-letter queue alongside the same human-review band the pipeline uses for sub-0.92 confidence records, so a document control specialist triages a stuck Procore pull through the identical workflow as a low-confidence OCR extraction.

Retry decision flow for a failed document pull An outbound GET produces an HTTP response that branches by status code. A 2xx success streams chunks and returns bytes. A 4xx response (400, 401, 403, 404, 422) is a permanent failure that fails fast with no retry. A 5xx, 429, 408, or timeout is a transient fault that waits for the Retry-After header or otherwise backs off with jitter, then checks whether attempts remain: if yes it loops back to the GET, if no it is exhausted. Both permanent failures and exhausted retries route to a dead-letter queue carrying the last exception and attempt count. Outbound GET fetch document HTTP response status code 2xx Success stream chunks Return bytes 4xx Permanent — fail fast 400·401·403·404·422 5xx Transient fault 5xx · 429 · 408 · timeout Wait Retry-After else backoff + jitter Attempts left? yes · retry no · exhausted Dead-letter queue last exception · attempt count

Common mistakes and gotchas

  • Retrying 4xx client errors. A blanket except requests.RequestException: retry() re-pulls a 401 five times before failing, wasting quota and delaying every queued document behind it while the token stays dead. Worse, retrying a 404 masks a genuinely deleted change order as a transient blip. Split the predicate: only 408/429/5xx and timeouts retry; permanent codes route straight to the dead-letter queue for credential rotation or estimator review.
  • Backoff with no jitter. Fixed or purely-exponential backoff makes every worker that hit the same 429 wake up at the same instant and stampede the vendor again — the thundering-herd failure. Always add randomized jitter, and honor Retry-After first since the vendor is telling you exactly when its quota window resets.
  • Blocking the event loop with synchronous retries. Dropping this synchronous requests call inside an async batching workflow freezes the whole worker for the entire backoff duration — a 5-minute Retry-After stalls every concurrent pull. Wrap it in asyncio.to_thread, or migrate to httpx with tenacity’s async decorators, so one slow vendor never blocks the rest of the burst.

Integration pointer

This retry boundary sits at the very edge of the pipeline, between the vendor API and everything downstream. A worker in the async batching workflows tier calls pull_with_dlq_routing for each queued document, and only successfully-fetched bytes proceed to extraction and the same schema validation rules gate every record must pass. When the pull is a versioned change order, the If-None-Match discipline here is what keeps the idempotency promise that parsing unstructured PDF change orders with Python and pypdf relies on downstream. Validate the boundary against simulated outages with toxiproxy or pytest-httpserver: assert that 429 honors Retry-After, that 404/401 bypass retries entirely, and that connection timeouts respect the (connect, read) tuple instead of hanging.

Frequently asked questions

Should I retry a 429 immediately or wait for the Retry-After header?

Always wait. A 429 Too Many Requests means you have already exceeded the tenant quota, so an immediate retry is guaranteed to fail and counts against you again. Parse the Retry-After header — vendors send it precisely to tell you when the window resets — and only fall back to exponential backoff with jitter when the header is absent or non-numeric.

Why exclude 404 from the retry set when documents do sometimes appear late?

A 404 is a permanent answer to the request as written: the resource is not at that URL right now. Eventual-consistency lag is real, but the correct fix is a separate scheduled re-poll with a fresh request, not a tight retry loop that burns its whole attempt budget in seconds while the document is still being published vendor-side. Route the 404 to the dead-letter queue and let a delayed re-enqueue handle late arrivals.

How do I keep retries from ingesting a duplicate change-order revision?

Send conditional request headers. Store the last ETag you successfully ingested and pass it as If-None-Match; the vendor returns 304 Not Modified when nothing changed, so a retry that races a successful pull cannot create a second copy of the same revision. For writes or claims, use If-Match against the expected version so a stale retry is rejected rather than applied twice.

What belongs in the dead-letter queue payload?

The original URL, the document_type, the final exception (status code and message), the attempt count, and a timestamp — enough for a document control specialist to triage without re-running the job. Keep it strongly typed and machine-readable so the same alert-routing logic that handles low-confidence extractions can classify an exhausted pull by error code.

← Back to Error Handling Protocols