Fallback Alert Routing

In construction project tracking, change orders rarely follow a linear approval path. Field conditions shift, budget thresholds trip multi-tier reviews, and mobile connectivity on active job sites drops without warning. The specific sub-problem this page solves is how an automated pipeline guarantees that a cost-impacting alert reaches a qualified approver even when the primary route fails — when the assigned project manager is unreachable, the network is down, or the cost-center owner cannot be resolved with confidence. When that guarantee is missing, change orders stall silently in transit: a $90,000 scope addition sits unacknowledged for days, the schedule slips against an unapproved baseline, and the audit trail shows a gap no one can explain. Inside a deterministic construction data architecture and taxonomy, fallback alert routing is the escalation layer that makes “the alert was never delivered” an impossible outcome. This page details the ingestion-to-acknowledgment pipeline for that layer: the schema contract that carries routing metadata, the confidence-scored resolution of the responsible approver, and the timeout-driven escalation that survives network fragmentation and approver unavailability. It targets Python automation builders, project managers, and estimators who need predictable alert delivery under real-world failure.

Prerequisites

This subsystem sits downstream of document parsing and approver-directory lookups, and upstream of the notification transport (SMS, email, push, or a workflow orchestrator). Before implementing the patterns below, you need:

Python 3.11+ with pydantic v2 for typed validation and the standard-library hashlib, decimal, re, enum, datetime, and logging modules. Every monetary field is a Decimal; no floating-point money participates in a routing threshold comparison.
A message broker — RabbitMQ, AWS SQS, or Kafka — with a configured dead-letter queue so payloads that fail validation after maximum retries are parked for reconciliation rather than dropped. The replay semantics here reuse the same dead-letter conventions described in error handling protocols.
A persistent local queue on the field client (SQLite or an embedded key-value store) so alerts raised while a device is offline survive an app restart. The offline-first dispatch contract is covered in depth in designing fallback routing for disconnected field devices.
A resolved cost taxonomy. Each alert references a WBS node that must align with the project’s breakdown structure, the same taxonomy that drives WBS mapping strategies, and a budget code validated against the canonical scheme from budget code standardization. Routing decisions are only as trustworthy as the cost data they key on.
An upstream extraction step that has already produced a structured change order record and a per-field confidence score. Alerts lifted from scanned change orders inherit confidence metadata; the resolver below depends on it to decide whether an approver target can be trusted automatically.

The pipeline assumes inbound payloads have already cleared structural schema validation rules at the gateway, so the work here is routing evaluation and escalation rather than document parsing. During bid-period bursts, alerts arrive through the same async queue architecture that buffers the rest of the ingestion pipeline.

Architecture: inputs, decisions, and escalation branches

Fallback routing is not a single if statement — it is an ordered decision pipeline where every branch terminates in a structured, replayable outcome rather than a dropped message. The engine evaluates three concurrent conditions: device connectivity, financial impact, and approver availability. A payload that passes connectivity can still exceed a financial threshold; an approver target resolved with low confidence should never auto-route. The diagram below traces a change order alert from inbound payload to either an acknowledged approval, an escalated fallback, or a quarantined record awaiting manual triage.

Every branch terminates in a structured, replayable outcome rather than a dropped message: connectivity is evaluated before identity, the confidence bands gate owner resolution, the financial threshold compresses the chain to the executive, and an unacknowledged alert promotes the next identifier — with each terminal outcome written to the append-only audit log.

The confidence branches map to the site-canonical bands, applied here to approver resolution: when the engine maps a payload’s wbs_node to a responsible cost-center owner, a match score of 0.92 or above auto-routes to that owner, a score of 0.75–0.92 routes the alert but flags it for human review, and anything below 0.75 is quarantined to a manual-triage queue rather than risk notifying the wrong stakeholder about a six-figure change.

Stage	Input	Output	Error branch
Schema validation	Raw payload dict	Typed `ChangeOrderPayload`	Pattern/type failure → quarantine
Connectivity check	Device heartbeat age	Online / degraded / offline	Stale heartbeat → offline fallback
Owner resolution	`wbs_node` + discipline	Approver id + confidence	Score < 0.75 → quarantine
Threshold routing	`budget_impact_usd`	Approver or executive chain	Over tolerance → executive bypass
Timeout evaluation	Acknowledgment age	Hold or promote next in chain	Timeout → promote fallback

Step-by-step implementation

The following module is production-grade: strict typing, Pydantic v2 validation, deterministic hashing for idempotent retries, confidence-scored resolution, and structured logging. It is designed for direct integration into serverless functions, Celery workers, or containerized microservices.

Step 1 — Model the payload with regex-validated routing metadata

Every alert must carry explicit routing metadata rather than relying on implicit organizational hierarchy. The schema constrains the construction-domain identifiers — the wbs_node follows the canonical PROJ-NNN-DIV-NN element pattern, the masterformat_division follows the XX XX XX MasterFormat pattern, and the discipline is a Literal so an unknown trade can never enter the router. Monetary impact is a Decimal; the escalation chain is validated to be non-empty so there is always a fallback target.

import hashlib
import logging
from datetime import datetime, timezone
from decimal import Decimal
from enum import Enum
from typing import Literal, Optional

from pydantic import BaseModel, Field, field_validator

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("fallback_router")

# Site-canonical confidence bands for approver resolution.
AUTO_ROUTE_THRESHOLD = Decimal("0.92")
HUMAN_REVIEW_THRESHOLD = Decimal("0.75")

# Financial threshold above which an alert bypasses mid-tier review.
EXECUTIVE_BYPASS_USD = Decimal("50000.00")

Discipline = Literal["ARCH", "STR", "MEP", "CIV", "ELEC", "PLMB"]


class ConnectivityStatus(str, Enum):
    ONLINE = "online"
    DEGRADED = "degraded"
    OFFLINE = "offline"


class ChangeOrderPayload(BaseModel):
    change_order_id: str = Field(..., min_length=1)
    primary_approver_id: str
    secondary_approver_id: str
    # Ordered, role-based fallback targets; last entry is the executive backstop.
    escalation_chain: list[str] = Field(default_factory=list)
    timeout_seconds: int = Field(ge=30, default=3600)
    budget_impact_usd: Decimal = Field(ge=Decimal("0"))
    # WBS element pattern PROJ-NNN-DIV-NN, e.g. RVR-014-03-02.
    wbs_node: str = Field(pattern=r"^[A-Z]{2,5}-\d{3}-\d{2}-\d{2}$")
    # MasterFormat division pattern XX XX XX, e.g. 03 30 00.
    masterformat_division: str = Field(pattern=r"^\d{2} \d{2} \d{2}$")
    discipline: Discipline
    device_connectivity_status: ConnectivityStatus
    submitted_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))

    @field_validator("escalation_chain")
    @classmethod
    def validate_chain(cls, v: list[str]) -> list[str]:
        if not v:
            raise ValueError("escalation_chain must contain at least one fallback identifier")
        return v

Step 2 — Resolve the cost-center owner with a confidence score

The router maps the payload’s wbs_node and discipline to a responsible approver. In production this is a directory lookup or fuzzy match against an org chart; the score it returns drives the routing state. Resolution is separated from the routing decision so the confidence band can be applied uniformly, exactly as it is for alias matching in budget code standardization.

class ResolverOutcome(str, Enum):
    AUTO = "auto_route"           # >= 0.92
    REVIEW = "human_review"       # 0.75 - 0.92
    QUARANTINE = "quarantine"     # < 0.75


class OwnerResolution(BaseModel):
    approver_id: Optional[str]
    confidence: Decimal
    outcome: ResolverOutcome


def resolve_cost_center_owner(
    payload: ChangeOrderPayload,
    directory: dict[str, tuple[str, Decimal]],
) -> OwnerResolution:
    """
    Map a WBS node to its cost-center owner and classify the match by confidence.

    `directory` maps a wbs_node to (approver_id, confidence). A real implementation
    would fall back to a discipline-level owner when the exact node is absent; the
    confidence band, not the lookup mechanics, is what gates the routing decision.
    """
    approver_id, confidence = directory.get(
        payload.wbs_node, (payload.primary_approver_id, Decimal("0.0"))
    )

    if confidence >= AUTO_ROUTE_THRESHOLD:
        outcome = ResolverOutcome.AUTO
    elif confidence >= HUMAN_REVIEW_THRESHOLD:
        outcome = ResolverOutcome.REVIEW
    else:
        # Below 0.75 we do not trust any owner mapping for a financial alert.
        outcome = ResolverOutcome.QUARANTINE
        approver_id = None

    logger.info(
        "Resolved CO %s -> %s (confidence=%s, outcome=%s)",
        payload.change_order_id, approver_id, confidence, outcome.value,
    )
    return OwnerResolution(approver_id=approver_id, confidence=confidence, outcome=outcome)

Step 3 — Evaluate the routing decision

The decision engine applies the three conditions in priority order. Connectivity comes first because an offline device can never receive a synchronous notification regardless of who the correct approver is. Financial impact compresses the chain — a change above the executive bypass threshold routes straight to the last identifier. Finally, a timed-out acknowledgment promotes the next identifier in the chain. The deterministic SHA-256 hash of the change order id keys idempotent processing so a broker redelivery is a no-op, not a duplicate alert.

class RoutingResult(BaseModel):
    target_approver: Optional[str]
    routing_path: str
    fallback_triggered: bool
    needs_human_review: bool
    reason: str
    payload_hash: str


def compute_idempotent_hash(order_id: str) -> str:
    """Deterministic SHA-256 of the change order id for idempotent retries."""
    return hashlib.sha256(order_id.encode("utf-8")).hexdigest()


def evaluate_routing(
    payload: ChangeOrderPayload,
    directory: dict[str, tuple[str, Decimal]],
    now: Optional[datetime] = None,
) -> RoutingResult:
    now = now or datetime.now(timezone.utc)
    payload_hash = compute_idempotent_hash(payload.change_order_id)

    # 1. Connectivity fallback — offline devices cannot be reached synchronously.
    if payload.device_connectivity_status == ConnectivityStatus.OFFLINE:
        target = payload.escalation_chain[0]
        logger.info("Offline fallback engaged for CO %s -> %s", payload.change_order_id, target)
        return RoutingResult(
            target_approver=target,
            routing_path="sms_email_fallback",
            fallback_triggered=True,
            needs_human_review=False,
            reason="device_offline_fallback",
            payload_hash=payload_hash,
        )

    # 2. Resolve the responsible owner and apply the confidence band.
    resolution = resolve_cost_center_owner(payload, directory)
    if resolution.outcome == ResolverOutcome.QUARANTINE:
        return RoutingResult(
            target_approver=None,
            routing_path="quarantine",
            fallback_triggered=True,
            needs_human_review=True,
            reason="owner_resolution_below_threshold",
            payload_hash=payload_hash,
        )

    target = resolution.approver_id or payload.primary_approver_id
    needs_review = resolution.outcome == ResolverOutcome.REVIEW
    reason = "primary_route_available"

    # 3. Financial-threshold escalation — bypass mid-tier for high-impact changes.
    if payload.budget_impact_usd > EXECUTIVE_BYPASS_USD:
        target = payload.escalation_chain[-1]
        reason = "executive_threshold_bypass"
        logger.info("High impact %s on CO %s -> executive %s",
                    payload.budget_impact_usd, payload.change_order_id, target)

    # 4. Timeout promotion — unacknowledged past timeout promotes the next target.
    age_seconds = (now - payload.submitted_at).total_seconds()
    if age_seconds > payload.timeout_seconds:
        target = payload.escalation_chain[0]
        reason = "approval_timeout"
        logger.warning("Timeout on CO %s -> promote %s", payload.change_order_id, target)
        return RoutingResult(
            target_approver=target, routing_path="escalation",
            fallback_triggered=True, needs_human_review=needs_review,
            reason=reason, payload_hash=payload_hash,
        )

    return RoutingResult(
        target_approver=target,
        routing_path="standard_approval",
        fallback_triggered=False,
        needs_human_review=needs_review,
        reason=reason,
        payload_hash=payload_hash,
    )

Step 4 — Wrap ingestion with deterministic error handling

The ingestion wrapper is the only place that touches untrusted input. A ValidationError routes to the dead-letter queue for reconciliation; any other exception is logged at critical and the payload is held, never silently dropped. The returned RoutingResult is what the transport layer dispatches and the audit log records.

from pydantic import ValidationError


def process_change_order(
    raw_data: dict,
    directory: dict[str, tuple[str, Decimal]],
) -> Optional[RoutingResult]:
    """Validate, route, and surface every outcome as a structured result or DLQ signal."""
    try:
        payload = ChangeOrderPayload.model_validate(raw_data)
    except ValidationError as exc:
        logger.error("Schema validation failed; routing to DLQ: %s", exc)
        return None  # caller publishes the raw message to the dead-letter queue

    try:
        result = evaluate_routing(payload, directory)
    except Exception as exc:  # defensive: never lose a financial alert
        logger.critical("Unexpected routing error for %s: %s", payload.change_order_id, exc)
        return None

    logger.info("CO %s routed via %s -> %s (review=%s)",
                payload.change_order_id, result.routing_path,
                result.target_approver, result.needs_human_review)
    return result


if __name__ == "__main__":
    owner_directory = {"RVR-014-03-02": ("USR_PM_01", Decimal("0.97"))}
    sample = {
        "change_order_id": "CO-2024-8842",
        "primary_approver_id": "USR_PM_01",
        "secondary_approver_id": "USR_EST_02",
        "escalation_chain": ["USR_DIR_03", "USR_VP_04"],
        "timeout_seconds": 3600,
        "budget_impact_usd": "75000.00",
        "wbs_node": "RVR-014-03-02",
        "masterformat_division": "03 30 00",
        "discipline": "STR",
        "device_connectivity_status": "online",
        "submitted_at": "2024-05-15T08:30:00Z",
    }
    decision = process_change_order(sample, owner_directory)
    print(decision.model_dump_json(indent=2) if decision else "REJECTED -> DLQ")

Schema and configuration reference

Field	Type / constraint	Purpose	Failure mode
`change_order_id`	`str`, min length 1	Idempotency key (hashed)	Empty → validation error → DLQ
`primary_approver_id`	`str`	First-line approver	—
`secondary_approver_id`	`str`	Backstop when chain is short	—
`escalation_chain`	`list[str]`, non-empty	Ordered fallback targets	Empty → validation error → DLQ
`timeout_seconds`	`int`, ≥ 30	Acknowledgment window	< 30 → validation error
`budget_impact_usd`	`Decimal`, ≥ 0	Threshold comparison	Float input loses precision
`wbs_node`	regex `^[A-Z]{2,5}-\d{3}-\d{2}-\d{2}$`	Resolver lookup key	Malformed → DLQ
`masterformat_division`	regex `^\d{2} \d{2} \d{2}$`	MasterFormat alignment	Malformed → DLQ
`discipline`	`Literal[...]`	Discipline-level owner fallback	Unknown trade → DLQ
`device_connectivity_status`	`Enum`	Selects synchronous vs. offline path	—

Constant	Value	Meaning
`AUTO_ROUTE_THRESHOLD`	`0.92`	Resolver score that auto-routes to the resolved owner
`HUMAN_REVIEW_THRESHOLD`	`0.75`	Floor for routing with a human-review flag
`EXECUTIVE_BYPASS_USD`	`50000.00`	Impact above which the chain compresses to the executive

Verification and testing

Confirm correct behavior with deterministic assertions over each branch. Because the engine accepts an injected now, timeout logic is testable without sleeping.

from datetime import timedelta

directory = {"RVR-014-03-02": ("USR_PM_01", Decimal("0.97"))}
base = {
    "change_order_id": "CO-TEST-1", "primary_approver_id": "USR_PM_01",
    "secondary_approver_id": "USR_EST_02", "escalation_chain": ["USR_DIR_03", "USR_VP_04"],
    "wbs_node": "RVR-014-03-02", "masterformat_division": "03 30 00",
    "discipline": "STR", "device_connectivity_status": "online",
    "budget_impact_usd": "1000.00",
    "submitted_at": datetime.now(timezone.utc).isoformat(),
}

# Offline payloads take the SMS/email fallback to the first chain entry.
offline = ChangeOrderPayload.model_validate({**base, "device_connectivity_status": "offline"})
assert evaluate_routing(offline, directory).routing_path == "sms_email_fallback"

# High impact compresses the chain to the executive backstop.
big = ChangeOrderPayload.model_validate({**base, "budget_impact_usd": "75000.00"})
assert evaluate_routing(big, directory).target_approver == "USR_VP_04"

# A stale payload past its timeout promotes the next identifier.
stale = ChangeOrderPayload.model_validate(base)
later = stale.submitted_at + timedelta(seconds=stale.timeout_seconds + 1)
assert evaluate_routing(stale, directory, now=later).reason == "approval_timeout"

# A low-confidence owner mapping quarantines rather than guessing.
low = ChangeOrderPayload.model_validate({**base, "wbs_node": "ZZZ-999-99-99"})
assert evaluate_routing(low, directory).routing_path == "quarantine"

# Idempotency: the same id always hashes identically.
assert compute_idempotent_hash("CO-TEST-1") == compute_idempotent_hash("CO-TEST-1")
print("all routing assertions passed")

Run the module directly (python fallback_router.py) to see a fully serialized RoutingResult, or wire these assertions into CI as a pytest module so a schema change that breaks a branch fails the build.

Troubleshooting

Alerts silently disappear during a site outage. The device went offline but the client never persisted the payload locally, so the message was lost when the app was backgrounded. Confirm the offline branch writes to the persistent local queue before attempting any network call, and pair it with the replay loop from designing fallback routing for disconnected field devices.

A six-figure change routes to the wrong reviewer. The wbs_node resolved to an owner with a confidence below 0.92, so the engine routed on a weak match. Inspect the resolver score: scores in the 0.75–0.92 band should carry a needs_human_review flag, and anything below 0.75 should quarantine. If a valid node is scoring low, the directory is stale — reconcile it against current WBS mapping strategies.

The same alert fires twice after a broker redelivery. Idempotency is not enforced at the sink. The payload_hash must be checked against an already-processed set before dispatch; redelivery after a transient fault is expected, so the second pass should be a no-op rather than a duplicate notification.

A threshold comparison routes a change to the wrong tier by one cent. budget_impact_usd was ingested as a float, so 49999.995 rounded across the 50000.00 boundary. Keep the field as Decimal end to end and reject float input at the gateway; binary floating point cannot represent currency exactly.

Valid change orders pile up in the dead-letter queue. The wbs_node or masterformat_division regex is rejecting a legitimate but differently formatted code — for example a European source emitting 03.30.00. Normalize the code to the canonical pattern during ingestion (the same normalization that powers budget code standardization) before it reaches this schema, rather than loosening the contract.

Frequently Asked Questions

Why carry an explicit escalation_chain instead of looking up the org chart at routing time?

Routing decisions must be deterministic and replayable. If the engine queried a live org chart, a re-run after a personnel change would produce a different target for the same historical alert, breaking the audit trail. Embedding the ordered chain in the payload freezes the routing intent at submission time, so a redelivered or replayed message always escalates to the same identifiers.

How do the confidence bands apply to alert routing?

They govern owner resolution. When the engine maps a wbs_node to a cost-center owner, a match of 0.92 or above auto-routes to that owner. A match of 0.75–0.92 still routes but sets a needs_human_review flag so a coordinator confirms the target. Below 0.75 the alert is quarantined to manual triage — a financial change is never sent to a weakly matched approver.

Why is budget_impact_usd a Decimal rather than a float?

Threshold comparisons decide which approval tier sees the change. Binary floating point cannot represent values like 50000.00 exactly, so a float can land a change on the wrong side of the executive bypass by a fraction of a cent. A Decimal field, validated at the boundary, makes the threshold comparison exact and auditable.

What happens when the device is offline at the moment the alert is raised?

The engine takes the offline branch immediately: it persists the payload to a local queue and marks the route as an SMS/email fallback to the first chain entry, because a synchronous push cannot reach a disconnected device. On reconnection, a dispatch loop with idempotency keys and exponential backoff replays the queue without creating duplicates.

Why hash the change order id for idempotency?

Brokers redeliver. A deterministic SHA-256 of the change_order_id is a stable key the sink can check before notifying an approver, so a message that is redelivered after a transient fault produces no second alert. Pairing the hash check with the frozen escalation chain makes the whole route idempotent end to end.

← Back to Construction Data Architecture & Taxonomy

Fallback Alert Routing

Explore in this section