Skip to content

Designing fallback routing for disconnected field devices

Connectivity on an active construction site is never a given: cellular dead zones below a deck pour, RF interference from tower cranes, and basement levels with no signal routinely cut a tablet or sensor off mid-submission. This page covers exactly one slice of that problem — how to design an on-device fallback queue that captures a change order, RFI, daily log, or safety incident the instant it is created, persists it durably while the device is offline, and then replays it deterministically when the radio comes back, without losing data, duplicating postings, or routing a high-impact payload to the wrong reviewer. It is the device-side complement to the broader fallback alert routing workflow, which assumes a payload has already reached the gateway; here the gateway is unreachable, so the routing decision must be staged locally and resolved on reconnect. Every field record it queues still carries the identifiers defined by the parent Construction Data Architecture & Taxonomy layer, so the codes that route it offline are the same ones that aggregate it once it lands.

Key rules and specification

Treat the disconnected window as a first-class operating mode, not an error path. The constraints below are what separate a queue that silently corrupts the ledger from one that is replay-safe.

Rule Specification Why it matters downstream
Durable local store Write to a transactional store (SQLite WAL), not in-memory or a flat file An app crash or battery death before sync must not lose the record
Idempotency key Deterministic hash of device_id + captured_at + doc_type Network flapping replays the same payload; the server must dedupe
Monotonic capture time Stamp captured_at in UTC; also record device clock-skew A device with a wrong clock posts a change order to the wrong day
Schema version pin Every record carries a schema_version Literal A device offline for weeks may pre-date the current gateway schema
Priority preemption Safety incidents and high-cost change orders jump the queue A routine daily log must never delay an injury report on reconnect
Confidence-band routing Score each record 0.92 auto-route, 0.75–0.92 review, <0.75 quarantine Stale or skewed records get a human, not a silent auto-post
Bounded retention Cap queue size and age; quarantine never auto-purges A device offline for a month cannot exhaust local storage

Two rules carry the most weight on a disconnected device. Monotonic capture time is the one most teams overlook: a field tablet whose clock has drifted (or reset to epoch after a dead battery) will stamp every queued record with a wrong time, so on reconnect the gateway routes a $40,000 change order against the wrong billing period. Record the skew at capture and let it drive the confidence score. And idempotency is non-negotiable, because the failure mode here is not a single outage but flapping — the radio oscillates between one and two bars, the dispatch loop fires repeatedly, and without a deterministic key the same safety incident is filed three times.

Production code example

The module below is a self-contained on-device router. It validates each field record with a Pydantic v2 model (Literal-typed document and discipline codes, regex-checked WBS element and MasterFormat division), persists it to a WAL-mode SQLite queue, and on reconnect assigns a routing confidence that follows the pipeline’s canonical bands — 0.92 and above auto-routes, 0.75–0.92 goes to human review, and anything below 0.75 (or any safety incident) is quarantined for a person to clear. It is the same confidence contract the schema validation rules gates apply at the gateway, pushed down to the edge.

from __future__ import annotations

import hashlib
import json
import logging
import sqlite3
from contextlib import contextmanager
from datetime import datetime, timezone
from typing import Any, Iterator, Literal

from pydantic import BaseModel, Field, field_validator

logger = logging.getLogger("construction.field_fallback")

DocType = Literal["change_order", "rfi", "daily_log", "submittal", "safety_incident"]
Discipline = Literal["ARCH", "STR", "MEP", "CIV", "ELEC", "PLMB"]
SchemaVersion = Literal["1.0", "1.1", "2.0"]
RoutingAction = Literal["auto_route", "human_review", "quarantine"]

# WBS elements follow PROJ-NNN-DIV-NN; MasterFormat divisions are XX XX XX.
_WBS = r"^[A-Z]{2,5}-\d{3}-\d{2}-\d{2}$"
_MASTERFORMAT = r"^\d{2} \d{2} \d{2}$"

# A payload below this confidence is never auto-posted on reconnect.
AUTO_ROUTE = 0.92
REVIEW_FLOOR = 0.75
# Beyond this skew the device clock is untrustworthy; force review.
MAX_TRUSTED_SKEW_SECONDS = 120


class FieldRecord(BaseModel):
    """A field-captured payload staged for routing once connectivity returns."""

    model_config = {"frozen": True, "extra": "forbid"}

    device_id: str = Field(min_length=3, max_length=48)
    doc_type: DocType
    discipline: Discipline
    wbs_element: str = Field(pattern=_WBS)
    masterformat_division: str = Field(pattern=_MASTERFORMAT)
    budget_impact_usd: float = Field(ge=0.0)
    schema_version: SchemaVersion
    captured_at: datetime
    clock_skew_seconds: float = Field(ge=0.0, default=0.0)
    body: dict[str, Any] = Field(default_factory=dict)

    @field_validator("captured_at")
    @classmethod
    def _require_utc(cls, v: datetime) -> datetime:
        # A naive timestamp from an offline device routes to the wrong day.
        if v.tzinfo is None:
            raise ValueError("captured_at must be timezone-aware UTC")
        return v.astimezone(timezone.utc)

    @property
    def idempotency_key(self) -> str:
        # Deterministic: flapping replays collapse to one server-side record.
        seed = f"{self.device_id}:{self.captured_at.isoformat()}:{self.doc_type}"
        return hashlib.sha256(seed.encode()).hexdigest()

    def routing(self, current_schema: SchemaVersion) -> tuple[RoutingAction, float, str]:
        """Score the record against the canonical confidence bands."""
        confidence = 1.0
        notes: list[str] = []
        if self.clock_skew_seconds > MAX_TRUSTED_SKEW_SECONDS:
            confidence -= 0.30
            notes.append(f"clock skew {self.clock_skew_seconds:.0f}s")
        if self.schema_version != current_schema:
            confidence -= 0.20
            notes.append(f"schema {self.schema_version} != {current_schema}")
        detail = "; ".join(notes) or "clean"

        # Safety incidents are escalated to a human regardless of score.
        if self.doc_type == "safety_incident":
            return "quarantine", confidence, "safety_incident: human review required"
        if confidence >= AUTO_ROUTE:
            return "auto_route", confidence, detail
        if confidence >= REVIEW_FLOOR:
            return "human_review", confidence, detail
        return "quarantine", confidence, detail


class FieldQueue:
    """Durable, replay-safe on-device queue backed by WAL-mode SQLite."""

    def __init__(self, db_path: str) -> None:
        self.db_path = db_path
        with self._conn() as c:
            c.execute("PRAGMA journal_mode=WAL")  # survive crashes mid-write
            c.execute(
                """CREATE TABLE IF NOT EXISTS queue (
                    key TEXT PRIMARY KEY,
                    priority INTEGER NOT NULL,
                    captured_at TEXT NOT NULL,
                    record TEXT NOT NULL,
                    synced INTEGER NOT NULL DEFAULT 0
                )"""
            )

    @contextmanager
    def _conn(self) -> Iterator[sqlite3.Connection]:
        conn = sqlite3.connect(self.db_path)
        try:
            yield conn
            conn.commit()
        except Exception:
            conn.rollback()
            raise
        finally:
            conn.close()

    def enqueue(self, record: FieldRecord) -> str:
        # Safety incidents (0) preempt change orders (1) preempt everything else (2).
        priority = 0 if record.doc_type == "safety_incident" else (
            1 if record.doc_type == "change_order" and record.budget_impact_usd > 50_000 else 2
        )
        with self._conn() as c:
            c.execute(
                "INSERT OR IGNORE INTO queue (key, priority, captured_at, record) "
                "VALUES (?, ?, ?, ?)",
                (record.idempotency_key, priority,
                 record.captured_at.isoformat(), record.model_dump_json()),
            )
        logger.info("queued %s key=%s", record.doc_type, record.idempotency_key[:12])
        return record.idempotency_key

    def drain(self, current_schema: SchemaVersion) -> list[dict[str, Any]]:
        """On reconnect: route pending records by priority, then capture order."""
        results: list[dict[str, Any]] = []
        with self._conn() as c:
            rows = c.execute(
                "SELECT key, record FROM queue WHERE synced = 0 "
                "ORDER BY priority ASC, captured_at ASC"
            ).fetchall()
            for key, raw in rows:
                record = FieldRecord.model_validate_json(raw)
                action, confidence, detail = record.routing(current_schema)
                if action == "auto_route":
                    # Real dispatch goes here; the key is the server dedupe token.
                    c.execute("UPDATE queue SET synced = 1 WHERE key = ?", (key,))
                    logger.info("auto_route %s conf=%.2f", key[:12], confidence)
                else:
                    logger.warning("%s %s conf=%.2f (%s)", action, key[:12], confidence, detail)
                results.append({"key": key, "action": action,
                                "confidence": round(confidence, 2), "detail": detail})
        return results
On-device fallback queue: capture offline, replay on reconnect While the device is offline in a cellular dead zone, a field capture (RFI, change order, or safety log) is validated by a Pydantic FieldRecord with Literal codes and a UTC capture time plus clock skew, then written to a crash-safe WAL SQLite queue under an idempotency key (sha256 of device, time, and type) that makes flapping retries deduplicate. When the radio returns, drain() reads unsynced records ordered by priority — safety incidents first, then change orders over fifty thousand dollars, then routine logs — and scores each one, subtracting 0.30 for clock skew beyond 120 seconds and 0.20 for schema drift. A score of 0.92 or higher auto-routes and marks the record synced; 0.75 to 0.92 goes to human review; below 0.75, or any safety incident, is quarantined for a person to clear and is never auto-purged. On device · offline (cellular dead zone) Field capture RFI · change order · safety log Pydantic FieldRecord Literal codes · UTC time + skew WAL SQLite queue atomic write · crash-safe INSERT OR IGNORE key = sha256(device · time · type) flap-safe dedupe radio returns On reconnect · replay drain() where synced = 0 ORDER BY priority, captured_at priority order 1 · safety_incident 2 · change order > $50k 3 · daily log · RFI confidence score − 0.30 skew > 120s − 0.20 schema drift ≥ 0.92 0.75–0.92 < 0.75 · safety Auto-route dispatch · SET synced = 1 Human review reconciliation console Quarantine person clears · never purged

Common mistakes and gotchas

  • Trusting the device clock. An offline tablet whose battery died comes back with a clock at epoch or hours adrift, and a naive datetime.now() stamp routes the record to the wrong billing period or sort position. Always capture clock_skew_seconds against a server time delta at sync and let it pull the confidence below 0.92 so a human confirms the date before it posts — the same naive-timestamp failure that breaks routing across time zones at the gateway.
  • Queuing without an idempotency key. Network flapping is the normal case, not the exception: the radio reconnects, the dispatch loop fires, and the connection drops again mid-POST. Without the deterministic idempotency_key, the retry files a second safety incident or a duplicate change order. The INSERT OR IGNORE on the hash collapses replays locally, and the same key becomes the server’s dedupe token. This is the device-side mirror of the retry discipline in error handling protocols.
  • Auto-purging to reclaim storage. When the bounded queue fills, deleting the oldest records to make room throws away exactly the data the device went offline to protect. Cap retention by count and age, but only ever purge records already marked synced; quarantined and unsent records must block new captures (and warn the user) rather than be silently dropped.

Integration pointer

This queue runs at the very edge of the pipeline, on the field device itself, and hands off to the server-side fallback alert routing engine the moment connectivity returns — auto_route records flow straight to the gateway under their idempotency key, while human_review and quarantine records surface in the same reconciliation console an estimator uses for flagged cost data. The budget_impact_usd threshold that preempts a change order in the local queue should match the executive-bypass threshold defined in budget code standardization, so a payload that is urgent offline stays urgent online. And because every queued record is already a validated, versioned model, it slots cleanly into the gateway’s RFI schema design contract without a second parse — the device and the server speak the same schema, the same confidence bands, and the same audit format.

Frequently asked questions

Why SQLite WAL instead of a JSON file or in-memory queue on the device?

A field device can lose power or be force-killed by the OS at any moment, often mid-write. An in-memory queue evaporates and a flat JSON file can be left half-written and unparseable. SQLite in WAL (write-ahead logging) mode commits each record atomically and recovers cleanly after a crash, so a change order captured one second before the battery dies is still there on the next boot. It also gives you indexed priority ordering for the reconnect drain without loading the whole queue into memory.

How does the confidence score decide routing on reconnect?

Each record starts at 1.0 and loses confidence for the two risks unique to a disconnected device: a clock skew beyond the trusted window and a schema_version that no longer matches the gateway. A score of 0.92 or higher auto-routes, 0.75 to 0.92 goes to human review, and anything below 0.75 is quarantined. Safety incidents are quarantined for human review regardless of score, because an injury report is never something a heuristic should silently auto-file.

What stops network flapping from filing the same record twice?

The idempotency_key is a SHA-256 hash of device_id, captured_at, and doc_type, so an identical capture always produces an identical key. Locally, INSERT OR IGNORE means a retried enqueue is a no-op. On the wire, the same key travels with the payload as the server’s dedupe token, so even if the radio drops mid-POST and the drain loop resends, the gateway resolves both attempts to one authoritative record instead of double-booking it.

How long should records stay on the device before they sync?

Retention is bounded by count and age, not by an aggressive purge. Records marked synced can be trimmed once they exceed the retention window, but unsent and quarantined records must persist until they are delivered or cleared by a person — even if that means the device for several weeks. If the queue approaches its cap, warn the user and block new low-priority captures rather than discarding queued data, since storage pressure is never a reason to lose an unsent safety or cost record.

← Back to Fallback Alert Routing