Designing fallback routing for disconnected field devices
Connectivity on an active construction site is never a given: cellular dead zones below a deck pour, RF interference from tower cranes, and basement levels with no signal routinely cut a tablet or sensor off mid-submission. This page covers exactly one slice of that problem — how to design an on-device fallback queue that captures a change order, RFI, daily log, or safety incident the instant it is created, persists it durably while the device is offline, and then replays it deterministically when the radio comes back, without losing data, duplicating postings, or routing a high-impact payload to the wrong reviewer. It is the device-side complement to the broader fallback alert routing workflow, which assumes a payload has already reached the gateway; here the gateway is unreachable, so the routing decision must be staged locally and resolved on reconnect. Every field record it queues still carries the identifiers defined by the parent Construction Data Architecture & Taxonomy layer, so the codes that route it offline are the same ones that aggregate it once it lands.
Key rules and specification
Treat the disconnected window as a first-class operating mode, not an error path. The constraints below are what separate a queue that silently corrupts the ledger from one that is replay-safe.
| Rule | Specification | Why it matters downstream |
|---|---|---|
| Durable local store | Write to a transactional store (SQLite WAL), not in-memory or a flat file | An app crash or battery death before sync must not lose the record |
| Idempotency key | Deterministic hash of device_id + captured_at + doc_type |
Network flapping replays the same payload; the server must dedupe |
| Monotonic capture time | Stamp captured_at in UTC; also record device clock-skew |
A device with a wrong clock posts a change order to the wrong day |
| Schema version pin | Every record carries a schema_version Literal |
A device offline for weeks may pre-date the current gateway schema |
| Priority preemption | Safety incidents and high-cost change orders jump the queue | A routine daily log must never delay an injury report on reconnect |
| Confidence-band routing | Score each record 0.92 auto-route, 0.75–0.92 review, <0.75 quarantine |
Stale or skewed records get a human, not a silent auto-post |
| Bounded retention | Cap queue size and age; quarantine never auto-purges | A device offline for a month cannot exhaust local storage |
Two rules carry the most weight on a disconnected device. Monotonic capture time is the one most teams overlook: a field tablet whose clock has drifted (or reset to epoch after a dead battery) will stamp every queued record with a wrong time, so on reconnect the gateway routes a $40,000 change order against the wrong billing period. Record the skew at capture and let it drive the confidence score. And idempotency is non-negotiable, because the failure mode here is not a single outage but flapping — the radio oscillates between one and two bars, the dispatch loop fires repeatedly, and without a deterministic key the same safety incident is filed three times.
Production code example
The module below is a self-contained on-device router. It validates each field record with a Pydantic v2 model (Literal-typed document and discipline codes, regex-checked WBS element and MasterFormat division), persists it to a WAL-mode SQLite queue, and on reconnect assigns a routing confidence that follows the pipeline’s canonical bands — 0.92 and above auto-routes, 0.75–0.92 goes to human review, and anything below 0.75 (or any safety incident) is quarantined for a person to clear. It is the same confidence contract the schema validation rules gates apply at the gateway, pushed down to the edge.
from __future__ import annotations
import hashlib
import json
import logging
import sqlite3
from contextlib import contextmanager
from datetime import datetime, timezone
from typing import Any, Iterator, Literal
from pydantic import BaseModel, Field, field_validator
logger = logging.getLogger("construction.field_fallback")
DocType = Literal["change_order", "rfi", "daily_log", "submittal", "safety_incident"]
Discipline = Literal["ARCH", "STR", "MEP", "CIV", "ELEC", "PLMB"]
SchemaVersion = Literal["1.0", "1.1", "2.0"]
RoutingAction = Literal["auto_route", "human_review", "quarantine"]
# WBS elements follow PROJ-NNN-DIV-NN; MasterFormat divisions are XX XX XX.
_WBS = r"^[A-Z]{2,5}-\d{3}-\d{2}-\d{2}$"
_MASTERFORMAT = r"^\d{2} \d{2} \d{2}$"
# A payload below this confidence is never auto-posted on reconnect.
AUTO_ROUTE = 0.92
REVIEW_FLOOR = 0.75
# Beyond this skew the device clock is untrustworthy; force review.
MAX_TRUSTED_SKEW_SECONDS = 120
class FieldRecord(BaseModel):
"""A field-captured payload staged for routing once connectivity returns."""
model_config = {"frozen": True, "extra": "forbid"}
device_id: str = Field(min_length=3, max_length=48)
doc_type: DocType
discipline: Discipline
wbs_element: str = Field(pattern=_WBS)
masterformat_division: str = Field(pattern=_MASTERFORMAT)
budget_impact_usd: float = Field(ge=0.0)
schema_version: SchemaVersion
captured_at: datetime
clock_skew_seconds: float = Field(ge=0.0, default=0.0)
body: dict[str, Any] = Field(default_factory=dict)
@field_validator("captured_at")
@classmethod
def _require_utc(cls, v: datetime) -> datetime:
# A naive timestamp from an offline device routes to the wrong day.
if v.tzinfo is None:
raise ValueError("captured_at must be timezone-aware UTC")
return v.astimezone(timezone.utc)
@property
def idempotency_key(self) -> str:
# Deterministic: flapping replays collapse to one server-side record.
seed = f"{self.device_id}:{self.captured_at.isoformat()}:{self.doc_type}"
return hashlib.sha256(seed.encode()).hexdigest()
def routing(self, current_schema: SchemaVersion) -> tuple[RoutingAction, float, str]:
"""Score the record against the canonical confidence bands."""
confidence = 1.0
notes: list[str] = []
if self.clock_skew_seconds > MAX_TRUSTED_SKEW_SECONDS:
confidence -= 0.30
notes.append(f"clock skew {self.clock_skew_seconds:.0f}s")
if self.schema_version != current_schema:
confidence -= 0.20
notes.append(f"schema {self.schema_version} != {current_schema}")
detail = "; ".join(notes) or "clean"
# Safety incidents are escalated to a human regardless of score.
if self.doc_type == "safety_incident":
return "quarantine", confidence, "safety_incident: human review required"
if confidence >= AUTO_ROUTE:
return "auto_route", confidence, detail
if confidence >= REVIEW_FLOOR:
return "human_review", confidence, detail
return "quarantine", confidence, detail
class FieldQueue:
"""Durable, replay-safe on-device queue backed by WAL-mode SQLite."""
def __init__(self, db_path: str) -> None:
self.db_path = db_path
with self._conn() as c:
c.execute("PRAGMA journal_mode=WAL") # survive crashes mid-write
c.execute(
"""CREATE TABLE IF NOT EXISTS queue (
key TEXT PRIMARY KEY,
priority INTEGER NOT NULL,
captured_at TEXT NOT NULL,
record TEXT NOT NULL,
synced INTEGER NOT NULL DEFAULT 0
)"""
)
@contextmanager
def _conn(self) -> Iterator[sqlite3.Connection]:
conn = sqlite3.connect(self.db_path)
try:
yield conn
conn.commit()
except Exception:
conn.rollback()
raise
finally:
conn.close()
def enqueue(self, record: FieldRecord) -> str:
# Safety incidents (0) preempt change orders (1) preempt everything else (2).
priority = 0 if record.doc_type == "safety_incident" else (
1 if record.doc_type == "change_order" and record.budget_impact_usd > 50_000 else 2
)
with self._conn() as c:
c.execute(
"INSERT OR IGNORE INTO queue (key, priority, captured_at, record) "
"VALUES (?, ?, ?, ?)",
(record.idempotency_key, priority,
record.captured_at.isoformat(), record.model_dump_json()),
)
logger.info("queued %s key=%s", record.doc_type, record.idempotency_key[:12])
return record.idempotency_key
def drain(self, current_schema: SchemaVersion) -> list[dict[str, Any]]:
"""On reconnect: route pending records by priority, then capture order."""
results: list[dict[str, Any]] = []
with self._conn() as c:
rows = c.execute(
"SELECT key, record FROM queue WHERE synced = 0 "
"ORDER BY priority ASC, captured_at ASC"
).fetchall()
for key, raw in rows:
record = FieldRecord.model_validate_json(raw)
action, confidence, detail = record.routing(current_schema)
if action == "auto_route":
# Real dispatch goes here; the key is the server dedupe token.
c.execute("UPDATE queue SET synced = 1 WHERE key = ?", (key,))
logger.info("auto_route %s conf=%.2f", key[:12], confidence)
else:
logger.warning("%s %s conf=%.2f (%s)", action, key[:12], confidence, detail)
results.append({"key": key, "action": action,
"confidence": round(confidence, 2), "detail": detail})
return resultsCommon mistakes and gotchas
- Trusting the device clock. An offline tablet whose battery died comes back with a clock at epoch or hours adrift, and a naive
datetime.now()stamp routes the record to the wrong billing period or sort position. Always captureclock_skew_secondsagainst a server time delta at sync and let it pull the confidence below0.92so a human confirms the date before it posts — the same naive-timestamp failure that breaks routing across time zones at the gateway. - Queuing without an idempotency key. Network flapping is the normal case, not the exception: the radio reconnects, the dispatch loop fires, and the connection drops again mid-POST. Without the deterministic
idempotency_key, the retry files a second safety incident or a duplicate change order. TheINSERT OR IGNOREon the hash collapses replays locally, and the same key becomes the server’s dedupe token. This is the device-side mirror of the retry discipline in error handling protocols. - Auto-purging to reclaim storage. When the bounded queue fills, deleting the oldest records to make room throws away exactly the data the device went offline to protect. Cap retention by count and age, but only ever purge records already marked
synced; quarantined and unsent records must block new captures (and warn the user) rather than be silently dropped.
Integration pointer
This queue runs at the very edge of the pipeline, on the field device itself, and hands off to the server-side fallback alert routing engine the moment connectivity returns — auto_route records flow straight to the gateway under their idempotency key, while human_review and quarantine records surface in the same reconciliation console an estimator uses for flagged cost data. The budget_impact_usd threshold that preempts a change order in the local queue should match the executive-bypass threshold defined in budget code standardization, so a payload that is urgent offline stays urgent online. And because every queued record is already a validated, versioned model, it slots cleanly into the gateway’s RFI schema design contract without a second parse — the device and the server speak the same schema, the same confidence bands, and the same audit format.
Frequently asked questions
Why SQLite WAL instead of a JSON file or in-memory queue on the device?
A field device can lose power or be force-killed by the OS at any moment, often mid-write. An in-memory queue evaporates and a flat JSON file can be left half-written and unparseable. SQLite in WAL (write-ahead logging) mode commits each record atomically and recovers cleanly after a crash, so a change order captured one second before the battery dies is still there on the next boot. It also gives you indexed priority ordering for the reconnect drain without loading the whole queue into memory.
How does the confidence score decide routing on reconnect?
Each record starts at 1.0 and loses confidence for the two risks unique to a disconnected device: a clock skew beyond the trusted window and a schema_version that no longer matches the gateway. A score of 0.92 or higher auto-routes, 0.75 to 0.92 goes to human review, and anything below 0.75 is quarantined. Safety incidents are quarantined for human review regardless of score, because an injury report is never something a heuristic should silently auto-file.
What stops network flapping from filing the same record twice?
The idempotency_key is a SHA-256 hash of device_id, captured_at, and doc_type, so an identical capture always produces an identical key. Locally, INSERT OR IGNORE means a retried enqueue is a no-op. On the wire, the same key travels with the payload as the server’s dedupe token, so even if the radio drops mid-POST and the drain loop resends, the gateway resolves both attempts to one authoritative record instead of double-booking it.
How long should records stay on the device before they sync?
Retention is bounded by count and age, not by an aggressive purge. Records marked synced can be trimmed once they exceed the retention window, but unsent and quarantined records must persist until they are delivered or cleared by a person — even if that means the device for several weeks. If the queue approaches its cap, warn the user and block new low-priority captures rather than discarding queued data, since storage pressure is never a reason to lose an unsent safety or cost record.
Related
- Fallback Alert Routing — the server-side engine this on-device queue hands off to
- WBS Mapping Strategies — the WBS element format each queued record carries
- Budget Code Standardization — where the high-cost preemption threshold is defined
- Best practices for structuring RFI JSON payloads for APIs — the envelope contract these records validate against
- Implementing retry logic for failed API document pulls — the backoff and replay discipline the reconnect drain reuses
← Back to Fallback Alert Routing