Best practices for structuring RFI JSON payloads for APIs

This page covers exactly one design decision: how to shape the JSON payload that carries a Request for Information (RFI) across an API boundary so that a construction automation pipeline can parse and route it deterministically. The payload is the wire contract — the literal bytes a subcontractor portal, a field-capture app, or an upstream ERP sends to your ingestion gateway. When that contract is loose, the failure is rarely a loud crash; it is silent data loss: an attachment array that arrives nested two levels deeper than expected, a status string nobody anticipated, or a naive timestamp that drifts an SLA window by five hours. A rigid, explicitly typed envelope eliminates that ambiguity at the boundary so downstream cost tracking, approval routing, and field coordination run without manual reconciliation. The payload here is the over-the-wire implementation of the typed model defined in RFI schema design, and it lives inside the broader construction data architecture and taxonomy that keeps field definitions consistent across platforms.

Key rules and specification

Treat the payload as a flat, forward-compatible envelope: immutable identity at the root, references to external entities by code rather than embedded objects, and every field explicitly typed. A deeply nested payload that inlines the full project manifest or a user record forces recursive parsing, invites circular-reference errors, and bloats serialization for no downstream benefit. The constraints below are the minimum contract every inbound RFI must satisfy.

Three rules carry the most weight. Pin a root schema_version so a field addition is a backward-compatible bump rather than a silent break. Keep nesting to a single level — metadata arrays such as attachments, tags, and assignees live at the root, and anything richer is referenced by code, not inlined. And forbid unexpected keys, because that single rule is what stops a misbehaving producer from smuggling an unvalidated field into your tracking database.

Production payload contract and validation

Validate the envelope at the gateway with a type-aware model, not with dictionary key lookups. The snippet below defines the canonical Pydantic v2 contract, coerces timestamps to timezone-aware UTC at the boundary, aggregates every field violation for precise debugging, and routes by the site-canonical confidence bands — at or above 0.92 the record auto-routes, 0.75–0.92 is held for human review, and anything below 0.75 is quarantined to a dead-letter queue rather than committed against a guessed scope. A structural validation failure overrides confidence entirely: a malformed payload is never safe to auto-route no matter how sure the producer was.

from __future__ import annotations

import json
import logging
import uuid
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from typing import Any, Literal

from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator

logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")
logger = logging.getLogger("rfi_payload")

# Controlled vocabularies travel as enums, never free strings.
Discipline = Literal["ARCH", "STR", "MEP", "CIV", "ELEC", "PLMB"]
RFIStatus = Literal["DRAFT", "OPEN", "PENDING_RESPONSE", "ANSWERED", "CLOSED", "REJECTED"]
RoutingState = Literal["AUTO_ROUTE", "HUMAN_REVIEW", "QUARANTINE"]

# Site-canonical routing bands, applied to producer/extraction confidence.
AUTO_ROUTE = 0.92
HUMAN_REVIEW = 0.75


class RFIEnvelope(BaseModel):
    """The flat, forward-compatible RFI wire contract."""

    model_config = ConfigDict(strict=True, extra="forbid")  # reject unexpected keys

    schema_version: str = Field(pattern=r"^v\d+\.\d+$")
    rfi_number: str = Field(pattern=r"^RFI-\d{4}-\d{3,5}$")
    project_uuid: uuid.UUID
    created_at: datetime
    discipline: Discipline
    status: RFIStatus
    description: str = Field(min_length=10, max_length=4000)
    # Canonical WBS element (PROJ-NNN-DIV-NN) and budget code, referenced not inlined.
    wbs_element: str | None = Field(default=None, pattern=r"^[A-Z]{3,5}-\d{3}-\d{2}-\d{2}$")
    budget_code: str | None = Field(default=None, pattern=r"^\d{6}[A-Z]{3}\d{3}$")
    cost_impact: Decimal | None = Field(default=None, ge=0, decimal_places=2)
    # Single-level metadata arrays, bounded to prevent payload bloat.
    attachments: list[str] = Field(default_factory=list, max_length=25)

    @field_validator("created_at", mode="before")
    @classmethod
    def normalize_utc(cls, v: Any) -> Any:
        """Coerce ISO 8601 strings to timezone-aware UTC; reject naive datetimes."""
        if isinstance(v, str):
            dt = datetime.fromisoformat(v.replace("Z", "+00:00"))
            if dt.tzinfo is None:
                raise ValueError("created_at must carry a timezone offset (ISO 8601)")
            return dt.astimezone(timezone.utc)
        if isinstance(v, datetime) and v.tzinfo is None:
            raise ValueError("created_at must be timezone-aware")
        return v

    @field_validator("cost_impact", mode="before")
    @classmethod
    def coerce_decimal(cls, v: Any) -> Any:
        """Coerce to Decimal at the boundary; never let a float drift a cost rollup."""
        if v is None or isinstance(v, Decimal):
            return v
        try:
            return Decimal(str(v))
        except InvalidOperation as exc:
            raise ValueError(f"invalid cost_impact: {v!r}") from exc


def parse_and_route(raw_json: str, producer_confidence: float) -> tuple[dict[str, Any], RoutingState]:
    """Validate an inbound RFI payload, then route by structural validity and confidence.

    Raises ValueError on malformed JSON or a schema violation so the caller can park
    the raw bytes in the dead-letter queue instead of committing a guessed record.
    """
    try:
        payload = json.loads(raw_json)
    except json.JSONDecodeError as exc:
        logger.error("Malformed JSON at pos %s: %s", exc.pos, exc.msg)
        raise ValueError("invalid JSON syntax in RFI payload") from exc

    try:
        record = RFIEnvelope.model_validate(payload)
    except ValidationError as exc:
        # Aggregate every field-level error for one-pass debugging, not just the first.
        logger.error("Schema validation failed: %s", exc.json())
        raise ValueError("payload violates RFI schema contract") from exc

    # Structural validity gates routing: a clean record then routes by confidence.
    if producer_confidence >= AUTO_ROUTE:
        state: RoutingState = "AUTO_ROUTE"
    elif producer_confidence >= HUMAN_REVIEW:
        state = "HUMAN_REVIEW"
    else:
        state = "QUARANTINE"

    logger.info("RFI %s validated -> %s", record.rfi_number, state)
    return record.model_dump(mode="json"), state


if __name__ == "__main__":
    sample = json.dumps(
        {
            "schema_version": "v1.0",
            "rfi_number": "RFI-2026-014",
            "project_uuid": str(uuid.uuid4()),
            "created_at": "2026-06-27T14:30:00-05:00",
            "discipline": "ELEC",
            "status": "OPEN",
            "description": "Confirm conduit routing at Level 3 east core.",
            "wbs_element": "TWR-103-09-02",
            "budget_code": "260000SUB014",
            "cost_impact": "31500.00",
            "attachments": ["s3://rfi-docs/RFI-2026-014/sketch.pdf"],
        }
    )
    clean, routing = parse_and_route(sample, producer_confidence=0.95)
    print(f"{clean['rfi_number']} -> {routing}")

The envelope serializes back out with model_dump(mode="json"), which renders the UUID, datetime, and Decimal types as JSON-native strings and numbers — so the same contract is both the validator on ingest and the canonical shape on the way to the tracking ledger.

Common mistakes and gotchas

• Deeply nested envelopes that inline external entities. Embedding the full project manifest, the submitting user object, or a historical audit log inside the RFI forces recursive parsing, risks circular-reference errors during serialization, and balloons every message. Reference those entities by project_uuid or code and keep metadata arrays at the root; the consumer hydrates what it needs from its own store.
• Naive timestamps causing cross-site routing failures. A portal that emits 2026-06-27T14:30:00 with no offset looks valid but silently anchors to whatever zone the parser assumes. Across project sites in different time zones, that drifts SLA windows and can compute a negative time-to-breach. Reject naive datetimes at the boundary and normalize every created_at to UTC, as the validator above does, while preserving the original offset in the audit trail.
• Implicit type coercion on cost fields. Letting "$31,500" or a bare float flow into the cost field either bounces a valid RFI or, worse, a “helpful” float(x or 0) fallback turns an unparseable amount into 0 and understates a change-order exposure. Coerce monetary values to Decimal in a field_validator(mode="before") and let a genuinely empty value become null, never zero — this is the same discipline that keeps budget code standardization rollups exact.

Integration pointer

This payload contract is the gateway-side bookend of the ingestion pipeline. The fields validated here are the same ones that the JSON Schema validation gate re-asserts on records lifted from documents — keep the envelope and that schema in lockstep so the API gateway and the schema validation rules layer never disagree about what a valid RFI looks like. Once validated, the wbs_element routes the RFI into WBS mapping strategies for cost allocation, while quarantined and SLA-breached payloads hand off to fallback alert routing rather than blocking the gateway. When a portal pushes RFIs in bursts during a bid period, the validated envelopes feed the async batching workflows that absorb the spike without dropping records.

Frequently asked questions

Related

• RFI Schema Design — the typed model this wire contract implements
• Validating extracted RFI fields against custom JSON schemas — the document-side gate that mirrors this envelope
• WBS Mapping Strategies — where the validated wbs_element allocates cost
• Budget Code Standardization — the canonical cost vocabulary the budget_code references
• Fallback Alert Routing — where quarantined and SLA-breached payloads hand off

← Back to RFI Schema Design