AutoForge/docs/telemetry-archive-v1.md

240 lines
9.7 KiB
Markdown

# Telemetry archive schema v1 and recording
## Scope and trust boundary
Recording captures validated diagnostic operations from one explicitly opened
synthetic session. It does not read the bounded 200-entry UI trace, poll a
dashboard, issue commands, step dynamics or decode guessed payloads. Captures
are held in memory until the user stops and explicitly saves them. The saved
file is local and source-sensitive; it is not a trusted statement about a real
vehicle, manufacturer data, current vehicle state or cryptographic authenticity.
Saved-data playback is a separate [archive-only replay feature](telemetry-replay.md).
Live disk streaming, unlimited retention and crash recovery are not implemented.
The shared panel is available in both `/diagnostics` (synthetic protocol v1)
and `/simulation` (synthetic protocol v2). Hiding or unmounting a dashboard or
recording panel does not own the capture or cause additional reads. Display-unit
changes affect presentation only; archives preserve source units and source
values.
## Capture lifecycle and clocks
Start attaches to the explicit session's actual-operation observation port.
Earlier trace history is excluded; Start and Stop do not operate the source.
Each observation is published after its validated trace entry is committed,
with that operation's immutable raw trace fields and, only for a successful
`read.signals`, the decoded snapshot from that same response. Exchange ordering
defines capture sequence; model time is not used to sort or group operations.
Stop establishes a cutover by detaching the observer through the session's
serialized observation dispatcher. Callbacks already in dispatch drain before
Stop freezes the result; operations dispatched after detachment are excluded.
Repeated Start on the active owner is idempotent. A second owner for that
session is rejected. Stopped, failed, discarded and replaced owners are
unregistered from the source session's lifetime list; the workspace retains
the current stopped data needed for inspection/save. Session close/reset stops
and drains an active capture, then retains its source identity and samples for
inspection and saving. A new source does not inherit old samples. Unsaved
capture replacement requires explicit Save or Discard.
The archive keeps three independent time values:
- `captureElapsedTicks`: nonnegative, nondecreasing `TimeSpan` ticks since
Start, from `TimeProvider`.
- `startLogicalTicks` / `endLogicalTicks`: diagnostic protocol logical ticks.
- Per-sample `logicalTimeTicks`, and when present `modelStepIndex` /
`modelTimeSeconds`: the original source/model context. A paused model may
repeat model time while capture elapsed time advances.
## Fixed limits and partial captures
All sizes use compact UTF-8 JSON encoding and binary MiB (1,048,576 bytes).
| Limit | Maximum |
|---|---:|
| Captured operations | 4,096 |
| Sum of encoded observation objects | 16,777,216 bytes |
| Encoded header object | 1,048,576 bytes |
| Complete archive input/output | 33,554,432 bytes |
| JSON parser depth | 128 |
At a count or observation-byte limit, the next operation is not retained; the
valid prefix is frozen with `state: "incomplete"` and
`stopReason: "limitExceeded"`. The diagnostic source operation still returns
its original result. No older record is silently discarded and no unknown
later-loss count is inferred. The UI offers a distinctly labeled **Save partial
capture** action; ordinary Save is rejected for incomplete data. Internal
capture errors use an explicit error state rather than a healthy completion.
Oversized headers and archive inputs fail validation before publishing state.
## JSON schema
The root object has exactly these required camelCase properties:
```json
{
"schemaVersion": 1,
"header": {},
"observations": [],
"completion": {}
}
```
Unknown and duplicate schema-object properties are rejected. Nullable metadata
is explicit `null`; it is not inferred from a current profile, session or
configuration.
`header` contains exactly:
```text
captureId
sessionIdentifier
profileIdentifier
profileRevision
vehicleIdentifier
providerIdentifier
targetKind
protocolName
protocolVersion
dynamicsConfigurationIdentifier
dynamicsRunIdentifier
sourceEndpoints
startedAtUtc
```
`captureId` is a lowercase opaque GUID in `N` format and is the only filename
input. `targetKind` uses the exact camelCase enum token. `protocolName` is
`autoforge.synthetic-diagnostics`; `protocolVersion` is 1 or 2. Static v1 has
null dynamics identifiers. Dynamic v2 carries both the original configuration
and run identifiers. `sourceEndpoints` is an array of `{ "moduleIdentifier",
"address" }` identities. UTC timestamps use the complete round-trip `"O"`
format with an explicit zero offset.
Every `observations` entry contains exactly:
```text
sequence
captureElapsedTicks
moduleIdentifier
address
correlationIdentifier
service
startLogicalTicks
endLogicalTicks
outcome
decodedSummary
error
requestPayloadBase64
responsePayloadBase64
signalSnapshot
```
`sequence` starts at 1 and is contiguous. Non-null raw payloads are canonical
base64. A null response means the trace retained no response bytes; this
includes an actual empty `TransportOutcome.Response` that became a
`ProtocolError`. The archive preserves that representation and does not invent
placeholder bytes. Positive and negative responses require response bytes;
timeout, no-response and cancellation outcomes prohibit them. A `ProtocolError`
may preserve malformed response bytes or have null bytes for an empty response.
The raw synthetic request and any positive/negative response are checked
against the recorded session, profile, configuration, correlation, endpoint,
service, protocol version and outcome. Identity fields use the source client's
existing comparison rules: selected response-envelope and module identities
are case-insensitive, while exact protocol tokens, signal IDs, units, payload
bytes and model/configuration values remain exact.
`signalSnapshot` is null except for a successful `read.signals`. It contains:
```text
moduleIdentifier
signals
modelStepIndex
modelTimeSeconds
modelConfigurationIdentifier
modelRunIdentifier
```
Each `signals` item contains:
```text
signalIdentifier
moduleIdentifier
value
unit
quality
logicalTimeTicks
origin
modelStepIndex
modelTimeSeconds
modelConfigurationIdentifier
modelRunIdentifier
```
`value` remains a JSON value with its original JSON kind and data; arbitrary
properties inside that value are not interpreted as archive schema. Unit,
quality, origin, diagnostic time and optional model/run context are source
metadata, not display-unit projections.
Protocol v1 retains its nullable, non-numeric and otherwise opaque source
values. Protocol v2 uses the existing source contract: known signal IDs, exact
units and JSON kinds, finite numeric values, unique signal IDs within the
snapshot and nonnegative model time. Raw and decoded v2 values are both checked
against that contract and against each other.
`completion` contains exactly:
```text
stoppedAtUtc
stopReason
state
observationCount
observationBytes
error
```
`observationCount` and `observationBytes` must match the validated list and its
actual compact encoding. A stopped healthy capture uses `state: "complete"`
and stop reason `"user"` or `"sourceClosed"`. Limit exhaustion is
`"incomplete"` / `"limitExceeded"`. A capture implementation error is
`"error"` / `"observationFailure"` with a visible error message.
## Local persistence and failure handling
The Client composes `FileTelemetryArchiveStore` under
`FileSystem.AppDataDirectory/recordings`. Save is available only after Stop.
The store validates and serializes the whole capture, writes a capture-specific
temporary file, flushes it to disk, closes it, then publishes using a
no-overwrite move. A collision, cancellation, or write failure cannot replace
an existing archive. Temporary-file cleanup failures are surfaced as typed
write failures with an explicit instruction to inspect the named temporary
file; they are not silently hidden.
List and Load enforce the 32 MiB input bound and validate the full schema.
Corrupt matching files, unsupported versions, malformed/truncated JSON,
unknown or duplicate schema properties, bad base64, inconsistent counts,
times, outcomes or source context are visible typed failures. A missing
first-use directory lists as empty; an existing file used as the storage root
or another listing failure is a typed visible error, not an empty catalogue.
The panel refreshes after a successful save only, preserving a failed save's
error and unsaved state. Invalid data does not replace an existing inspection.
Valid incomplete archives remain incomplete on readback. Readback proves
file/schema consistency only; it does not validate authenticity or source
truth.
The `/telemetry-replay` route lists and explicitly loads these same files using
the strict reader. Playback never calls Save, rebinds the recording workspace,
changes original completion or uses the original profile/model as a fallback.
Current RecordedReplay identity is separate from the archived source identity.
## Verification
The focused `TelemetryRecordingIntegrationTests` exercise real v1/v2 sessions,
more than 200 operations, all observed outcomes (including an actual empty
reply), case-variant source identities, coherent v2 archive edits, controlled
capture time, deterministic owner release, Stop draining, session reset, exact
count/observation/header/archive boundaries, filesystem round-trip,
collision/cancellation/I/O failures, corrupt-file rejection and actual panel
save/retry behavior. The original implementation evidence is preserved in the
[Step-07 worker report](../prompts/Step-07-Report.md); corrective execution
evidence and manual-review limitations are recorded in the
[Step-07A worker report](../prompts/Step-07A-Report.md).