305 lines
20 KiB
Markdown
305 lines
20 KiB
Markdown
# Step-13 - Verified synthetic calibration comparison and demonstration
|
|
|
|
## STEP IDENTIFIER
|
|
|
|
Step-13. Step-12 with Step-12A accepted 2026-10-31.
|
|
This is the only active implementation assignment; Step-14 remains blocked.
|
|
Read [STATE.md](STATE.md), [ROADMAP.md](ROADMAP.md), the
|
|
[accepted execution report](Step-12A-Execution-Report.md) and
|
|
[WORKER-QUALITY-GATE.md](WORKER-QUALITY-GATE.md).
|
|
Establish the executable matrix before coding, self-review after first green,
|
|
save `prompts/Step-13-Report.md` in one plain fenced report and STOP.
|
|
|
|
## STEP TITLE
|
|
|
|
Compare equivalent real synthetic runs using original configuration and an
|
|
explicit NEW configuration derived from independently verified rehearsal values.
|
|
|
|
## OBJECTIVE
|
|
|
|
One user outcome: an auditable before/after synthetic calibration comparison,
|
|
with real captures/replay and clear separation of proposal, programmed-image
|
|
verification and observed model behavior.
|
|
One primary new contract: trusted verified-image admission and exact immutable
|
|
projection into an equivalent two-run comparison, never mutation of existing
|
|
configuration/run or implicit programming authority.
|
|
|
|
No hardware, general executor, new protocol/archive schema or runtime manager.
|
|
Use existing local AI/manual authoring/rehearsal/diagnostic/recording/replay
|
|
workflows; do not implement parallel versions of those services.
|
|
|
|
## ARCHITECTURAL CONTEXT
|
|
|
|
Keep all nine projects, one Client and independent Components.
|
|
Core may own the narrow pure supported-values/configuration projection;
|
|
Application owns trusted source admission, comparison/metrics/capture/save
|
|
workflow; UI owns explicit draft/evaluation/report controls and lifetime.
|
|
Use Diagnostics ports and explicit Simulation provider, not an Application/UI
|
|
reference to Simulation implementations or privileged access to DynamicsEngine.
|
|
|
|
Rehearsal Apply continues to change only the Simulation-owned supported logical
|
|
image. Step-13 introduces a separately named explicit projection to a NEW
|
|
immutable configuration. Original configuration, authoring binding/history,
|
|
target approvals and existing runs are never overwritten or silently rebased.
|
|
Replay remains archive playback, not rerunning physics.
|
|
|
|
## CURRENT IMPLEMENTATION STATE
|
|
|
|
- Accepted baseline: 179 unit + 417 integration = 596 passes, zero failed/
|
|
skipped; clean nine-project Release/native evidence from the worker.
|
|
- `/runbooks` validates original local model content, exact immutable preview/
|
|
diff/current-Stale source, and grants no Save/approval/target/model authority.
|
|
- Authoring supports exact source/head scalar/fixed-axis table proposals.
|
|
Rehearsal supports actual two-read backup, separately approved single-use CAS,
|
|
complete readback/reconciliation/Restore and bounded audit.
|
|
- Rehearsal snapshots contain verified backup/current image/audit results, but
|
|
public records alone are not unforgeable verified-source admission.
|
|
- Dynamics config has immutable WarmIdleTargetRpm/EngineTorqueCurve, initial
|
|
conditions and full five-phase scenario/fingerprint. Existing diagnostics/v2
|
|
exposes actual engine/speed/fuel/distance and transmission shift-count signals.
|
|
- Archive v1 capture/store/save/readback and exact replay are accepted.
|
|
- No comparison, trusted verified-image projection or complete connected
|
|
before/after demonstration exists yet.
|
|
- Local-only Ollama/model is already qualified. No new pull, startup/restart/
|
|
config change or approval form for that existing setup is assigned.
|
|
|
|
Use [authoring](../docs/calibration-authoring.md),
|
|
[rehearsal](../docs/calibration-rehearsal.md),
|
|
[runbooks](../docs/runbooks-v1.md) and
|
|
[local generation](../docs/local-runbook-generation.md).
|
|
|
|
## REQUIRED IMPLEMENTATION
|
|
|
|
### Frozen contract matrix
|
|
|
|
| ID | Precondition/action | Required result |
|
|
|---|---|---|
|
|
| C-01 | Explicit current preview-to-draft action versus invalid/stale/superseded/different source | Exact copied canonical candidate enters the existing pinned authoring draft only; no Save, target call, rebase or automatic action on render |
|
|
| C-02 | Current owning rehearsal with original verified backup and verified Apply/full actual readback | Trusted copied evidence admits comparison: exact target instance/revision/action/operation/backup/workspace/head/config/definition and complete values; caller-constructed success records are insufficient |
|
|
| C-03 | Exact supported scalar/table mapping into original config | NEW immutable config changes only WarmIdleTargetRpm and torque cells; axes/all other parameters/initial state/scenario/profile/module/model refs unchanged; explicit lineage and fingerprint |
|
|
| C-04 | Fresh baseline/derived runs and fixed bounded acquisition | Same explicit real Simulation/v2 provider/scenario/initial state, separate run IDs and actual canonical samples/captures; no replay/oracle/canned telemetry or diagnostics-triggered physics |
|
|
| C-05 | Decisive warm baseline, real model proposal, explicit Save/backup/approval/Apply/readback | 650->1250 rpm and 2000-rpm torque 360->350 N*m; derived mapping exact, actual last ten warm-idle samples within 1 rpm of each target, stationary; all unrelated config/image values unchanged |
|
|
| C-06 | Complete equivalent run comparison/report and existing save/replay | Actual idle/speed/fuel/distance/time/shift metrics, exact configuration/proof/run/capture lineage, synthetic limits; explicit bounded saves/readback and real archive replay preserve canonical evidence |
|
|
| C-07 | Busy, cancellation, close/unmount and source/head/proof supersession during pending work | One admitted owner, responsive Hybrid events, drain outside required gates, no late healthy result/rebase; previous success inspectable with truthful stale/partial status |
|
|
| C-08 | Missing/invalid proof/config/signals, failure/limits or partially successful pair save | Typed visible rejection before side effects where applicable; incomplete/faulted run is not a successful comparison; exact persisted/unsaved IDs and no overwrite/compensating deletion |
|
|
| C-09 | Independent verification and supported fault regression matrix | Test-only independent whole-config/image/mapping/numerical assertions and actual producer/consumer faults; no UI/AI access to a private oracle or invented manufacturer checksum/firmware behavior |
|
|
| C-10 | Real native complete demonstration and preserved behavior | All 596 cases/new focused coverage/nine-project Release build pass, actual local-model-to-verified-image-to-run/capture/replay path and no physical/unassigned mutation |
|
|
|
|
### A. Explicit preview handoff and trusted verified source
|
|
|
|
Add an explicit current-preview-to-existing-draft action. Application validates
|
|
actual workspace/head/applicability and expected preview identity at admission;
|
|
copy the complete canonical candidate. The existing draft pins display units;
|
|
no string/rounded-display reconstruction. A user must still explicitly Save
|
|
the Proposed version through existing validation/head checks.
|
|
No Save/approval/execution on AI validation, navigation, notification or render.
|
|
Rejected handoff leaves prior draft/head/source/preview unchanged and shows error.
|
|
|
|
Use existing rehearsal commands for backup/preparation/approval/CAS/readback.
|
|
At an explicit evaluation admission, the actual owning rehearsal must be idle/
|
|
Ready with a verified original backup and selected current verified Apply
|
|
result (including genuinely verified post-reconciliation Apply).
|
|
Match exact current identity/revision/image, candidate, saved workspace/head/
|
|
applicability and original configuration. Reject RecoveryRequired, missing/
|
|
divergent/unavailable/stale/closed evidence or forged caller result flags.
|
|
|
|
Use a narrow owning-service admission/helper with copied immutable evidence,
|
|
not a public constructor accepting arbitrary records and declaring them verified.
|
|
Bind original backup values to original supported config values and selected
|
|
saved candidate to actual readback in full, changed AND unchanged values.
|
|
Preserve target/programming semantics. Any required readonly admission addition
|
|
does not open/approve/write a target or bypass its owner/generation guards.
|
|
Captured evidence is historical proof at a stated time, not perpetual target
|
|
state; supersession must be detected/labeled before current-result publication.
|
|
|
|
### B. Exact NEW configuration and two equivalent real runs
|
|
|
|
Implement one pure supported projection using existing constructor/validation/
|
|
fingerprint logic. Map only:
|
|
|
|
- `engine.warm-idle-target` rpm -> Model.WarmIdleTargetRpm.
|
|
- Fixed `engine.torque-curve` rpm/N*m axes/cells -> Model.EngineTorqueCurve.
|
|
|
|
Validate exact definition/units/axis/bounds first. Copy every other model field,
|
|
initial condition, five-phase scenario and identity/reference unchanged.
|
|
Register/select the NEW config through existing validation/catalogue logic,
|
|
reusing/extracting helpers rather than JSON string replacement or duplicating
|
|
an importer. Do not modify either the original config or an existing run.
|
|
Show original/derived fingerprints and explicit source/proof/value lineage.
|
|
A no-change supported mapping is a truthful no-change outcome, not fake benefit.
|
|
|
|
One comparison admits immutable original/derived pair and runs two fresh owned
|
|
diagnostic dynamics sessions sequentially with explicit Simulation/v2 provider.
|
|
Use real control StepAsync and validated signal replies; no private engine
|
|
sample list, test oracle, model-generated telemetry or replay as physics.
|
|
Capture phase before advancing a step where needed for boundary attribution.
|
|
|
|
Fixed limits/cadence for this slice: at most 3900 model steps per run; engine
|
|
AND transmission signal reads every five completed steps and at completion
|
|
(avoid duplicate final reads). The decisive 0.1-second warm configuration is
|
|
therefore sampled every 0.5 virtual seconds. Capture those actual observations
|
|
through existing recording services, inside unchanged 4096-observation/byte/
|
|
archive limits; at most 1562 scheduled signal observations per run.
|
|
Advance/control phase counts at each step, but do not add unrecorded/private
|
|
telemetry or per-step polling that exhausts archive byte limits.
|
|
A run needing step 3901 rejects as incomplete,
|
|
not Completed or a reason to enlarge existing limits.
|
|
Retain one current pair/last successful result only, with bounded immutable
|
|
samples/metrics/captures; no hidden history or unbounded collection.
|
|
|
|
Ordinary manual diagnostic/capture/replay owners are not taken over or cleared.
|
|
Reuse independent accepted capture/store APIs for the pair; do not silently
|
|
replace another recording workspace or discard an unsaved first capture when
|
|
the second begins. Extract a narrow shared helper only if needed.
|
|
|
|
### C. Comparison, save/readback/replay and truthful lifecycle
|
|
|
|
Compare actual canonical data: last-ten warm-idle RPM and stationary speed,
|
|
warm phase, acceleration/braking elapsed model time, sampled cruise speed
|
|
range/count (separate from total cruise model-step count),
|
|
fuel consumed delta from initial state, distance, total model time and final
|
|
actual transmission.shift-count. Distinguish fixed-cadence shift sampling from
|
|
the measured final count; do not infer an exact full shift timeline.
|
|
Missing/non-Good/nonfinite/wrong-unit/context replies cannot become zero or a
|
|
healthy complete metric. Failed/limited/cancelled runs retain bounded partial
|
|
evidence and visible reason, not a complete comparison.
|
|
Do not promise reduced fuel, better shifts or real-world improvement.
|
|
|
|
For the exact demo, compare 650 rpm baseline with 1250 rpm candidate and torque
|
|
at 2000 rpm 360->350 N*m. Last ten warm-phase samples on each run must be within
|
|
1 rpm of its target and vehicle speed within 0.001 m/s of zero.
|
|
Fewer than ten valid warm samples is an explicit incomplete/unsupported case,
|
|
not a shortened passing window. Label sampled metrics with their cadence;
|
|
do not claim continuous extrema from sparse observations.
|
|
Report measured driving/fuel/shift differences without requiring an invented
|
|
sign. Keep all original full-scenario numerical regressions, including baseline
|
|
3361 samples/336.1 s and 3000 cruise samples within 45 MPH +/-1 MPH.
|
|
|
|
Report separate proposal/diff, backup/approved mutation/full readback and
|
|
functional simulation evidence. Include exact source/head/definition/model,
|
|
target instance/revision/operation/backup, original/derived configs, provider/
|
|
run IDs, capture IDs/completion and canonical before/after values/metrics.
|
|
Provide a safe copyable readonly report; no new persistence/schema/export engine.
|
|
|
|
Save is explicit, reusing archive v1/atomic no-overwrite publication/readback.
|
|
Save each capture with visible per-ID outcome. If one succeeds and the other
|
|
fails, preserve the successful file and unsaved capture, identify partial save
|
|
and retry only missing work; no pair-atomic claim, rollback deletion or fake
|
|
all-saved state. Open saved results through actual existing replay and raw trace.
|
|
Units project display only; source/archive/report canonical values stay exact.
|
|
|
|
Use one cancellable owner, generation/source/proof admission and publication
|
|
checks. Queue only admitted blocking work and release Hybrid command events
|
|
as accepted in Step-12A. Close/unmount drains owned diagnostic/capture resources
|
|
without holding required gates, late-result publication or global interference.
|
|
Invalid competing requests reject before starting/mutating. Retained earlier
|
|
results remain inspectable, with clear stale/as-of status.
|
|
|
|
## FILES OR PROJECTS EXPECTED TO CHANGE
|
|
|
|
Narrow Core projection if needed, Application evaluation/trusted admission/
|
|
shared-helper reuse, calibration/evaluation UI and DI, tests and related docs/
|
|
README. Only `prompts/Step-13-Report.md` under `prompts`.
|
|
No new project/package/host, archive/runbook/profile format, runtime install/
|
|
model pull/configuration change or unrelated recording navigation cleanup.
|
|
|
|
## INTERFACE AND DEPENDENCY REQUIREMENTS
|
|
|
|
Preserve nine references, independent Components, typed Diagnostics ports,
|
|
immutable original configs/authoring/runbooks, volatile backup/approval/CAS/
|
|
readback/Restore, archive limits/version and replay metadata semantics.
|
|
No general runbook executor, physical programming, arbitrary frames/offsets,
|
|
cloud fallback, Git backup, new templates/unit families or privileged oracle.
|
|
The model supplies untrusted preview data only; neither proof nor authority.
|
|
No agents, automatic handoff, staging/commit/push or orchestration/shared-gate
|
|
changes. Step-12A's setup authorization is not recurring runtime management.
|
|
|
|
## TESTING REQUIREMENTS
|
|
|
|
Map C-01..C-10 before implementation. Verify exact normal/rejected/boundary/
|
|
lifecycle/output cases, not only counts:
|
|
|
|
- Actual current preview -> draft -> explicit Save, stale/superseded/wrong
|
|
source and pinned units. No extra source/target calls on rejected handoff.
|
|
- Actual owning rehearsal baseline/two-read backup/approval/CAS/full readback
|
|
-> trusted evaluation source. Forged flags, wrong instance/revision/source/
|
|
candidate/backup, recovery/divergence and supersession reject atomically.
|
|
- Independent whole-config serialized/property comparison proves only the
|
|
supported warm-idle/torque fields change; originals remain byte/content exact.
|
|
Nonuniform table/cell changes and invalid/nonfinite/unit/axis/boundaries.
|
|
- Real two-run v2/recording/store/replay boundary and independent test-only
|
|
numerical expectations, last-ten idle thresholds, exact baseline scenario
|
|
and fresh equivalent initial state. Independent tests retain verification of
|
|
all 3000 cruise model steps; production captures/report explicitly distinguish
|
|
sampled signal counts/ranges from full model-step counts. No private oracle
|
|
exposed to application/UI/AI.
|
|
- Controlled actual owner/transport/save/readback delays and cancel/unmount/
|
|
late/superseded/busy failure paths; 3900/3901 limit and unchanged archive caps.
|
|
Partial saves/readback corruption and wrong archive/config/replay references.
|
|
- Supported earlier timeout/unsupported service/invalid wire/wrong identity/
|
|
version/config/interruption/reset/DTC/recovery/reconciliation/Restore/archive
|
|
corruption tests remain; name executed evidence. No invented firmware checksum
|
|
or partial physical-write mechanism merely to fill a fault table.
|
|
|
|
Release test barriers in finally; no sleeps/polling as timing correctness proof.
|
|
Run focused combined selectors, then sequentially:
|
|
|
|
- `dotnet build .\src\AutoForge.slnx --configuration Release --no-restore`
|
|
- `dotnet test .\src\AutoForge.slnx --configuration Release --no-build`
|
|
|
|
Preserve all 596 cases/full nine-project/native build. Restore code dependencies
|
|
only after genuine chosen-command missing dependency; no SDK/workload/model
|
|
install/update/repair or alternate host.
|
|
|
|
Required actual native demonstration: original warm run/capture; exact local
|
|
model goal; current preview handoff and separate Proposed Save; actual Simulation
|
|
target backup/approval/Apply/fresh full readback; explicit trusted projection/
|
|
equivalent baseline-derived pair; real measured behavior/report/capture saves/
|
|
readback/replay/raw trace; failure/cancel/stale/units and unchanged original
|
|
configuration/history outside deliberate commands. Existing routes still work.
|
|
Record exact source/config/run/capture/proof IDs, model original response and
|
|
measurable outcomes. Controlled responses are not native model proof.
|
|
No physical target or unrestricted operation; cleanup only exact task-owned
|
|
native helpers/archives/profiles/PIDs, preserving user data/runtime/model.
|
|
|
|
## DOCUMENTATION REQUIREMENTS
|
|
|
|
Align README/index/architecture/build/authoring/rehearsal/runbook/evaluation
|
|
usage with explicit proposal handoff, trusted verified-source admission,
|
|
NEW-config projection, equivalent fixed-cadence runs, archive bounds, canonical
|
|
metrics/lineage, partial save, stale/cancel/close and synthetic fidelity limits.
|
|
Step-12/12A accepted; Step-13 pending manual review; hardware/firmware checksum/
|
|
table shift schedules/broader AI evaluation remain future.
|
|
|
|
## ACCEPTANCE CRITERIA
|
|
|
|
| ID | Criterion | Required evidence |
|
|
|---|---|---|
|
|
| AC-01 | C-01..C-03 explicit draft/proof admission and exact immutable projection | Actual owning workflows, independent full-value/config invariants and atomic negative/supersession tests |
|
|
| AC-02 | C-04..C-06 equivalent real scenarios/captures/measured intent/report/save/replay | Exact idle thresholds/baseline driving gates, actual canonical samples/metrics/lineage and real archive readback/playback |
|
|
| AC-03 | C-07..C-09 responsive safe owner/failures/limits and independent verification | Controlled real-boundary interleavings/3900 boundary/partial saves/corruption, preserved no-side-effect/isolation and supported fault evidence |
|
|
| AC-04 | C-10 full 596 regressions/new coverage/nine-project Release and actual native demo pass | Exact worker commands/exits/counts/numerical results, original real model content, proof and captures/replay; no mock-as-native claims |
|
|
| AC-05 | Matrix/self-review/docs/history/protection/manual gate accurate | Own post-green producer/consumer review, aligned scope/fidelity and `prompts/Step-13-Report.md` |
|
|
|
|
## OUT-OF-SCOPE WORK
|
|
|
|
Physical writes/read-only hardware onboarding, broad AI/RAG/tools/executors,
|
|
runtime/model installation/management, firmware layouts/checksums, dynamics
|
|
fidelity redesign, changed source configs/runs, new archive/report persistence/
|
|
Git backups, generic evaluation frameworks, new projects/packages/hosts and
|
|
unrelated fixes. No Step-14, agents or automatic handoff/commits.
|
|
|
|
## REPORTING REQUIREMENTS
|
|
|
|
Save `prompts/Step-13-Report.md` wholly inside exactly one plain triple-backtick
|
|
fence, no language tag/outside text/nested fences.
|
|
Include identifier/title/status/functions/files/interfaces; C-01..C-10/
|
|
AC-01..AC-05 named executed matrix; exact preview/draft/save/proof/projection
|
|
before/after and unchanged config fields; actual equivalent numerical sample/
|
|
metric/run/capture/lineage/save/readback/replay outcomes; owner/negative/limit/
|
|
partial failures; commands/exits/warnings/per-project counts; native versus
|
|
controlled evidence and original model content; docs/self-review/deviations/
|
|
limitations/blockers/preservation/cleanup. No secrets/private thinking traces.
|
|
COMPLETE requires all assigned checks, otherwise INCOMPLETE/BLOCKED with exact
|
|
unmet criterion. Save and STOP for manual review; do not begin Step-14.
|