AutoForge/prompts/Step-13.md

305 lines
20 KiB
Markdown

# Step-13 - Verified synthetic calibration comparison and demonstration
## STEP IDENTIFIER
Step-13. Step-12 with Step-12A accepted 2026-10-31.
This is the only active implementation assignment; Step-14 remains blocked.
Read [STATE.md](STATE.md), [ROADMAP.md](ROADMAP.md), the
[accepted execution report](Step-12A-Execution-Report.md) and
[WORKER-QUALITY-GATE.md](WORKER-QUALITY-GATE.md).
Establish the executable matrix before coding, self-review after first green,
save `prompts/Step-13-Report.md` in one plain fenced report and STOP.
## STEP TITLE
Compare equivalent real synthetic runs using original configuration and an
explicit NEW configuration derived from independently verified rehearsal values.
## OBJECTIVE
One user outcome: an auditable before/after synthetic calibration comparison,
with real captures/replay and clear separation of proposal, programmed-image
verification and observed model behavior.
One primary new contract: trusted verified-image admission and exact immutable
projection into an equivalent two-run comparison, never mutation of existing
configuration/run or implicit programming authority.
No hardware, general executor, new protocol/archive schema or runtime manager.
Use existing local AI/manual authoring/rehearsal/diagnostic/recording/replay
workflows; do not implement parallel versions of those services.
## ARCHITECTURAL CONTEXT
Keep all nine projects, one Client and independent Components.
Core may own the narrow pure supported-values/configuration projection;
Application owns trusted source admission, comparison/metrics/capture/save
workflow; UI owns explicit draft/evaluation/report controls and lifetime.
Use Diagnostics ports and explicit Simulation provider, not an Application/UI
reference to Simulation implementations or privileged access to DynamicsEngine.
Rehearsal Apply continues to change only the Simulation-owned supported logical
image. Step-13 introduces a separately named explicit projection to a NEW
immutable configuration. Original configuration, authoring binding/history,
target approvals and existing runs are never overwritten or silently rebased.
Replay remains archive playback, not rerunning physics.
## CURRENT IMPLEMENTATION STATE
- Accepted baseline: 179 unit + 417 integration = 596 passes, zero failed/
skipped; clean nine-project Release/native evidence from the worker.
- `/runbooks` validates original local model content, exact immutable preview/
diff/current-Stale source, and grants no Save/approval/target/model authority.
- Authoring supports exact source/head scalar/fixed-axis table proposals.
Rehearsal supports actual two-read backup, separately approved single-use CAS,
complete readback/reconciliation/Restore and bounded audit.
- Rehearsal snapshots contain verified backup/current image/audit results, but
public records alone are not unforgeable verified-source admission.
- Dynamics config has immutable WarmIdleTargetRpm/EngineTorqueCurve, initial
conditions and full five-phase scenario/fingerprint. Existing diagnostics/v2
exposes actual engine/speed/fuel/distance and transmission shift-count signals.
- Archive v1 capture/store/save/readback and exact replay are accepted.
- No comparison, trusted verified-image projection or complete connected
before/after demonstration exists yet.
- Local-only Ollama/model is already qualified. No new pull, startup/restart/
config change or approval form for that existing setup is assigned.
Use [authoring](../docs/calibration-authoring.md),
[rehearsal](../docs/calibration-rehearsal.md),
[runbooks](../docs/runbooks-v1.md) and
[local generation](../docs/local-runbook-generation.md).
## REQUIRED IMPLEMENTATION
### Frozen contract matrix
| ID | Precondition/action | Required result |
|---|---|---|
| C-01 | Explicit current preview-to-draft action versus invalid/stale/superseded/different source | Exact copied canonical candidate enters the existing pinned authoring draft only; no Save, target call, rebase or automatic action on render |
| C-02 | Current owning rehearsal with original verified backup and verified Apply/full actual readback | Trusted copied evidence admits comparison: exact target instance/revision/action/operation/backup/workspace/head/config/definition and complete values; caller-constructed success records are insufficient |
| C-03 | Exact supported scalar/table mapping into original config | NEW immutable config changes only WarmIdleTargetRpm and torque cells; axes/all other parameters/initial state/scenario/profile/module/model refs unchanged; explicit lineage and fingerprint |
| C-04 | Fresh baseline/derived runs and fixed bounded acquisition | Same explicit real Simulation/v2 provider/scenario/initial state, separate run IDs and actual canonical samples/captures; no replay/oracle/canned telemetry or diagnostics-triggered physics |
| C-05 | Decisive warm baseline, real model proposal, explicit Save/backup/approval/Apply/readback | 650->1250 rpm and 2000-rpm torque 360->350 N*m; derived mapping exact, actual last ten warm-idle samples within 1 rpm of each target, stationary; all unrelated config/image values unchanged |
| C-06 | Complete equivalent run comparison/report and existing save/replay | Actual idle/speed/fuel/distance/time/shift metrics, exact configuration/proof/run/capture lineage, synthetic limits; explicit bounded saves/readback and real archive replay preserve canonical evidence |
| C-07 | Busy, cancellation, close/unmount and source/head/proof supersession during pending work | One admitted owner, responsive Hybrid events, drain outside required gates, no late healthy result/rebase; previous success inspectable with truthful stale/partial status |
| C-08 | Missing/invalid proof/config/signals, failure/limits or partially successful pair save | Typed visible rejection before side effects where applicable; incomplete/faulted run is not a successful comparison; exact persisted/unsaved IDs and no overwrite/compensating deletion |
| C-09 | Independent verification and supported fault regression matrix | Test-only independent whole-config/image/mapping/numerical assertions and actual producer/consumer faults; no UI/AI access to a private oracle or invented manufacturer checksum/firmware behavior |
| C-10 | Real native complete demonstration and preserved behavior | All 596 cases/new focused coverage/nine-project Release build pass, actual local-model-to-verified-image-to-run/capture/replay path and no physical/unassigned mutation |
### A. Explicit preview handoff and trusted verified source
Add an explicit current-preview-to-existing-draft action. Application validates
actual workspace/head/applicability and expected preview identity at admission;
copy the complete canonical candidate. The existing draft pins display units;
no string/rounded-display reconstruction. A user must still explicitly Save
the Proposed version through existing validation/head checks.
No Save/approval/execution on AI validation, navigation, notification or render.
Rejected handoff leaves prior draft/head/source/preview unchanged and shows error.
Use existing rehearsal commands for backup/preparation/approval/CAS/readback.
At an explicit evaluation admission, the actual owning rehearsal must be idle/
Ready with a verified original backup and selected current verified Apply
result (including genuinely verified post-reconciliation Apply).
Match exact current identity/revision/image, candidate, saved workspace/head/
applicability and original configuration. Reject RecoveryRequired, missing/
divergent/unavailable/stale/closed evidence or forged caller result flags.
Use a narrow owning-service admission/helper with copied immutable evidence,
not a public constructor accepting arbitrary records and declaring them verified.
Bind original backup values to original supported config values and selected
saved candidate to actual readback in full, changed AND unchanged values.
Preserve target/programming semantics. Any required readonly admission addition
does not open/approve/write a target or bypass its owner/generation guards.
Captured evidence is historical proof at a stated time, not perpetual target
state; supersession must be detected/labeled before current-result publication.
### B. Exact NEW configuration and two equivalent real runs
Implement one pure supported projection using existing constructor/validation/
fingerprint logic. Map only:
- `engine.warm-idle-target` rpm -> Model.WarmIdleTargetRpm.
- Fixed `engine.torque-curve` rpm/N*m axes/cells -> Model.EngineTorqueCurve.
Validate exact definition/units/axis/bounds first. Copy every other model field,
initial condition, five-phase scenario and identity/reference unchanged.
Register/select the NEW config through existing validation/catalogue logic,
reusing/extracting helpers rather than JSON string replacement or duplicating
an importer. Do not modify either the original config or an existing run.
Show original/derived fingerprints and explicit source/proof/value lineage.
A no-change supported mapping is a truthful no-change outcome, not fake benefit.
One comparison admits immutable original/derived pair and runs two fresh owned
diagnostic dynamics sessions sequentially with explicit Simulation/v2 provider.
Use real control StepAsync and validated signal replies; no private engine
sample list, test oracle, model-generated telemetry or replay as physics.
Capture phase before advancing a step where needed for boundary attribution.
Fixed limits/cadence for this slice: at most 3900 model steps per run; engine
AND transmission signal reads every five completed steps and at completion
(avoid duplicate final reads). The decisive 0.1-second warm configuration is
therefore sampled every 0.5 virtual seconds. Capture those actual observations
through existing recording services, inside unchanged 4096-observation/byte/
archive limits; at most 1562 scheduled signal observations per run.
Advance/control phase counts at each step, but do not add unrecorded/private
telemetry or per-step polling that exhausts archive byte limits.
A run needing step 3901 rejects as incomplete,
not Completed or a reason to enlarge existing limits.
Retain one current pair/last successful result only, with bounded immutable
samples/metrics/captures; no hidden history or unbounded collection.
Ordinary manual diagnostic/capture/replay owners are not taken over or cleared.
Reuse independent accepted capture/store APIs for the pair; do not silently
replace another recording workspace or discard an unsaved first capture when
the second begins. Extract a narrow shared helper only if needed.
### C. Comparison, save/readback/replay and truthful lifecycle
Compare actual canonical data: last-ten warm-idle RPM and stationary speed,
warm phase, acceleration/braking elapsed model time, sampled cruise speed
range/count (separate from total cruise model-step count),
fuel consumed delta from initial state, distance, total model time and final
actual transmission.shift-count. Distinguish fixed-cadence shift sampling from
the measured final count; do not infer an exact full shift timeline.
Missing/non-Good/nonfinite/wrong-unit/context replies cannot become zero or a
healthy complete metric. Failed/limited/cancelled runs retain bounded partial
evidence and visible reason, not a complete comparison.
Do not promise reduced fuel, better shifts or real-world improvement.
For the exact demo, compare 650 rpm baseline with 1250 rpm candidate and torque
at 2000 rpm 360->350 N*m. Last ten warm-phase samples on each run must be within
1 rpm of its target and vehicle speed within 0.001 m/s of zero.
Fewer than ten valid warm samples is an explicit incomplete/unsupported case,
not a shortened passing window. Label sampled metrics with their cadence;
do not claim continuous extrema from sparse observations.
Report measured driving/fuel/shift differences without requiring an invented
sign. Keep all original full-scenario numerical regressions, including baseline
3361 samples/336.1 s and 3000 cruise samples within 45 MPH +/-1 MPH.
Report separate proposal/diff, backup/approved mutation/full readback and
functional simulation evidence. Include exact source/head/definition/model,
target instance/revision/operation/backup, original/derived configs, provider/
run IDs, capture IDs/completion and canonical before/after values/metrics.
Provide a safe copyable readonly report; no new persistence/schema/export engine.
Save is explicit, reusing archive v1/atomic no-overwrite publication/readback.
Save each capture with visible per-ID outcome. If one succeeds and the other
fails, preserve the successful file and unsaved capture, identify partial save
and retry only missing work; no pair-atomic claim, rollback deletion or fake
all-saved state. Open saved results through actual existing replay and raw trace.
Units project display only; source/archive/report canonical values stay exact.
Use one cancellable owner, generation/source/proof admission and publication
checks. Queue only admitted blocking work and release Hybrid command events
as accepted in Step-12A. Close/unmount drains owned diagnostic/capture resources
without holding required gates, late-result publication or global interference.
Invalid competing requests reject before starting/mutating. Retained earlier
results remain inspectable, with clear stale/as-of status.
## FILES OR PROJECTS EXPECTED TO CHANGE
Narrow Core projection if needed, Application evaluation/trusted admission/
shared-helper reuse, calibration/evaluation UI and DI, tests and related docs/
README. Only `prompts/Step-13-Report.md` under `prompts`.
No new project/package/host, archive/runbook/profile format, runtime install/
model pull/configuration change or unrelated recording navigation cleanup.
## INTERFACE AND DEPENDENCY REQUIREMENTS
Preserve nine references, independent Components, typed Diagnostics ports,
immutable original configs/authoring/runbooks, volatile backup/approval/CAS/
readback/Restore, archive limits/version and replay metadata semantics.
No general runbook executor, physical programming, arbitrary frames/offsets,
cloud fallback, Git backup, new templates/unit families or privileged oracle.
The model supplies untrusted preview data only; neither proof nor authority.
No agents, automatic handoff, staging/commit/push or orchestration/shared-gate
changes. Step-12A's setup authorization is not recurring runtime management.
## TESTING REQUIREMENTS
Map C-01..C-10 before implementation. Verify exact normal/rejected/boundary/
lifecycle/output cases, not only counts:
- Actual current preview -> draft -> explicit Save, stale/superseded/wrong
source and pinned units. No extra source/target calls on rejected handoff.
- Actual owning rehearsal baseline/two-read backup/approval/CAS/full readback
-> trusted evaluation source. Forged flags, wrong instance/revision/source/
candidate/backup, recovery/divergence and supersession reject atomically.
- Independent whole-config serialized/property comparison proves only the
supported warm-idle/torque fields change; originals remain byte/content exact.
Nonuniform table/cell changes and invalid/nonfinite/unit/axis/boundaries.
- Real two-run v2/recording/store/replay boundary and independent test-only
numerical expectations, last-ten idle thresholds, exact baseline scenario
and fresh equivalent initial state. Independent tests retain verification of
all 3000 cruise model steps; production captures/report explicitly distinguish
sampled signal counts/ranges from full model-step counts. No private oracle
exposed to application/UI/AI.
- Controlled actual owner/transport/save/readback delays and cancel/unmount/
late/superseded/busy failure paths; 3900/3901 limit and unchanged archive caps.
Partial saves/readback corruption and wrong archive/config/replay references.
- Supported earlier timeout/unsupported service/invalid wire/wrong identity/
version/config/interruption/reset/DTC/recovery/reconciliation/Restore/archive
corruption tests remain; name executed evidence. No invented firmware checksum
or partial physical-write mechanism merely to fill a fault table.
Release test barriers in finally; no sleeps/polling as timing correctness proof.
Run focused combined selectors, then sequentially:
- `dotnet build .\src\AutoForge.slnx --configuration Release --no-restore`
- `dotnet test .\src\AutoForge.slnx --configuration Release --no-build`
Preserve all 596 cases/full nine-project/native build. Restore code dependencies
only after genuine chosen-command missing dependency; no SDK/workload/model
install/update/repair or alternate host.
Required actual native demonstration: original warm run/capture; exact local
model goal; current preview handoff and separate Proposed Save; actual Simulation
target backup/approval/Apply/fresh full readback; explicit trusted projection/
equivalent baseline-derived pair; real measured behavior/report/capture saves/
readback/replay/raw trace; failure/cancel/stale/units and unchanged original
configuration/history outside deliberate commands. Existing routes still work.
Record exact source/config/run/capture/proof IDs, model original response and
measurable outcomes. Controlled responses are not native model proof.
No physical target or unrestricted operation; cleanup only exact task-owned
native helpers/archives/profiles/PIDs, preserving user data/runtime/model.
## DOCUMENTATION REQUIREMENTS
Align README/index/architecture/build/authoring/rehearsal/runbook/evaluation
usage with explicit proposal handoff, trusted verified-source admission,
NEW-config projection, equivalent fixed-cadence runs, archive bounds, canonical
metrics/lineage, partial save, stale/cancel/close and synthetic fidelity limits.
Step-12/12A accepted; Step-13 pending manual review; hardware/firmware checksum/
table shift schedules/broader AI evaluation remain future.
## ACCEPTANCE CRITERIA
| ID | Criterion | Required evidence |
|---|---|---|
| AC-01 | C-01..C-03 explicit draft/proof admission and exact immutable projection | Actual owning workflows, independent full-value/config invariants and atomic negative/supersession tests |
| AC-02 | C-04..C-06 equivalent real scenarios/captures/measured intent/report/save/replay | Exact idle thresholds/baseline driving gates, actual canonical samples/metrics/lineage and real archive readback/playback |
| AC-03 | C-07..C-09 responsive safe owner/failures/limits and independent verification | Controlled real-boundary interleavings/3900 boundary/partial saves/corruption, preserved no-side-effect/isolation and supported fault evidence |
| AC-04 | C-10 full 596 regressions/new coverage/nine-project Release and actual native demo pass | Exact worker commands/exits/counts/numerical results, original real model content, proof and captures/replay; no mock-as-native claims |
| AC-05 | Matrix/self-review/docs/history/protection/manual gate accurate | Own post-green producer/consumer review, aligned scope/fidelity and `prompts/Step-13-Report.md` |
## OUT-OF-SCOPE WORK
Physical writes/read-only hardware onboarding, broad AI/RAG/tools/executors,
runtime/model installation/management, firmware layouts/checksums, dynamics
fidelity redesign, changed source configs/runs, new archive/report persistence/
Git backups, generic evaluation frameworks, new projects/packages/hosts and
unrelated fixes. No Step-14, agents or automatic handoff/commits.
## REPORTING REQUIREMENTS
Save `prompts/Step-13-Report.md` wholly inside exactly one plain triple-backtick
fence, no language tag/outside text/nested fences.
Include identifier/title/status/functions/files/interfaces; C-01..C-10/
AC-01..AC-05 named executed matrix; exact preview/draft/save/proof/projection
before/after and unchanged config fields; actual equivalent numerical sample/
metric/run/capture/lineage/save/readback/replay outcomes; owner/negative/limit/
partial failures; commands/exits/warnings/per-project counts; native versus
controlled evidence and original model content; docs/self-review/deviations/
limitations/blockers/preservation/cleanup. No secrets/private thinking traces.
COMPLETE requires all assigned checks, otherwise INCOMPLETE/BLOCKED with exact
unmet criterion. Save and STOP for manual review; do not begin Step-14.