7.3 KiB
End-to-end runbook
This procedure starts with catalogs and ends with a merged model. Run commands from the repository root unless stated otherwise.
1. Record the run
Capture:
- Git revision and clean/dirty state;
- operator and UTC start time;
- catalog/pattern checksums and provenance manifest;
- SQL Server target;
- LM Studio model artifact/version and settings;
- base model path/revision/checksum;
- GPU, driver, CUDA, Python, and dependency versions;
- expected output locations and acceptance thresholds.
2. Prerequisite checks
Install/prepare:
- .NET 10 SDK;
- SQL Server reachable with schema-creation privileges;
- LM Studio with
qwen/qwen3.5-9b(or an intentionally approved replacement); - Windows PowerShell for SQL export;
- WSL/Linux, Git, build tools, and
uv; - Python 3.12;
- NVIDIA CUDA GPU with BF16 and sufficient VRAM;
- local
Llama3-OpenBioLLM-8Bbase model; - enough CPU RAM/disk for an unquantized 8B BF16 merge.
Correct the legacy /d/ai/training/openbiollm-phi path in every shell scaffold,
or use the direct commands below from the real checkout.
3. Validate generator inputs
Confirm:
- exactly 12 category files;
- one positive and one negative pattern file per category;
- subjects exactly
He,She,They,Member; - no legacy
[s]tokens; - canonical/variant forms are unique;
- each category has values and compatible
[x]patterns.
The generator also enforces these checks when a run starts.
4. Configure and migrate SQL Server
Review the RLHFGenerator connection string in
PHI.Data.Generator/RLHF.Generator.Api/appsettings.json. Prefer environment
configuration or user secrets for deployment-specific values.
From PHI.Data.Generator:
dotnet restore .\RLHF.Generator.slnx
dotnet ef database update `
--project .\RLHF.Generator\RLHF.Generator.csproj `
--startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj
Back up an existing database before migration or reset.
5. Test the generator
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
Stop if any test fails.
6. Start LM Studio
- Load the configured normalization model.
- Start the local OpenAI-compatible server.
- Confirm
http://localhost:1234/v1/chat/completionsis reachable. - Keep temperature, seed, token limit, and model name aligned with generator configuration.
7. Start the generator API
dotnet run --project `
.\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj
Note the actual listening URL. In Development, inspect Swagger/Scalar. Call
GET / as a basic process check.
8. Load keywords and generate
In another PowerShell window:
$base = "http://localhost:<port>"
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
The generation request remains open until completion. Monitor API logs and SQL
GenerationRun. On cancellation/failure, correct the cause and call the same
endpoint to resume only if the service accepts the matching fingerprints.
Do not edit catalogs, patterns, generation settings, or LM settings mid-run.
Optional keyword linkage:
Invoke-RestMethod -Method Post `
-Uri "$base/api/training-items/backfill-keywords?dryRun=true"
Invoke-RestMethod -Method Post `
-Uri "$base/api/training-items/backfill-keywords?dryRun=false"
9. Export SQL data
Review SQL server/database values in config/pipeline.json, then run:
.\scaffolding\03-export-sql.ps1
Confirm data/source.jsonl exists and record positive/negative totals.
10. Prepare the Python environment in WSL
From the actual WSL path of the repository:
uv python install 3.12.13
uv venv --python 3.12.13
uv lock
uv sync --frozen
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"
Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the
environment and contain git add/git commit commands.
11. Review pipeline configuration
Update config/pipeline.json deliberately:
- SQL connection metadata;
- actual WSL base-model path;
- split proportions/seed;
- sequence length;
- LoRA and training hyperparameters;
- output directories.
Archive the exact configuration used.
12. Validate and split
uv run python scripts/validate_source.py
uv run python scripts/create_splits.py
Review counts per category/polarity/split. Pay special attention to small strata.
13. Build SFT conversations
uv run python scripts/build_sft.py
This overwrites all split files. If repeated, run create_splits.py first.
Inspect a positive and negative example from every category.
14. Smoke-test the base model
uv run python scripts/model_smoke_test.py
Confirm successful loading, finite logits, and expected GPU memory use.
15. Run the pilot
uv run python scripts/train_pilot.py
Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss before full training.
16. Train the full adapter
uv run python scripts/train.py
This replaces outputs/final-adapter. Preserve any prior accepted adapter first.
Monitor training/evaluation loss, checkpoints, utilization, and disk.
17. Evaluate and rescore
uv run python scripts/evaluate.py
uv run python scripts/rescore_evaluation.py
Compare outputs/development-evaluation/summary.json with acceptance thresholds.
Review every error, particularly false negatives, false positives, category
errors, malformed JSON, and contextual-negative failures.
18. Verify stopping
uv run python scripts/verify_generation_stop.py
All five sampled outputs must be exact parseable JSON.
19. Merge the adapter
uv run python scripts/merge_model.py
This replaces outputs/Invited-PHI-8B atomically after verification. It can
require substantial CPU RAM and disk.
20. Validate and publish artifacts
Before release:
- Add/run a merged-model inference smoke test.
- Confirm tokenizer, chat template, system prompt, configs, and all weight shards are present.
- Record checksums and total sizes.
- Preserve adapter, merged model, metrics, predictions, config, source/split checksums, Git revision, environment lock, and provenance manifest.
- If GGUF is required, use an approved conversion/quantization process and record it; no such process is tracked here.
- Run downstream Presidio integration and end-user acceptance tests.
Recovery
| Failure | Action |
|---|---|
| LM Studio unavailable/timeout | Restore service/model, then resume unchanged run |
| Generator invalid output | Inspect request/model/prompt; do not bypass validation |
| Fingerprint mismatch | Restore original inputs/settings or intentionally reset/start clean |
| SQL batch failure | Correct database cause; checkpointed committed batches remain |
| Split/SFT rerun failure | Recreate splits from source.jsonl, then rebuild SFT |
| CUDA OOM | Reduce approved batch/sequence settings, rerun pilot, document change |
| Training output exists | Archive accepted output; scripts intentionally replace output dirs |
| Evaluation failure | Preserve raw logs/artifacts, fix root cause, rerun evaluation |
| Merge resource failure | Increase CPU RAM/disk or use an approved merge host |