Models-Phi-8B/docs/runbook.md

7.3 KiB

End-to-end runbook

This procedure starts with catalogs and ends with a merged model. Run commands from the repository root unless stated otherwise.

1. Record the run

Capture:

  • Git revision and clean/dirty state;
  • operator and UTC start time;
  • catalog/pattern checksums and provenance manifest;
  • SQL Server target;
  • LM Studio model artifact/version and settings;
  • base model path/revision/checksum;
  • GPU, driver, CUDA, Python, and dependency versions;
  • expected output locations and acceptance thresholds.

2. Prerequisite checks

Install/prepare:

  • .NET 10 SDK;
  • SQL Server reachable with schema-creation privileges;
  • LM Studio with qwen/qwen3.5-9b (or an intentionally approved replacement);
  • Windows PowerShell for SQL export;
  • WSL/Linux, Git, build tools, and uv;
  • Python 3.12;
  • NVIDIA CUDA GPU with BF16 and sufficient VRAM;
  • local Llama3-OpenBioLLM-8B base model;
  • enough CPU RAM/disk for an unquantized 8B BF16 merge.

Correct the legacy /d/ai/training/openbiollm-phi path in every shell scaffold, or use the direct commands below from the real checkout.

3. Validate generator inputs

Confirm:

  • exactly 12 category files;
  • one positive and one negative pattern file per category;
  • subjects exactly He, She, They, Member;
  • no legacy [s] tokens;
  • canonical/variant forms are unique;
  • each category has values and compatible [x] patterns.

The generator also enforces these checks when a run starts.

4. Configure and migrate SQL Server

Review the RLHFGenerator connection string in PHI.Data.Generator/RLHF.Generator.Api/appsettings.json. Prefer environment configuration or user secrets for deployment-specific values.

From PHI.Data.Generator:

dotnet restore .\RLHF.Generator.slnx
dotnet ef database update `
  --project .\RLHF.Generator\RLHF.Generator.csproj `
  --startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj

Back up an existing database before migration or reset.

5. Test the generator

dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx

Stop if any test fails.

6. Start LM Studio

  1. Load the configured normalization model.
  2. Start the local OpenAI-compatible server.
  3. Confirm http://localhost:1234/v1/chat/completions is reachable.
  4. Keep temperature, seed, token limit, and model name aligned with generator configuration.

7. Start the generator API

dotnet run --project `
  .\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj

Note the actual listening URL. In Development, inspect Swagger/Scalar. Call GET / as a basic process check.

8. Load keywords and generate

In another PowerShell window:

$base = "http://localhost:<port>"

Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"

The generation request remains open until completion. Monitor API logs and SQL GenerationRun. On cancellation/failure, correct the cause and call the same endpoint to resume only if the service accepts the matching fingerprints.

Do not edit catalogs, patterns, generation settings, or LM settings mid-run.

Optional keyword linkage:

Invoke-RestMethod -Method Post `
  -Uri "$base/api/training-items/backfill-keywords?dryRun=true"
Invoke-RestMethod -Method Post `
  -Uri "$base/api/training-items/backfill-keywords?dryRun=false"

9. Export SQL data

Review SQL server/database values in config/pipeline.json, then run:

.\scaffolding\03-export-sql.ps1

Confirm data/source.jsonl exists and record positive/negative totals.

10. Prepare the Python environment in WSL

From the actual WSL path of the repository:

uv python install 3.12.13
uv venv --python 3.12.13
uv lock
uv sync --frozen
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"

Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the environment and contain git add/git commit commands.

11. Review pipeline configuration

Update config/pipeline.json deliberately:

  • SQL connection metadata;
  • actual WSL base-model path;
  • split proportions/seed;
  • sequence length;
  • LoRA and training hyperparameters;
  • output directories.

Archive the exact configuration used.

12. Validate and split

uv run python scripts/validate_source.py
uv run python scripts/create_splits.py

Review counts per category/polarity/split. Pay special attention to small strata.

13. Build SFT conversations

uv run python scripts/build_sft.py

This overwrites all split files. If repeated, run create_splits.py first. Inspect a positive and negative example from every category.

14. Smoke-test the base model

uv run python scripts/model_smoke_test.py

Confirm successful loading, finite logits, and expected GPU memory use.

15. Run the pilot

uv run python scripts/train_pilot.py

Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss before full training.

16. Train the full adapter

uv run python scripts/train.py

This replaces outputs/final-adapter. Preserve any prior accepted adapter first. Monitor training/evaluation loss, checkpoints, utilization, and disk.

17. Evaluate and rescore

uv run python scripts/evaluate.py
uv run python scripts/rescore_evaluation.py

Compare outputs/development-evaluation/summary.json with acceptance thresholds. Review every error, particularly false negatives, false positives, category errors, malformed JSON, and contextual-negative failures.

18. Verify stopping

uv run python scripts/verify_generation_stop.py

All five sampled outputs must be exact parseable JSON.

19. Merge the adapter

uv run python scripts/merge_model.py

This replaces outputs/Invited-PHI-8B atomically after verification. It can require substantial CPU RAM and disk.

20. Validate and publish artifacts

Before release:

  1. Add/run a merged-model inference smoke test.
  2. Confirm tokenizer, chat template, system prompt, configs, and all weight shards are present.
  3. Record checksums and total sizes.
  4. Preserve adapter, merged model, metrics, predictions, config, source/split checksums, Git revision, environment lock, and provenance manifest.
  5. If GGUF is required, use an approved conversion/quantization process and record it; no such process is tracked here.
  6. Run downstream Presidio integration and end-user acceptance tests.

Recovery

Failure Action
LM Studio unavailable/timeout Restore service/model, then resume unchanged run
Generator invalid output Inspect request/model/prompt; do not bypass validation
Fingerprint mismatch Restore original inputs/settings or intentionally reset/start clean
SQL batch failure Correct database cause; checkpointed committed batches remain
Split/SFT rerun failure Recreate splits from source.jsonl, then rebuild SFT
CUDA OOM Reduce approved batch/sequence settings, rerun pilot, document change
Training output exists Archive accepted output; scripts intentionally replace output dirs
Evaluation failure Preserve raw logs/artifacts, fix root cause, rerun evaluation
Merge resource failure Increase CPU RAM/disk or use an approved merge host