PHI-Model-8B/docs/runbook.md

255 lines
7.3 KiB
Markdown

# End-to-end runbook
This procedure starts with catalogs and ends with a merged model. Run commands
from the repository root unless stated otherwise.
## 1. Record the run
Capture:
- Git revision and clean/dirty state;
- operator and UTC start time;
- catalog/pattern checksums and provenance manifest;
- SQL Server target;
- LM Studio model artifact/version and settings;
- base model path/revision/checksum;
- GPU, driver, CUDA, Python, and dependency versions;
- expected output locations and acceptance thresholds.
## 2. Prerequisite checks
Install/prepare:
- .NET 10 SDK;
- SQL Server reachable with schema-creation privileges;
- LM Studio with `qwen/qwen3.5-9b` (or an intentionally approved replacement);
- Windows PowerShell for SQL export;
- WSL/Linux, Git, build tools, and `uv`;
- Python 3.12;
- NVIDIA CUDA GPU with BF16 and sufficient VRAM;
- local `Llama3-OpenBioLLM-8B` base model;
- enough CPU RAM/disk for an unquantized 8B BF16 merge.
Correct the legacy `/d/ai/training/openbiollm-phi` path in every shell scaffold,
or use the direct commands below from the real checkout.
## 3. Validate generator inputs
Confirm:
- exactly 12 category files;
- one positive and one negative pattern file per category;
- subjects exactly `He`, `She`, `They`, `Member`;
- no legacy `[s]` tokens;
- canonical/variant forms are unique;
- each category has values and compatible `[x]` patterns.
The generator also enforces these checks when a run starts.
## 4. Configure and migrate SQL Server
Review the `RLHFGenerator` connection string in
`PHI.Data.Generator/RLHF.Generator.Api/appsettings.json`. Prefer environment
configuration or user secrets for deployment-specific values.
From `PHI.Data.Generator`:
```powershell
dotnet restore .\RLHF.Generator.slnx
dotnet ef database update `
--project .\RLHF.Generator\RLHF.Generator.csproj `
--startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj
```
Back up an existing database before migration or reset.
## 5. Test the generator
```powershell
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
```
Stop if any test fails.
## 6. Start LM Studio
1. Load the configured normalization model.
2. Start the local OpenAI-compatible server.
3. Confirm `http://localhost:1234/v1/chat/completions` is reachable.
4. Keep temperature, seed, token limit, and model name aligned with generator
configuration.
## 7. Start the generator API
```powershell
dotnet run --project `
.\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj
```
Note the actual listening URL. In Development, inspect Swagger/Scalar. Call
`GET /` as a basic process check.
## 8. Load keywords and generate
In another PowerShell window:
```powershell
$base = "http://localhost:<port>"
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
```
The generation request remains open until completion. Monitor API logs and SQL
`GenerationRun`. On cancellation/failure, correct the cause and call the same
endpoint to resume only if the service accepts the matching fingerprints.
Do not edit catalogs, patterns, generation settings, or LM settings mid-run.
Optional keyword linkage:
```powershell
Invoke-RestMethod -Method Post `
-Uri "$base/api/training-items/backfill-keywords?dryRun=true"
Invoke-RestMethod -Method Post `
-Uri "$base/api/training-items/backfill-keywords?dryRun=false"
```
## 9. Export SQL data
Review SQL server/database values in `config/pipeline.json`, then run:
```powershell
.\scaffolding\03-export-sql.ps1
```
Confirm `data/source.jsonl` exists and record positive/negative totals.
## 10. Prepare the Python environment in WSL
From the actual WSL path of the repository:
```bash
uv python install 3.12.13
uv venv --python 3.12.13
uv lock
uv sync --frozen
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"
```
Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the
environment and contain `git add`/`git commit` commands.
## 11. Review pipeline configuration
Update `config/pipeline.json` deliberately:
- SQL connection metadata;
- actual WSL base-model path;
- split proportions/seed;
- sequence length;
- LoRA and training hyperparameters;
- output directories.
Archive the exact configuration used.
## 12. Validate and split
```bash
uv run python scripts/validate_source.py
uv run python scripts/create_splits.py
```
Review counts per category/polarity/split. Pay special attention to small strata.
## 13. Build SFT conversations
```bash
uv run python scripts/build_sft.py
```
This overwrites all split files. If repeated, run `create_splits.py` first.
Inspect a positive and negative example from every category.
## 14. Smoke-test the base model
```bash
uv run python scripts/model_smoke_test.py
```
Confirm successful loading, finite logits, and expected GPU memory use.
## 15. Run the pilot
```bash
uv run python scripts/train_pilot.py
```
Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss
before full training.
## 16. Train the full adapter
```bash
uv run python scripts/train.py
```
This replaces `outputs/final-adapter`. Preserve any prior accepted adapter first.
Monitor training/evaluation loss, checkpoints, utilization, and disk.
## 17. Evaluate and rescore
```bash
uv run python scripts/evaluate.py
uv run python scripts/rescore_evaluation.py
```
Compare `outputs/development-evaluation/summary.json` with acceptance thresholds.
Review every error, particularly false negatives, false positives, category
errors, malformed JSON, and contextual-negative failures.
## 18. Verify stopping
```bash
uv run python scripts/verify_generation_stop.py
```
All five sampled outputs must be exact parseable JSON.
## 19. Merge the adapter
```bash
uv run python scripts/merge_model.py
```
This replaces `outputs/Invited-PHI-8B` atomically after verification. It can
require substantial CPU RAM and disk.
## 20. Validate and publish artifacts
Before release:
1. Add/run a merged-model inference smoke test.
2. Confirm tokenizer, chat template, system prompt, configs, and all weight
shards are present.
3. Record checksums and total sizes.
4. Preserve adapter, merged model, metrics, predictions, config, source/split
checksums, Git revision, environment lock, and provenance manifest.
5. If GGUF is required, use an approved conversion/quantization process and
record it; no such process is tracked here.
6. Run downstream Presidio integration and end-user acceptance tests.
## Recovery
| Failure | Action |
|---|---|
| LM Studio unavailable/timeout | Restore service/model, then resume unchanged run |
| Generator invalid output | Inspect request/model/prompt; do not bypass validation |
| Fingerprint mismatch | Restore original inputs/settings or intentionally reset/start clean |
| SQL batch failure | Correct database cause; checkpointed committed batches remain |
| Split/SFT rerun failure | Recreate splits from `source.jsonl`, then rebuild SFT |
| CUDA OOM | Reduce approved batch/sequence settings, rerun pilot, document change |
| Training output exists | Archive accepted output; scripts intentionally replace output dirs |
| Evaluation failure | Preserve raw logs/artifacts, fix root cause, rerun evaluation |
| Merge resource failure | Increase CPU RAM/disk or use an approved merge host |