255 lines
7.3 KiB
Markdown
255 lines
7.3 KiB
Markdown
# End-to-end runbook
|
|
|
|
This procedure starts with catalogs and ends with a merged model. Run commands
|
|
from the repository root unless stated otherwise.
|
|
|
|
## 1. Record the run
|
|
|
|
Capture:
|
|
|
|
- Git revision and clean/dirty state;
|
|
- operator and UTC start time;
|
|
- catalog/pattern checksums and provenance manifest;
|
|
- SQL Server target;
|
|
- LM Studio model artifact/version and settings;
|
|
- base model path/revision/checksum;
|
|
- GPU, driver, CUDA, Python, and dependency versions;
|
|
- expected output locations and acceptance thresholds.
|
|
|
|
## 2. Prerequisite checks
|
|
|
|
Install/prepare:
|
|
|
|
- .NET 10 SDK;
|
|
- SQL Server reachable with schema-creation privileges;
|
|
- LM Studio with `qwen/qwen3.5-9b` (or an intentionally approved replacement);
|
|
- Windows PowerShell for SQL export;
|
|
- WSL/Linux, Git, build tools, and `uv`;
|
|
- Python 3.12;
|
|
- NVIDIA CUDA GPU with BF16 and sufficient VRAM;
|
|
- local `Llama3-OpenBioLLM-8B` base model;
|
|
- enough CPU RAM/disk for an unquantized 8B BF16 merge.
|
|
|
|
Correct the legacy `/d/ai/training/openbiollm-phi` path in every shell scaffold,
|
|
or use the direct commands below from the real checkout.
|
|
|
|
## 3. Validate generator inputs
|
|
|
|
Confirm:
|
|
|
|
- exactly 12 category files;
|
|
- one positive and one negative pattern file per category;
|
|
- subjects exactly `He`, `She`, `They`, `Member`;
|
|
- no legacy `[s]` tokens;
|
|
- canonical/variant forms are unique;
|
|
- each category has values and compatible `[x]` patterns.
|
|
|
|
The generator also enforces these checks when a run starts.
|
|
|
|
## 4. Configure and migrate SQL Server
|
|
|
|
Review the `RLHFGenerator` connection string in
|
|
`PHI.Data.Generator/RLHF.Generator.Api/appsettings.json`. Prefer environment
|
|
configuration or user secrets for deployment-specific values.
|
|
|
|
From `PHI.Data.Generator`:
|
|
|
|
```powershell
|
|
dotnet restore .\RLHF.Generator.slnx
|
|
dotnet ef database update `
|
|
--project .\RLHF.Generator\RLHF.Generator.csproj `
|
|
--startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj
|
|
```
|
|
|
|
Back up an existing database before migration or reset.
|
|
|
|
## 5. Test the generator
|
|
|
|
```powershell
|
|
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
|
|
```
|
|
|
|
Stop if any test fails.
|
|
|
|
## 6. Start LM Studio
|
|
|
|
1. Load the configured normalization model.
|
|
2. Start the local OpenAI-compatible server.
|
|
3. Confirm `http://localhost:1234/v1/chat/completions` is reachable.
|
|
4. Keep temperature, seed, token limit, and model name aligned with generator
|
|
configuration.
|
|
|
|
## 7. Start the generator API
|
|
|
|
```powershell
|
|
dotnet run --project `
|
|
.\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj
|
|
```
|
|
|
|
Note the actual listening URL. In Development, inspect Swagger/Scalar. Call
|
|
`GET /` as a basic process check.
|
|
|
|
## 8. Load keywords and generate
|
|
|
|
In another PowerShell window:
|
|
|
|
```powershell
|
|
$base = "http://localhost:<port>"
|
|
|
|
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
|
|
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
|
|
```
|
|
|
|
The generation request remains open until completion. Monitor API logs and SQL
|
|
`GenerationRun`. On cancellation/failure, correct the cause and call the same
|
|
endpoint to resume only if the service accepts the matching fingerprints.
|
|
|
|
Do not edit catalogs, patterns, generation settings, or LM settings mid-run.
|
|
|
|
Optional keyword linkage:
|
|
|
|
```powershell
|
|
Invoke-RestMethod -Method Post `
|
|
-Uri "$base/api/training-items/backfill-keywords?dryRun=true"
|
|
Invoke-RestMethod -Method Post `
|
|
-Uri "$base/api/training-items/backfill-keywords?dryRun=false"
|
|
```
|
|
|
|
## 9. Export SQL data
|
|
|
|
Review SQL server/database values in `config/pipeline.json`, then run:
|
|
|
|
```powershell
|
|
.\scaffolding\03-export-sql.ps1
|
|
```
|
|
|
|
Confirm `data/source.jsonl` exists and record positive/negative totals.
|
|
|
|
## 10. Prepare the Python environment in WSL
|
|
|
|
From the actual WSL path of the repository:
|
|
|
|
```bash
|
|
uv python install 3.12.13
|
|
uv venv --python 3.12.13
|
|
uv lock
|
|
uv sync --frozen
|
|
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"
|
|
```
|
|
|
|
Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the
|
|
environment and contain `git add`/`git commit` commands.
|
|
|
|
## 11. Review pipeline configuration
|
|
|
|
Update `config/pipeline.json` deliberately:
|
|
|
|
- SQL connection metadata;
|
|
- actual WSL base-model path;
|
|
- split proportions/seed;
|
|
- sequence length;
|
|
- LoRA and training hyperparameters;
|
|
- output directories.
|
|
|
|
Archive the exact configuration used.
|
|
|
|
## 12. Validate and split
|
|
|
|
```bash
|
|
uv run python scripts/validate_source.py
|
|
uv run python scripts/create_splits.py
|
|
```
|
|
|
|
Review counts per category/polarity/split. Pay special attention to small strata.
|
|
|
|
## 13. Build SFT conversations
|
|
|
|
```bash
|
|
uv run python scripts/build_sft.py
|
|
```
|
|
|
|
This overwrites all split files. If repeated, run `create_splits.py` first.
|
|
Inspect a positive and negative example from every category.
|
|
|
|
## 14. Smoke-test the base model
|
|
|
|
```bash
|
|
uv run python scripts/model_smoke_test.py
|
|
```
|
|
|
|
Confirm successful loading, finite logits, and expected GPU memory use.
|
|
|
|
## 15. Run the pilot
|
|
|
|
```bash
|
|
uv run python scripts/train_pilot.py
|
|
```
|
|
|
|
Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss
|
|
before full training.
|
|
|
|
## 16. Train the full adapter
|
|
|
|
```bash
|
|
uv run python scripts/train.py
|
|
```
|
|
|
|
This replaces `outputs/final-adapter`. Preserve any prior accepted adapter first.
|
|
Monitor training/evaluation loss, checkpoints, utilization, and disk.
|
|
|
|
## 17. Evaluate and rescore
|
|
|
|
```bash
|
|
uv run python scripts/evaluate.py
|
|
uv run python scripts/rescore_evaluation.py
|
|
```
|
|
|
|
Compare `outputs/development-evaluation/summary.json` with acceptance thresholds.
|
|
Review every error, particularly false negatives, false positives, category
|
|
errors, malformed JSON, and contextual-negative failures.
|
|
|
|
## 18. Verify stopping
|
|
|
|
```bash
|
|
uv run python scripts/verify_generation_stop.py
|
|
```
|
|
|
|
All five sampled outputs must be exact parseable JSON.
|
|
|
|
## 19. Merge the adapter
|
|
|
|
```bash
|
|
uv run python scripts/merge_model.py
|
|
```
|
|
|
|
This replaces `outputs/Invited-PHI-8B` atomically after verification. It can
|
|
require substantial CPU RAM and disk.
|
|
|
|
## 20. Validate and publish artifacts
|
|
|
|
Before release:
|
|
|
|
1. Add/run a merged-model inference smoke test.
|
|
2. Confirm tokenizer, chat template, system prompt, configs, and all weight
|
|
shards are present.
|
|
3. Record checksums and total sizes.
|
|
4. Preserve adapter, merged model, metrics, predictions, config, source/split
|
|
checksums, Git revision, environment lock, and provenance manifest.
|
|
5. If GGUF is required, use an approved conversion/quantization process and
|
|
record it; no such process is tracked here.
|
|
6. Run downstream Presidio integration and end-user acceptance tests.
|
|
|
|
## Recovery
|
|
|
|
| Failure | Action |
|
|
|---|---|
|
|
| LM Studio unavailable/timeout | Restore service/model, then resume unchanged run |
|
|
| Generator invalid output | Inspect request/model/prompt; do not bypass validation |
|
|
| Fingerprint mismatch | Restore original inputs/settings or intentionally reset/start clean |
|
|
| SQL batch failure | Correct database cause; checkpointed committed batches remain |
|
|
| Split/SFT rerun failure | Recreate splits from `source.jsonl`, then rebuild SFT |
|
|
| CUDA OOM | Reduce approved batch/sequence settings, rerun pilot, document change |
|
|
| Training output exists | Archive accepted output; scripts intentionally replace output dirs |
|
|
| Evaluation failure | Preserve raw logs/artifacts, fix root cause, rerun evaluation |
|
|
| Merge resource failure | Increase CPU RAM/disk or use an approved merge host |
|