Models-Phi-8B/docs/training-pipeline.md

153 lines
5.1 KiB
Markdown

# Training pipeline
## Runtime
- Python `3.12.*` (the setup script installs 3.12.13)
- `uv` for interpreter/dependency management
- PyTorch 2.11.0 from the CUDA 12.8 index
- Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1
- bitsandbytes 0.49.2 for 4-bit NF4 loading
- CUDA-capable NVIDIA GPU with BF16 support
- WSL/Linux for shell wrappers
- substantial CPU RAM/disk for BF16 merge
The canonical configuration is `config/pipeline.json`.
## Pipeline stages
### 00 - Audit
`scaffolding/00-audit.sh` checks Linux/WSL, Python, Git, compiler/build tools,
download tools, NVIDIA utilities, memory, disk, base model, and repository state.
### 01 - Create environment
`scaffolding/01-create-venv.sh` installs `uv`, installs Python 3.12.13, and
recreates `.venv`. It also attempts to stage and commit files, so inspect/remove
those Git commands before using it in an automated or shared worktree.
### 02 - Install dependencies
`scaffolding/02-install-dependencies.sh` locks/synchronizes dependencies and
fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime
information. It also contains Git commit commands.
### 03 - Export SQL
`scaffolding/03-export-sql.ps1` uses Windows integrated authentication, queries
SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative
counts.
### 04 - Validate source
`scripts/validate_source.py` enforces the four-field schema, IDs, taxonomy,
Boolean labels, non-empty values, and ID uniqueness.
### 05 - Create splits
`scripts/create_splits.py` creates deterministic stratified 80/10/10 partitions
using seed `20260726` and verifies that every input ID appears exactly once.
### 06 - Build SFT
`scripts/build_sft.py` loads `config/system_prompt.txt`, builds the exact target
JSON, and overwrites all three split files with chat conversations. Always run
stage 05 immediately before rerunning stage 06.
### 07 - Base-model smoke test
`scripts/model_smoke_test.py` loads one training example, tokenizer/template, and
the 4-bit base model; performs a forward pass; and rejects non-finite logits.
It does not test the adapter or merged model.
### 08 - Pilot training
`scripts/train_pilot.py` selects the first up-to-64 training records and runs 10
steps. It deletes/recreates `outputs/pilot-adapter`.
### 09 - Full training
`scripts/train.py` uses:
- NF4 4-bit base weights with double quantization and BF16 compute;
- LoRA rank 16, alpha 32, dropout 0.05, target `all-linear`;
- completion-only loss over the assistant response;
- gradient checkpointing;
- batch size 2 and accumulation 8;
- 2 epochs, learning rate 0.0002, 100 warmup steps;
- epoch evaluation and checkpointing;
- lowest validation loss as the best saved checkpoint.
It deletes/recreates `outputs/final-adapter`. There is no early-stopping
callback.
### 10 - Evaluate
`scripts/evaluate.py` loads the base model plus final adapter and generates all
development outputs in deterministic batches of eight. Generation stops at EOS,
end-of-turn, or a complete balanced JSON object. Results are atomically written
to `outputs/development-evaluation`.
Metrics cover JSON/schema validity, classification confusion matrix and
precision/recall/F1, result count, category, text, and evidence.
### Rescore without inference
`scripts/rescore_evaluation.py` reparses existing raw outputs, annotates the
prediction records, and rewrites predictions/summary without rerunning a model.
### 11 - Verify generation stopping
`scripts/verify_generation_stop.py` generates the first five development
examples and requires each complete output to parse with `json.loads`. Balanced
braces trigger stopping; schema validation remains a later responsibility.
### 12 - Merge
`scripts/merge_model.py` loads unquantized BF16 base weights on CPU, merges the
final adapter, saves safe-serialized 5 GB shards to a temporary directory,
verifies key files, and atomically promotes the directory to
`outputs/Invited-PHI-8B`.
This stage creates a standalone Hugging Face model, not GGUF.
## Chat contract
`config/chat_template.jinja` uses Llama-style role headers and end-of-turn
tokens. `config/system_prompt.txt` requires exactly one JSON object:
```json
{
"is_medical_phi": true,
"score": 1.0,
"reason": "...",
"results": [
{
"entity_type": "MEDICAL_DIAGNOSIS",
"text": "...",
"score": 1.0,
"evidence": "..."
}
]
}
```
No Markdown, extra prose, inferred facts, or non-`MEDICAL_*` types are allowed.
## Evaluation caveats
- A valid JSON prefix followed by trailing text can count as parseable in the
evaluator/rescorer because `raw_decode` is used.
- Entity comparisons use the first expected and predicted result.
- Entity accuracy denominators are all positive records.
- Exact whole-input text/evidence reproduction is measured, not span overlap.
- Result-count correctness is reported separately from schema validity.
- Evaluation covers base-plus-adapter, not the merged standalone model.
## Path caveat
All `.sh` wrappers use `/d/ai/training/openbiollm-phi`, while this checkout may
be elsewhere (for example `/d/ai/training/Models-Phi-8B`). Fix the `PROJECT_ROOT`
line in every wrapper or invoke the underlying `uv run python` commands from the
actual repository root.