5.1 KiB
Training pipeline
Runtime
- Python
3.12.*(the setup script installs 3.12.13) uvfor interpreter/dependency management- PyTorch 2.11.0 from the CUDA 12.8 index
- Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1
- bitsandbytes 0.49.2 for 4-bit NF4 loading
- CUDA-capable NVIDIA GPU with BF16 support
- WSL/Linux for shell wrappers
- substantial CPU RAM/disk for BF16 merge
The canonical configuration is config/pipeline.json.
Pipeline stages
00 - Audit
scaffolding/00-audit.sh checks Linux/WSL, Python, Git, compiler/build tools,
download tools, NVIDIA utilities, memory, disk, base model, and repository state.
01 - Create environment
scaffolding/01-create-venv.sh installs uv, installs Python 3.12.13, and
recreates .venv. It also attempts to stage and commit files, so inspect/remove
those Git commands before using it in an automated or shared worktree.
02 - Install dependencies
scaffolding/02-install-dependencies.sh locks/synchronizes dependencies and
fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime
information. It also contains Git commit commands.
03 - Export SQL
scaffolding/03-export-sql.ps1 uses Windows integrated authentication, queries
SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative
counts.
04 - Validate source
scripts/validate_source.py enforces the four-field schema, IDs, taxonomy,
Boolean labels, non-empty values, and ID uniqueness.
05 - Create splits
scripts/create_splits.py creates deterministic stratified 80/10/10 partitions
using seed 20260726 and verifies that every input ID appears exactly once.
06 - Build SFT
scripts/build_sft.py loads config/system_prompt.txt, builds the exact target
JSON, and overwrites all three split files with chat conversations. Always run
stage 05 immediately before rerunning stage 06.
07 - Base-model smoke test
scripts/model_smoke_test.py loads one training example, tokenizer/template, and
the 4-bit base model; performs a forward pass; and rejects non-finite logits.
It does not test the adapter or merged model.
08 - Pilot training
scripts/train_pilot.py selects the first up-to-64 training records and runs 10
steps. It deletes/recreates outputs/pilot-adapter.
09 - Full training
scripts/train.py uses:
- NF4 4-bit base weights with double quantization and BF16 compute;
- LoRA rank 16, alpha 32, dropout 0.05, target
all-linear; - completion-only loss over the assistant response;
- gradient checkpointing;
- batch size 2 and accumulation 8;
- 2 epochs, learning rate 0.0002, 100 warmup steps;
- epoch evaluation and checkpointing;
- lowest validation loss as the best saved checkpoint.
It deletes/recreates outputs/final-adapter. There is no early-stopping
callback.
10 - Evaluate
scripts/evaluate.py loads the base model plus final adapter and generates all
development outputs in deterministic batches of eight. Generation stops at EOS,
end-of-turn, or a complete balanced JSON object. Results are atomically written
to outputs/development-evaluation.
Metrics cover JSON/schema validity, classification confusion matrix and precision/recall/F1, result count, category, text, and evidence.
Rescore without inference
scripts/rescore_evaluation.py reparses existing raw outputs, annotates the
prediction records, and rewrites predictions/summary without rerunning a model.
11 - Verify generation stopping
scripts/verify_generation_stop.py generates the first five development
examples and requires each complete output to parse with json.loads. Balanced
braces trigger stopping; schema validation remains a later responsibility.
12 - Merge
scripts/merge_model.py loads unquantized BF16 base weights on CPU, merges the
final adapter, saves safe-serialized 5 GB shards to a temporary directory,
verifies key files, and atomically promotes the directory to
outputs/Invited-PHI-8B.
This stage creates a standalone Hugging Face model, not GGUF.
Chat contract
config/chat_template.jinja uses Llama-style role headers and end-of-turn
tokens. config/system_prompt.txt requires exactly one JSON object:
{
"is_medical_phi": true,
"score": 1.0,
"reason": "...",
"results": [
{
"entity_type": "MEDICAL_DIAGNOSIS",
"text": "...",
"score": 1.0,
"evidence": "..."
}
]
}
No Markdown, extra prose, inferred facts, or non-MEDICAL_* types are allowed.
Evaluation caveats
- A valid JSON prefix followed by trailing text can count as parseable in the
evaluator/rescorer because
raw_decodeis used. - Entity comparisons use the first expected and predicted result.
- Entity accuracy denominators are all positive records.
- Exact whole-input text/evidence reproduction is measured, not span overlap.
- Result-count correctness is reported separately from schema validity.
- Evaluation covers base-plus-adapter, not the merged standalone model.
Path caveat
All .sh wrappers use /d/ai/training/openbiollm-phi, while this checkout may
be elsewhere (for example /d/ai/training/Models-Phi-8B). Fix the PROJECT_ROOT
line in every wrapper or invoke the underlying uv run python commands from the
actual repository root.