PHI-Model-8B/docs/training-pipeline.md

5.1 KiB

Training pipeline

Runtime

  • Python 3.12.* (the setup script installs 3.12.13)
  • uv for interpreter/dependency management
  • PyTorch 2.11.0 from the CUDA 12.8 index
  • Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1
  • bitsandbytes 0.49.2 for 4-bit NF4 loading
  • CUDA-capable NVIDIA GPU with BF16 support
  • WSL/Linux for shell wrappers
  • substantial CPU RAM/disk for BF16 merge

The canonical configuration is config/pipeline.json.

Pipeline stages

00 - Audit

scaffolding/00-audit.sh checks Linux/WSL, Python, Git, compiler/build tools, download tools, NVIDIA utilities, memory, disk, base model, and repository state.

01 - Create environment

scaffolding/01-create-venv.sh installs uv, installs Python 3.12.13, and recreates .venv. It also attempts to stage and commit files, so inspect/remove those Git commands before using it in an automated or shared worktree.

02 - Install dependencies

scaffolding/02-install-dependencies.sh locks/synchronizes dependencies and fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime information. It also contains Git commit commands.

03 - Export SQL

scaffolding/03-export-sql.ps1 uses Windows integrated authentication, queries SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative counts.

04 - Validate source

scripts/validate_source.py enforces the four-field schema, IDs, taxonomy, Boolean labels, non-empty values, and ID uniqueness.

05 - Create splits

scripts/create_splits.py creates deterministic stratified 80/10/10 partitions using seed 20260726 and verifies that every input ID appears exactly once.

06 - Build SFT

scripts/build_sft.py loads config/system_prompt.txt, builds the exact target JSON, and overwrites all three split files with chat conversations. Always run stage 05 immediately before rerunning stage 06.

07 - Base-model smoke test

scripts/model_smoke_test.py loads one training example, tokenizer/template, and the 4-bit base model; performs a forward pass; and rejects non-finite logits. It does not test the adapter or merged model.

08 - Pilot training

scripts/train_pilot.py selects the first up-to-64 training records and runs 10 steps. It deletes/recreates outputs/pilot-adapter.

09 - Full training

scripts/train.py uses:

  • NF4 4-bit base weights with double quantization and BF16 compute;
  • LoRA rank 16, alpha 32, dropout 0.05, target all-linear;
  • completion-only loss over the assistant response;
  • gradient checkpointing;
  • batch size 2 and accumulation 8;
  • 2 epochs, learning rate 0.0002, 100 warmup steps;
  • epoch evaluation and checkpointing;
  • lowest validation loss as the best saved checkpoint.

It deletes/recreates outputs/final-adapter. There is no early-stopping callback.

10 - Evaluate

scripts/evaluate.py loads the base model plus final adapter and generates all development outputs in deterministic batches of eight. Generation stops at EOS, end-of-turn, or a complete balanced JSON object. Results are atomically written to outputs/development-evaluation.

Metrics cover JSON/schema validity, classification confusion matrix and precision/recall/F1, result count, category, text, and evidence.

Rescore without inference

scripts/rescore_evaluation.py reparses existing raw outputs, annotates the prediction records, and rewrites predictions/summary without rerunning a model.

11 - Verify generation stopping

scripts/verify_generation_stop.py generates the first five development examples and requires each complete output to parse with json.loads. Balanced braces trigger stopping; schema validation remains a later responsibility.

12 - Merge

scripts/merge_model.py loads unquantized BF16 base weights on CPU, merges the final adapter, saves safe-serialized 5 GB shards to a temporary directory, verifies key files, and atomically promotes the directory to outputs/Invited-PHI-8B.

This stage creates a standalone Hugging Face model, not GGUF.

Chat contract

config/chat_template.jinja uses Llama-style role headers and end-of-turn tokens. config/system_prompt.txt requires exactly one JSON object:

{
  "is_medical_phi": true,
  "score": 1.0,
  "reason": "...",
  "results": [
    {
      "entity_type": "MEDICAL_DIAGNOSIS",
      "text": "...",
      "score": 1.0,
      "evidence": "..."
    }
  ]
}

No Markdown, extra prose, inferred facts, or non-MEDICAL_* types are allowed.

Evaluation caveats

  • A valid JSON prefix followed by trailing text can count as parseable in the evaluator/rescorer because raw_decode is used.
  • Entity comparisons use the first expected and predicted result.
  • Entity accuracy denominators are all positive records.
  • Exact whole-input text/evidence reproduction is measured, not span overlap.
  • Result-count correctness is reported separately from schema validity.
  • Evaluation covers base-plus-adapter, not the merged standalone model.

Path caveat

All .sh wrappers use /d/ai/training/openbiollm-phi, while this checkout may be elsewhere (for example /d/ai/training/Models-Phi-8B). Fix the PROJECT_ROOT line in every wrapper or invoke the underlying uv run python commands from the actual repository root.