# Training pipeline ## Runtime - Python `3.12.*` (the setup script installs 3.12.13) - `uv` for interpreter/dependency management - PyTorch 2.11.0 from the CUDA 12.8 index - Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1 - bitsandbytes 0.49.2 for 4-bit NF4 loading - CUDA-capable NVIDIA GPU with BF16 support - WSL/Linux for shell wrappers - substantial CPU RAM/disk for BF16 merge The canonical configuration is `config/pipeline.json`. ## Pipeline stages ### 00 - Audit `scaffolding/00-audit.sh` checks Linux/WSL, Python, Git, compiler/build tools, download tools, NVIDIA utilities, memory, disk, base model, and repository state. ### 01 - Create environment `scaffolding/01-create-venv.sh` installs `uv`, installs Python 3.12.13, and recreates `.venv`. It also attempts to stage and commit files, so inspect/remove those Git commands before using it in an automated or shared worktree. ### 02 - Install dependencies `scaffolding/02-install-dependencies.sh` locks/synchronizes dependencies and fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime information. It also contains Git commit commands. ### 03 - Export SQL `scaffolding/03-export-sql.ps1` uses Windows integrated authentication, queries SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative counts. ### 04 - Validate source `scripts/validate_source.py` enforces the four-field schema, IDs, taxonomy, Boolean labels, non-empty values, and ID uniqueness. ### 05 - Create splits `scripts/create_splits.py` creates deterministic stratified 80/10/10 partitions using seed `20260726` and verifies that every input ID appears exactly once. ### 06 - Build SFT `scripts/build_sft.py` loads `config/system_prompt.txt`, builds the exact target JSON, and overwrites all three split files with chat conversations. Always run stage 05 immediately before rerunning stage 06. ### 07 - Base-model smoke test `scripts/model_smoke_test.py` loads one training example, tokenizer/template, and the 4-bit base model; performs a forward pass; and rejects non-finite logits. It does not test the adapter or merged model. ### 08 - Pilot training `scripts/train_pilot.py` selects the first up-to-64 training records and runs 10 steps. It deletes/recreates `outputs/pilot-adapter`. ### 09 - Full training `scripts/train.py` uses: - NF4 4-bit base weights with double quantization and BF16 compute; - LoRA rank 16, alpha 32, dropout 0.05, target `all-linear`; - completion-only loss over the assistant response; - gradient checkpointing; - batch size 2 and accumulation 8; - 2 epochs, learning rate 0.0002, 100 warmup steps; - epoch evaluation and checkpointing; - lowest validation loss as the best saved checkpoint. It deletes/recreates `outputs/final-adapter`. There is no early-stopping callback. ### 10 - Evaluate `scripts/evaluate.py` loads the base model plus final adapter and generates all development outputs in deterministic batches of eight. Generation stops at EOS, end-of-turn, or a complete balanced JSON object. Results are atomically written to `outputs/development-evaluation`. Metrics cover JSON/schema validity, classification confusion matrix and precision/recall/F1, result count, category, text, and evidence. ### Rescore without inference `scripts/rescore_evaluation.py` reparses existing raw outputs, annotates the prediction records, and rewrites predictions/summary without rerunning a model. ### 11 - Verify generation stopping `scripts/verify_generation_stop.py` generates the first five development examples and requires each complete output to parse with `json.loads`. Balanced braces trigger stopping; schema validation remains a later responsibility. ### 12 - Merge `scripts/merge_model.py` loads unquantized BF16 base weights on CPU, merges the final adapter, saves safe-serialized 5 GB shards to a temporary directory, verifies key files, and atomically promotes the directory to `outputs/Invited-PHI-8B`. This stage creates a standalone Hugging Face model, not GGUF. ## Chat contract `config/chat_template.jinja` uses Llama-style role headers and end-of-turn tokens. `config/system_prompt.txt` requires exactly one JSON object: ```json { "is_medical_phi": true, "score": 1.0, "reason": "...", "results": [ { "entity_type": "MEDICAL_DIAGNOSIS", "text": "...", "score": 1.0, "evidence": "..." } ] } ``` No Markdown, extra prose, inferred facts, or non-`MEDICAL_*` types are allowed. ## Evaluation caveats - A valid JSON prefix followed by trailing text can count as parseable in the evaluator/rescorer because `raw_decode` is used. - Entity comparisons use the first expected and predicted result. - Entity accuracy denominators are all positive records. - Exact whole-input text/evidence reproduction is measured, not span overlap. - Result-count correctness is reported separately from schema validity. - Evaluation covers base-plus-adapter, not the merged standalone model. ## Path caveat All `.sh` wrappers use `/d/ai/training/openbiollm-phi`, while this checkout may be elsewhere (for example `/d/ai/training/Models-Phi-8B`). Fix the `PROJECT_ROOT` line in every wrapper or invoke the underlying `uv run python` commands from the actual repository root.