153 lines
5.1 KiB
Markdown
153 lines
5.1 KiB
Markdown
# Training pipeline
|
|
|
|
## Runtime
|
|
|
|
- Python `3.12.*` (the setup script installs 3.12.13)
|
|
- `uv` for interpreter/dependency management
|
|
- PyTorch 2.11.0 from the CUDA 12.8 index
|
|
- Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1
|
|
- bitsandbytes 0.49.2 for 4-bit NF4 loading
|
|
- CUDA-capable NVIDIA GPU with BF16 support
|
|
- WSL/Linux for shell wrappers
|
|
- substantial CPU RAM/disk for BF16 merge
|
|
|
|
The canonical configuration is `config/pipeline.json`.
|
|
|
|
## Pipeline stages
|
|
|
|
### 00 - Audit
|
|
|
|
`scaffolding/00-audit.sh` checks Linux/WSL, Python, Git, compiler/build tools,
|
|
download tools, NVIDIA utilities, memory, disk, base model, and repository state.
|
|
|
|
### 01 - Create environment
|
|
|
|
`scaffolding/01-create-venv.sh` installs `uv`, installs Python 3.12.13, and
|
|
recreates `.venv`. It also attempts to stage and commit files, so inspect/remove
|
|
those Git commands before using it in an automated or shared worktree.
|
|
|
|
### 02 - Install dependencies
|
|
|
|
`scaffolding/02-install-dependencies.sh` locks/synchronizes dependencies and
|
|
fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime
|
|
information. It also contains Git commit commands.
|
|
|
|
### 03 - Export SQL
|
|
|
|
`scaffolding/03-export-sql.ps1` uses Windows integrated authentication, queries
|
|
SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative
|
|
counts.
|
|
|
|
### 04 - Validate source
|
|
|
|
`scripts/validate_source.py` enforces the four-field schema, IDs, taxonomy,
|
|
Boolean labels, non-empty values, and ID uniqueness.
|
|
|
|
### 05 - Create splits
|
|
|
|
`scripts/create_splits.py` creates deterministic stratified 80/10/10 partitions
|
|
using seed `20260726` and verifies that every input ID appears exactly once.
|
|
|
|
### 06 - Build SFT
|
|
|
|
`scripts/build_sft.py` loads `config/system_prompt.txt`, builds the exact target
|
|
JSON, and overwrites all three split files with chat conversations. Always run
|
|
stage 05 immediately before rerunning stage 06.
|
|
|
|
### 07 - Base-model smoke test
|
|
|
|
`scripts/model_smoke_test.py` loads one training example, tokenizer/template, and
|
|
the 4-bit base model; performs a forward pass; and rejects non-finite logits.
|
|
It does not test the adapter or merged model.
|
|
|
|
### 08 - Pilot training
|
|
|
|
`scripts/train_pilot.py` selects the first up-to-64 training records and runs 10
|
|
steps. It deletes/recreates `outputs/pilot-adapter`.
|
|
|
|
### 09 - Full training
|
|
|
|
`scripts/train.py` uses:
|
|
|
|
- NF4 4-bit base weights with double quantization and BF16 compute;
|
|
- LoRA rank 16, alpha 32, dropout 0.05, target `all-linear`;
|
|
- completion-only loss over the assistant response;
|
|
- gradient checkpointing;
|
|
- batch size 2 and accumulation 8;
|
|
- 2 epochs, learning rate 0.0002, 100 warmup steps;
|
|
- epoch evaluation and checkpointing;
|
|
- lowest validation loss as the best saved checkpoint.
|
|
|
|
It deletes/recreates `outputs/final-adapter`. There is no early-stopping
|
|
callback.
|
|
|
|
### 10 - Evaluate
|
|
|
|
`scripts/evaluate.py` loads the base model plus final adapter and generates all
|
|
development outputs in deterministic batches of eight. Generation stops at EOS,
|
|
end-of-turn, or a complete balanced JSON object. Results are atomically written
|
|
to `outputs/development-evaluation`.
|
|
|
|
Metrics cover JSON/schema validity, classification confusion matrix and
|
|
precision/recall/F1, result count, category, text, and evidence.
|
|
|
|
### Rescore without inference
|
|
|
|
`scripts/rescore_evaluation.py` reparses existing raw outputs, annotates the
|
|
prediction records, and rewrites predictions/summary without rerunning a model.
|
|
|
|
### 11 - Verify generation stopping
|
|
|
|
`scripts/verify_generation_stop.py` generates the first five development
|
|
examples and requires each complete output to parse with `json.loads`. Balanced
|
|
braces trigger stopping; schema validation remains a later responsibility.
|
|
|
|
### 12 - Merge
|
|
|
|
`scripts/merge_model.py` loads unquantized BF16 base weights on CPU, merges the
|
|
final adapter, saves safe-serialized 5 GB shards to a temporary directory,
|
|
verifies key files, and atomically promotes the directory to
|
|
`outputs/Invited-PHI-8B`.
|
|
|
|
This stage creates a standalone Hugging Face model, not GGUF.
|
|
|
|
## Chat contract
|
|
|
|
`config/chat_template.jinja` uses Llama-style role headers and end-of-turn
|
|
tokens. `config/system_prompt.txt` requires exactly one JSON object:
|
|
|
|
```json
|
|
{
|
|
"is_medical_phi": true,
|
|
"score": 1.0,
|
|
"reason": "...",
|
|
"results": [
|
|
{
|
|
"entity_type": "MEDICAL_DIAGNOSIS",
|
|
"text": "...",
|
|
"score": 1.0,
|
|
"evidence": "..."
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
No Markdown, extra prose, inferred facts, or non-`MEDICAL_*` types are allowed.
|
|
|
|
## Evaluation caveats
|
|
|
|
- A valid JSON prefix followed by trailing text can count as parseable in the
|
|
evaluator/rescorer because `raw_decode` is used.
|
|
- Entity comparisons use the first expected and predicted result.
|
|
- Entity accuracy denominators are all positive records.
|
|
- Exact whole-input text/evidence reproduction is measured, not span overlap.
|
|
- Result-count correctness is reported separately from schema validity.
|
|
- Evaluation covers base-plus-adapter, not the merged standalone model.
|
|
|
|
## Path caveat
|
|
|
|
All `.sh` wrappers use `/d/ai/training/openbiollm-phi`, while this checkout may
|
|
be elsewhere (for example `/d/ai/training/Models-Phi-8B`). Fix the `PROJECT_ROOT`
|
|
line in every wrapper or invoke the underlying `uv run python` commands from the
|
|
actual repository root.
|