Models-Phi-8B/docs/artifacts.md

3.0 KiB

Artifacts and repository map

Source tree

.
|-- PHI.Data.Generator/
|   |-- RLHF.Generator/          Core .NET generator and Data catalogs
|   |-- RLHF.Generator.Api/      ASP.NET Core API
|   |-- RLHF.Generator.Tests/    xUnit tests
|   |-- AddKeywordCatalog.sql
|   `-- RLHF.Generator.slnx
|-- config/                      Training, prompt, and chat-template config
|-- data/                        Exported and transformed JSONL
|-- scaffolding/                 Ordered operational wrappers
|-- scripts/                     Python pipeline implementation
|-- outputs/                     Adapters, evaluation, merged model, GGUF files
|-- docs/                        Project documentation and draw.io diagrams
|-- pyproject.toml
`-- uv.lock

Configuration

Path Purpose
PHI.Data.Generator/RLHF.Generator.Api/appsettings.json SQL, LM Studio, generation defaults
config/pipeline.json SQL export metadata, paths, splits, base model, LoRA/training/merge
config/system_prompt.txt Inference/SFT output contract
config/chat_template.jinja Llama-style chat serialization
pyproject.toml / uv.lock Python requirements and resolved environment

Data artifacts

Path State
data/source.jsonl Source-shaped SQL export
data/train.jsonl Source-shaped after split; SFT-shaped after build
data/validation.jsonl Source-shaped after split; SFT-shaped after build
data/development.jsonl Source-shaped after split; SFT-shaped after build

Because the same split paths change schema in place, record stage/checksums when archiving or automating the pipeline.

Model artifacts

Path Meaning
outputs/pilot-adapter Small pilot LoRA adapter
outputs/final-adapter Full LoRA adapter and retained checkpoints
outputs/development-evaluation Predictions and summary metrics
outputs/Invited-PHI-8B Standalone merged BF16 Hugging Face model
outputs/gguf BF16/Q4_K_M/Q5_K_M GGUF files

The GGUF artifacts have no corresponding tracked conversion script, flags, tool revision, or verification step. They cannot be reproduced from repository automation alone.

Reproducibility boundary

Tracked:

  • source code, catalogs, patterns, seeds, prompts, configurations;
  • Python lock file;
  • generator tests;
  • current data/model/evaluation artifacts.

Not fully tracked:

  • exact NIH/WHO/CDC source datasets and transformation history;
  • LM Studio model checksum/runtime build;
  • base model checksum/revision;
  • SQL Server contents before export;
  • host drivers/CUDA/toolchain snapshot;
  • GGUF conversion/quantization commands;
  • downstream Presidio implementation.

Large files and source control

The repository tracks multi-gigabyte model/training artifacts. For sustainable operation, consider an artifact registry or Git LFS with immutable versioning, checksums, retention, access control, and release metadata. Keep source, configuration, metrics, and artifact manifests in Git even if weights move to external storage.