Models-Phi-8B/README.md

55 lines
2.5 KiB
Markdown

# Medical PHI detection model
This repository contains two connected systems:
1. A .NET generator that combines curated medical terminology with positive and
negative language patterns, normalizes the results through an
OpenAI-compatible LM Studio endpoint, and persists labeled examples to SQL
Server.
2. A Python/WSL training pipeline that exports those examples, validates and
splits them, converts them to supervised fine-tuning (SFT) conversations,
trains and evaluates a QLoRA adapter for `Llama3-OpenBioLLM-8B`, and merges
the adapter into a standalone Hugging Face model.
The repository also contains generated data, trained adapters, a merged model,
evaluation results, and GGUF artifacts. The tracked automation creates the
Hugging Face model; a GGUF conversion command is not currently tracked.
## Documentation
Start with [the documentation index](docs/README.md).
- [Architecture](docs/architecture.md)
- [Generator](docs/generator.md)
- [Data and provenance](docs/data-and-provenance.md)
- [Training pipeline](docs/training-pipeline.md)
- [API reference](docs/api-reference.md)
- [Testing and validation](docs/testing-and-validation.md)
- [Step-by-step runbook](docs/runbook.md)
- [Terminology and verified claims](docs/claims-and-terminology.md)
- [Artifacts and repository map](docs/artifacts.md)
- [Editable C4 diagrams](docs/diagrams/)
## Important semantics
- `Category` is the label/taxonomy term used throughout this repository. The
names are compatible with the intended Presidio extension/override, but this
repository does not contain the Presidio integration itself.
- A pattern with `[sub]` receives a shuffled subject (`He`, `She`, `They`, or
`Member`). A pattern without `[sub]` is a subjectless fragment. The LM prompt
treats that fragment as having an implied singular member/patient for grammar
only; it must not insert that subject into the output.
- Positive and negative patterns teach contextual classification. Medical words
can therefore appear in both classes: a real medical assertion is positive,
while educational, fictional, historical, quoted, or unrelated use can be
negative.
- The medical terminology catalogs were assembled from NIH, WHO, and CDC data,
according to the project owner. See the provenance documentation for the
verification and licensing work still required.
## Quick start
The complete, ordered procedure is in [the runbook](docs/runbook.md). Do not run
the numbered shell wrappers without first correcting their legacy hard-coded
project path.