Go to file
GAJ3T 55065d88ff added mmd and pdf 2026-10-15 10:38:18 -04:00
PHI.Data.Generator consolidating Generator into this repo 2026-10-13 15:54:55 -04:00
config init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
data init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
docs added mmd and pdf 2026-10-15 10:38:18 -04:00
outputs init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
scaffolding init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
scripts init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
.gitignore Initial commit 2026-09-21 14:54:07 +00:00
.python-version init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
README.md updated docs 2026-10-15 10:32:34 -04:00
pyproject.toml init... needs .venv from requirements 2026-09-21 12:55:10 -04:00
uv.lock init... needs .venv from requirements 2026-09-21 12:55:10 -04:00

README.md

Medical PHI detection model

This repository contains two connected systems:

  1. A .NET generator that combines curated medical terminology with positive and negative language patterns, normalizes the results through an OpenAI-compatible LM Studio endpoint, and persists labeled examples to SQL Server.
  2. A Python/WSL training pipeline that exports those examples, validates and splits them, converts them to supervised fine-tuning (SFT) conversations, trains and evaluates a QLoRA adapter for Llama3-OpenBioLLM-8B, and merges the adapter into a standalone Hugging Face model.

The repository also contains generated data, trained adapters, a merged model, evaluation results, and GGUF artifacts. The tracked automation creates the Hugging Face model; a GGUF conversion command is not currently tracked.

Documentation

Start with the documentation index.

Important semantics

  • Category is the label/taxonomy term used throughout this repository. The names are compatible with the intended Presidio extension/override, but this repository does not contain the Presidio integration itself.
  • A pattern with [sub] receives a shuffled subject (He, She, They, or Member). A pattern without [sub] is a subjectless fragment. The LM prompt treats that fragment as having an implied singular member/patient for grammar only; it must not insert that subject into the output.
  • Positive and negative patterns teach contextual classification. Medical words can therefore appear in both classes: a real medical assertion is positive, while educational, fictional, historical, quoted, or unrelated use can be negative.
  • The medical terminology catalogs were assembled from NIH, WHO, and CDC data, according to the project owner. See the provenance documentation for the verification and licensing work still required.

Quick start

The complete, ordered procedure is in the runbook. Do not run the numbered shell wrappers without first correcting their legacy hard-coded project path.