5.3 KiB
5.3 KiB
Architecture
Goals
The system produces a domain-adapted model that classifies contextual medical PHI and returns a strict JSON response. Its principal design goals are:
- create reproducible positive and negative training examples;
- preserve a controlled 12-category medical taxonomy;
- use contextual negatives so keywords alone do not determine classification;
- durably generate large datasets with resumable SQL persistence;
- fine-tune an 8B model on commodity GPU hardware through QLoRA;
- evaluate JSON validity, classification, and category extraction separately;
- publish both an adapter and a standalone merged model.
System boundary
Inside the repository are:
- the .NET 10 generator library, API, EF Core migrations, catalogs, and tests;
- curated category values, grammar variants, subjects, and positive/negative patterns;
- Python 3.12 data preparation, training, evaluation, and merge scripts;
- PowerShell/Bash orchestration scripts;
- generated JSONL data and model/evaluation artifacts.
External dependencies are:
- SQL Server (
RLHFGenerator); - LM Studio exposing an OpenAI-compatible chat completions endpoint;
- the
qwen/qwen3.5-9bnormalization model configured in the API; - the local
Llama3-OpenBioLLM-8Bbase model; - WSL/Linux, CUDA 12.8-compatible PyTorch, and a BF16-capable NVIDIA GPU;
- NIH, WHO, and CDC as owner-reported origins of the terminology catalogs.
Microsoft Presidio is an intended downstream integration, not a runtime component of this repository.
End-to-end data flow
- The generator validates 12 category files, 24 polarity-specific pattern files, and the exact four-subject catalog.
- For each category and polarity, it deterministically shuffles values, patterns, subjects, and tenses using the configured random seed.
- It substitutes protected medical values, resolves some grammar locally, and asks LM Studio to normalize remaining expressions without changing meaning or polarity.
- It validates one-line output and writes ordered
TrainingItemrows to SQL Server in atomic batches, updating a durable generation checkpoint in the same transaction. - PowerShell exports
TrainingItemplusCategorytodata/source.jsonl. - Python validates the schema and taxonomy, then deterministically stratifies
by
(category, is_positive)into train, validation, and development sets. - The SFT builder replaces each split record with a three-message conversation containing the system policy, input text, and exact target JSON.
- Pilot and full training load the base model in 4-bit NF4 and train LoRA adapters using completion-only loss.
- Evaluation loads the base model plus final adapter and generates development predictions deterministically.
- The merge step loads the unquantized base model on CPU, merges the adapter, and saves a BF16 Hugging Face model.
Persistence model
| Entity | Responsibility |
|---|---|
Category |
Stable seeded category ID/name |
Keyword |
Canonical category-specific medical value |
KeywordVariant |
Alternate grammatical/surface form |
GenerationRun |
Seed, fingerprints, status, checkpoint, counts, errors |
TrainingItem |
Generated text, label, category, optional keyword, plan ordinal |
Important constraints include unique category names, unique canonical values
within a category, unique variants within a keyword, unique training record
keys, and unique (GenerationRunId, PlanOrdinal).
Reliability model
- Deterministic plan:
System.Randomplus seed 42 creates a repeatable plan for unchanged inputs and configuration. - External-model caveat: repeatable planning does not guarantee byte-identical LM Studio responses if model files, runtime, or service behavior change.
- Fingerprinting: generation hashes source files and effective settings; changed inputs/settings reject an unsafe resume.
- Ordered parallelism: generation is concurrent, but a single ordered writer buffers out-of-order results.
- Atomic checkpoints: inserted rows and the checkpoint advance commit in the same database transaction.
- Fail closed: invalid LM output, request failures, persistence failures, cancellation, mismatched durable state, or ambiguous incomplete runs fail the operation rather than silently producing success-shaped output.
Classification model
The model is trained as a binary classifier plus category reporter:
- A positive SFT target has
is_medical_phi: true, score1.0, a single result, and an entity type equal to one of the 12 categories. - A negative SFT target has
is_medical_phi: false, score0.0, and no results. - Positive targets currently mark the entire input as
textandevidence. This is classification with whole-input evidence, not span extraction.
Known architecture gaps
- No tracked Presidio recognizer, registry, mapping, or deployment integration.
- No tracked GGUF conversion or quantization workflow, although GGUF files are present.
- Shell scaffolds have a legacy hard-coded checkout path.
- The SFT conversion overwrites split files and is not idempotent.
- The generator depends on an external normalization model whose exact artifact version is not fingerprinted.
- Terminology source URLs, versions, retrieval dates, transformations, and licenses are not captured in a machine-readable manifest.