114 lines
5.3 KiB
Markdown
114 lines
5.3 KiB
Markdown
# Architecture
|
|
|
|
## Goals
|
|
|
|
The system produces a domain-adapted model that classifies contextual medical
|
|
PHI and returns a strict JSON response. Its principal design goals are:
|
|
|
|
- create reproducible positive and negative training examples;
|
|
- preserve a controlled 12-category medical taxonomy;
|
|
- use contextual negatives so keywords alone do not determine classification;
|
|
- durably generate large datasets with resumable SQL persistence;
|
|
- fine-tune an 8B model on commodity GPU hardware through QLoRA;
|
|
- evaluate JSON validity, classification, and category extraction separately;
|
|
- publish both an adapter and a standalone merged model.
|
|
|
|
## System boundary
|
|
|
|
Inside the repository are:
|
|
|
|
- the .NET 10 generator library, API, EF Core migrations, catalogs, and tests;
|
|
- curated category values, grammar variants, subjects, and positive/negative
|
|
patterns;
|
|
- Python 3.12 data preparation, training, evaluation, and merge scripts;
|
|
- PowerShell/Bash orchestration scripts;
|
|
- generated JSONL data and model/evaluation artifacts.
|
|
|
|
External dependencies are:
|
|
|
|
- SQL Server (`RLHFGenerator`);
|
|
- LM Studio exposing an OpenAI-compatible chat completions endpoint;
|
|
- the `qwen/qwen3.5-9b` normalization model configured in the API;
|
|
- the local `Llama3-OpenBioLLM-8B` base model;
|
|
- WSL/Linux, CUDA 12.8-compatible PyTorch, and a BF16-capable NVIDIA GPU;
|
|
- NIH, WHO, and CDC as owner-reported origins of the terminology catalogs.
|
|
|
|
Microsoft Presidio is an intended downstream integration, not a runtime
|
|
component of this repository.
|
|
|
|
## End-to-end data flow
|
|
|
|
1. The generator validates 12 category files, 24 polarity-specific pattern
|
|
files, and the exact four-subject catalog.
|
|
2. For each category and polarity, it deterministically shuffles values,
|
|
patterns, subjects, and tenses using the configured random seed.
|
|
3. It substitutes protected medical values, resolves some grammar locally, and
|
|
asks LM Studio to normalize remaining expressions without changing meaning or
|
|
polarity.
|
|
4. It validates one-line output and writes ordered `TrainingItem` rows to SQL
|
|
Server in atomic batches, updating a durable generation checkpoint in the
|
|
same transaction.
|
|
5. PowerShell exports `TrainingItem` plus `Category` to `data/source.jsonl`.
|
|
6. Python validates the schema and taxonomy, then deterministically stratifies
|
|
by `(category, is_positive)` into train, validation, and development sets.
|
|
7. The SFT builder replaces each split record with a three-message conversation
|
|
containing the system policy, input text, and exact target JSON.
|
|
8. Pilot and full training load the base model in 4-bit NF4 and train LoRA
|
|
adapters using completion-only loss.
|
|
9. Evaluation loads the base model plus final adapter and generates development
|
|
predictions deterministically.
|
|
10. The merge step loads the unquantized base model on CPU, merges the adapter,
|
|
and saves a BF16 Hugging Face model.
|
|
|
|
## Persistence model
|
|
|
|
| Entity | Responsibility |
|
|
|---|---|
|
|
| `Category` | Stable seeded category ID/name |
|
|
| `Keyword` | Canonical category-specific medical value |
|
|
| `KeywordVariant` | Alternate grammatical/surface form |
|
|
| `GenerationRun` | Seed, fingerprints, status, checkpoint, counts, errors |
|
|
| `TrainingItem` | Generated text, label, category, optional keyword, plan ordinal |
|
|
|
|
Important constraints include unique category names, unique canonical values
|
|
within a category, unique variants within a keyword, unique training record
|
|
keys, and unique `(GenerationRunId, PlanOrdinal)`.
|
|
|
|
## Reliability model
|
|
|
|
- **Deterministic plan:** `System.Random` plus seed 42 creates a repeatable plan
|
|
for unchanged inputs and configuration.
|
|
- **External-model caveat:** repeatable planning does not guarantee byte-identical
|
|
LM Studio responses if model files, runtime, or service behavior change.
|
|
- **Fingerprinting:** generation hashes source files and effective settings;
|
|
changed inputs/settings reject an unsafe resume.
|
|
- **Ordered parallelism:** generation is concurrent, but a single ordered writer
|
|
buffers out-of-order results.
|
|
- **Atomic checkpoints:** inserted rows and the checkpoint advance commit in the
|
|
same database transaction.
|
|
- **Fail closed:** invalid LM output, request failures, persistence failures,
|
|
cancellation, mismatched durable state, or ambiguous incomplete runs fail the
|
|
operation rather than silently producing success-shaped output.
|
|
|
|
## Classification model
|
|
|
|
The model is trained as a binary classifier plus category reporter:
|
|
|
|
- A positive SFT target has `is_medical_phi: true`, score `1.0`, a single result,
|
|
and an entity type equal to one of the 12 categories.
|
|
- A negative SFT target has `is_medical_phi: false`, score `0.0`, and no results.
|
|
- Positive targets currently mark the entire input as `text` and `evidence`.
|
|
This is classification with whole-input evidence, not span extraction.
|
|
|
|
## Known architecture gaps
|
|
|
|
- No tracked Presidio recognizer, registry, mapping, or deployment integration.
|
|
- No tracked GGUF conversion or quantization workflow, although GGUF files are
|
|
present.
|
|
- Shell scaffolds have a legacy hard-coded checkout path.
|
|
- The SFT conversion overwrites split files and is not idempotent.
|
|
- The generator depends on an external normalization model whose exact artifact
|
|
version is not fingerprinted.
|
|
- Terminology source URLs, versions, retrieval dates, transformations, and
|
|
licenses are not captured in a machine-readable manifest.
|