updated docs

This commit is contained in:
GAJ3T 2026-10-15 10:32:34 -04:00
parent 2f5686c3a1
commit 42ee4c2605
15 changed files with 1571 additions and 2 deletions

View File

@ -1,3 +1,54 @@
# Model for flagging and redacting PHI
# Medical PHI detection model
Simple, repeatable pipeline for exporting labeled "Generator" created SQL Server data, fine-tuning / extending Llama3-OpenBioLLM-8B, evaluating it, and producing a GGUF model.
This repository contains two connected systems:
1. A .NET generator that combines curated medical terminology with positive and
negative language patterns, normalizes the results through an
OpenAI-compatible LM Studio endpoint, and persists labeled examples to SQL
Server.
2. A Python/WSL training pipeline that exports those examples, validates and
splits them, converts them to supervised fine-tuning (SFT) conversations,
trains and evaluates a QLoRA adapter for `Llama3-OpenBioLLM-8B`, and merges
the adapter into a standalone Hugging Face model.
The repository also contains generated data, trained adapters, a merged model,
evaluation results, and GGUF artifacts. The tracked automation creates the
Hugging Face model; a GGUF conversion command is not currently tracked.
## Documentation
Start with [the documentation index](docs/README.md).
- [Architecture](docs/architecture.md)
- [Generator](docs/generator.md)
- [Data and provenance](docs/data-and-provenance.md)
- [Training pipeline](docs/training-pipeline.md)
- [API reference](docs/api-reference.md)
- [Testing and validation](docs/testing-and-validation.md)
- [Step-by-step runbook](docs/runbook.md)
- [Terminology and verified claims](docs/claims-and-terminology.md)
- [Artifacts and repository map](docs/artifacts.md)
- [Editable C4 diagrams](docs/diagrams/)
## Important semantics
- `Category` is the label/taxonomy term used throughout this repository. The
names are compatible with the intended Presidio extension/override, but this
repository does not contain the Presidio integration itself.
- A pattern with `[sub]` receives a shuffled subject (`He`, `She`, `They`, or
`Member`). A pattern without `[sub]` is a subjectless fragment. The LM prompt
treats that fragment as having an implied singular member/patient for grammar
only; it must not insert that subject into the output.
- Positive and negative patterns teach contextual classification. Medical words
can therefore appear in both classes: a real medical assertion is positive,
while educational, fictional, historical, quoted, or unrelated use can be
negative.
- The medical terminology catalogs were assembled from NIH, WHO, and CDC data,
according to the project owner. See the provenance documentation for the
verification and licensing work still required.
## Quick start
The complete, ordered procedure is in [the runbook](docs/runbook.md). Do not run
the numbered shell wrappers without first correcting their legacy hard-coded
project path.

64
docs/README.md Normal file
View File

@ -0,0 +1,64 @@
# Documentation index
## System overview
This project creates a medical-context PHI classifier in two stages:
```text
NIH/WHO/CDC-derived terminology + curated patterns
|
v
.NET generator
|
v
SQL Server TrainingItem
|
v
export -> validate -> split -> SFT
|
v
QLoRA train and evaluate
|
v
adapter -> merged Hugging Face model
```
The generator creates labeled natural-language examples. The Python pipeline
uses those examples to teach a model to return a strict JSON decision describing
whether the input contains medical PHI and, for positives, its category.
## Documents
| Document | Purpose |
|---|---|
| [Architecture](architecture.md) | System boundaries, data flow, persistence, and design decisions |
| [Generator](generator.md) | Catalogs, patterns, planning, randomization, LM normalization, durability |
| [Data and provenance](data-and-provenance.md) | Schemas, categories, terminology sources, lineage, and governance |
| [Training pipeline](training-pipeline.md) | Every Python script and scaffold step |
| [API reference](api-reference.md) | Generator HTTP endpoints, contracts, and failure behavior |
| [Testing and validation](testing-and-validation.md) | .NET tests, Python verification, evaluation semantics, current metrics |
| [Runbook](runbook.md) | Prerequisites and complete operator procedure |
| [Terminology and verified claims](claims-and-terminology.md) | Corrections and precise wording for the project explanation |
| [Artifacts and repository map](artifacts.md) | Source tree, inputs, outputs, and reproducibility boundaries |
## C4 diagrams
The editable draw.io documents are:
- [Level 1: System context](diagrams/01-system-context.drawio)
- [Level 2: Containers](diagrams/02-containers.drawio)
- [Level 3: Components](diagrams/03-generator-components.drawio)
- [Level 4: Code](diagrams/04-generator-code.drawio)
Open the files with diagrams.net, the draw.io desktop application, or the VS
Code Draw.io Integration extension.
## Documentation conventions
- **Verified** means behavior is directly represented by tracked code, tests, or
configuration.
- **Owner-provided provenance** means the fact was supplied by the project owner
but source URLs, versions, retrieval dates, and licenses are not tracked here.
- Paths in commands are examples. The checked-in shell wrappers still use the
legacy WSL path `/d/ai/training/openbiollm-phi`; update them before use if the
checkout is elsewhere.

90
docs/api-reference.md Normal file
View File

@ -0,0 +1,90 @@
# Generator API reference
## Hosting
Run the ASP.NET Core .NET 10 project
`PHI.Data.Generator/RLHF.Generator.Api`. Production requires:
- the `RLHFGenerator` SQL Server connection string;
- migrated database schema;
- copied generator `Data` files;
- LM Studio listening at the configured OpenAI-compatible endpoint.
Swagger and Scalar are enabled only in the Development environment.
## Endpoints
### `GET /`
Returns a small service status/identity response. Use it as a basic process
health check; it does not prove database or LM Studio readiness.
### `POST /api/lm-studio/complete`
Normalizes one supplied sentence or fragment without persistence.
The request contract supports:
- text/sentence;
- required fragment flag;
- optional protected subject;
- optional protected medical term;
- optional allowed medical forms;
- optional requested tense.
Validation rejects empty input, missing fragment status, invalid tense,
inconsistent protected-term/forms metadata, and duplicate allowed forms.
Successful output is the normalized plain-text line.
### `POST /api/keywords/load`
Loads the category catalogs into `Keyword`/`KeywordVariant`.
- additive;
- case-insensitive;
- idempotent;
- does not backfill existing training items.
### `POST /api/training-items/backfill-keywords`
Matches existing generated text to keywords.
Query behavior includes a `dryRun` option and bounded batch processing. Only a
unique match is assigned. Ambiguous and unmatched records remain null.
### `POST /api/generation/run`
Starts a new compatible generation run or resumes an incomplete one. The HTTP
request waits for generation and all persistence commits to complete; it does
not merely enqueue background work.
The response reports run identity/status and generation counts. Failures and
cancellation update durable run state before surfacing the error.
### `POST /api/training-items/reset`
Clears generated training items and generation checkpoints while preserving the
seeded category rows. Treat this as destructive and restrict it operationally.
## Suggested invocation sequence
```powershell
$base = "http://localhost:<port>"
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
Invoke-RestMethod -Method Post -Uri "$base/api/training-items/backfill-keywords?dryRun=true"
Invoke-RestMethod -Method Post -Uri "$base/api/training-items/backfill-keywords?dryRun=false"
```
Use the actual port shown by `dotnet run`/launch settings.
## Operational controls
- Put authentication/authorization and network controls in front of the API;
the tracked host does not define an application authorization policy.
- Restrict reset and generation endpoints to operators.
- Do not expose the configured SQL Server connection details or LM endpoint.
- Monitor generation progress, LM failures, request duration, committed counts,
and generation-run status.
- Back up the database before reset or migration.

113
docs/architecture.md Normal file
View File

@ -0,0 +1,113 @@
# Architecture
## Goals
The system produces a domain-adapted model that classifies contextual medical
PHI and returns a strict JSON response. Its principal design goals are:
- create reproducible positive and negative training examples;
- preserve a controlled 12-category medical taxonomy;
- use contextual negatives so keywords alone do not determine classification;
- durably generate large datasets with resumable SQL persistence;
- fine-tune an 8B model on commodity GPU hardware through QLoRA;
- evaluate JSON validity, classification, and category extraction separately;
- publish both an adapter and a standalone merged model.
## System boundary
Inside the repository are:
- the .NET 10 generator library, API, EF Core migrations, catalogs, and tests;
- curated category values, grammar variants, subjects, and positive/negative
patterns;
- Python 3.12 data preparation, training, evaluation, and merge scripts;
- PowerShell/Bash orchestration scripts;
- generated JSONL data and model/evaluation artifacts.
External dependencies are:
- SQL Server (`RLHFGenerator`);
- LM Studio exposing an OpenAI-compatible chat completions endpoint;
- the `qwen/qwen3.5-9b` normalization model configured in the API;
- the local `Llama3-OpenBioLLM-8B` base model;
- WSL/Linux, CUDA 12.8-compatible PyTorch, and a BF16-capable NVIDIA GPU;
- NIH, WHO, and CDC as owner-reported origins of the terminology catalogs.
Microsoft Presidio is an intended downstream integration, not a runtime
component of this repository.
## End-to-end data flow
1. The generator validates 12 category files, 24 polarity-specific pattern
files, and the exact four-subject catalog.
2. For each category and polarity, it deterministically shuffles values,
patterns, subjects, and tenses using the configured random seed.
3. It substitutes protected medical values, resolves some grammar locally, and
asks LM Studio to normalize remaining expressions without changing meaning or
polarity.
4. It validates one-line output and writes ordered `TrainingItem` rows to SQL
Server in atomic batches, updating a durable generation checkpoint in the
same transaction.
5. PowerShell exports `TrainingItem` plus `Category` to `data/source.jsonl`.
6. Python validates the schema and taxonomy, then deterministically stratifies
by `(category, is_positive)` into train, validation, and development sets.
7. The SFT builder replaces each split record with a three-message conversation
containing the system policy, input text, and exact target JSON.
8. Pilot and full training load the base model in 4-bit NF4 and train LoRA
adapters using completion-only loss.
9. Evaluation loads the base model plus final adapter and generates development
predictions deterministically.
10. The merge step loads the unquantized base model on CPU, merges the adapter,
and saves a BF16 Hugging Face model.
## Persistence model
| Entity | Responsibility |
|---|---|
| `Category` | Stable seeded category ID/name |
| `Keyword` | Canonical category-specific medical value |
| `KeywordVariant` | Alternate grammatical/surface form |
| `GenerationRun` | Seed, fingerprints, status, checkpoint, counts, errors |
| `TrainingItem` | Generated text, label, category, optional keyword, plan ordinal |
Important constraints include unique category names, unique canonical values
within a category, unique variants within a keyword, unique training record
keys, and unique `(GenerationRunId, PlanOrdinal)`.
## Reliability model
- **Deterministic plan:** `System.Random` plus seed 42 creates a repeatable plan
for unchanged inputs and configuration.
- **External-model caveat:** repeatable planning does not guarantee byte-identical
LM Studio responses if model files, runtime, or service behavior change.
- **Fingerprinting:** generation hashes source files and effective settings;
changed inputs/settings reject an unsafe resume.
- **Ordered parallelism:** generation is concurrent, but a single ordered writer
buffers out-of-order results.
- **Atomic checkpoints:** inserted rows and the checkpoint advance commit in the
same database transaction.
- **Fail closed:** invalid LM output, request failures, persistence failures,
cancellation, mismatched durable state, or ambiguous incomplete runs fail the
operation rather than silently producing success-shaped output.
## Classification model
The model is trained as a binary classifier plus category reporter:
- A positive SFT target has `is_medical_phi: true`, score `1.0`, a single result,
and an entity type equal to one of the 12 categories.
- A negative SFT target has `is_medical_phi: false`, score `0.0`, and no results.
- Positive targets currently mark the entire input as `text` and `evidence`.
This is classification with whole-input evidence, not span extraction.
## Known architecture gaps
- No tracked Presidio recognizer, registry, mapping, or deployment integration.
- No tracked GGUF conversion or quantization workflow, although GGUF files are
present.
- Shell scaffolds have a legacy hard-coded checkout path.
- The SFT conversion overwrites split files and is not idempotent.
- The generator depends on an external normalization model whose exact artifact
version is not fingerprinted.
- Terminology source URLs, versions, retrieval dates, transformations, and
licenses are not captured in a machine-readable manifest.

84
docs/artifacts.md Normal file
View File

@ -0,0 +1,84 @@
# Artifacts and repository map
## Source tree
```text
.
|-- PHI.Data.Generator/
| |-- RLHF.Generator/ Core .NET generator and Data catalogs
| |-- RLHF.Generator.Api/ ASP.NET Core API
| |-- RLHF.Generator.Tests/ xUnit tests
| |-- AddKeywordCatalog.sql
| `-- RLHF.Generator.slnx
|-- config/ Training, prompt, and chat-template config
|-- data/ Exported and transformed JSONL
|-- scaffolding/ Ordered operational wrappers
|-- scripts/ Python pipeline implementation
|-- outputs/ Adapters, evaluation, merged model, GGUF files
|-- docs/ Project documentation and draw.io diagrams
|-- pyproject.toml
`-- uv.lock
```
## Configuration
| Path | Purpose |
|---|---|
| `PHI.Data.Generator/RLHF.Generator.Api/appsettings.json` | SQL, LM Studio, generation defaults |
| `config/pipeline.json` | SQL export metadata, paths, splits, base model, LoRA/training/merge |
| `config/system_prompt.txt` | Inference/SFT output contract |
| `config/chat_template.jinja` | Llama-style chat serialization |
| `pyproject.toml` / `uv.lock` | Python requirements and resolved environment |
## Data artifacts
| Path | State |
|---|---|
| `data/source.jsonl` | Source-shaped SQL export |
| `data/train.jsonl` | Source-shaped after split; SFT-shaped after build |
| `data/validation.jsonl` | Source-shaped after split; SFT-shaped after build |
| `data/development.jsonl` | Source-shaped after split; SFT-shaped after build |
Because the same split paths change schema in place, record stage/checksums when
archiving or automating the pipeline.
## Model artifacts
| Path | Meaning |
|---|---|
| `outputs/pilot-adapter` | Small pilot LoRA adapter |
| `outputs/final-adapter` | Full LoRA adapter and retained checkpoints |
| `outputs/development-evaluation` | Predictions and summary metrics |
| `outputs/Invited-PHI-8B` | Standalone merged BF16 Hugging Face model |
| `outputs/gguf` | BF16/Q4_K_M/Q5_K_M GGUF files |
The GGUF artifacts have no corresponding tracked conversion script, flags,
tool revision, or verification step. They cannot be reproduced from repository
automation alone.
## Reproducibility boundary
Tracked:
- source code, catalogs, patterns, seeds, prompts, configurations;
- Python lock file;
- generator tests;
- current data/model/evaluation artifacts.
Not fully tracked:
- exact NIH/WHO/CDC source datasets and transformation history;
- LM Studio model checksum/runtime build;
- base model checksum/revision;
- SQL Server contents before export;
- host drivers/CUDA/toolchain snapshot;
- GGUF conversion/quantization commands;
- downstream Presidio implementation.
## Large files and source control
The repository tracks multi-gigabyte model/training artifacts. For sustainable
operation, consider an artifact registry or Git LFS with immutable versioning,
checksums, retention, access control, and release metadata. Keep source,
configuration, metrics, and artifact manifests in Git even if weights move to
external storage.

View File

@ -0,0 +1,105 @@
# Terminology and verified claims
## Recommended project explanation
> The generator combines medical category values with curated positive and
> negative contextual patterns. It deterministically shuffles category values,
> patterns, explicit subjects, and requested tenses, then uses LM Studio to
> normalize grammar while preserving the protected medical concept and the
> example's polarity. Patterns without an explicit subject remain subjectless
> fragments; the normalizer assumes an implied singular member/patient only for
> grammar and must not add that subject to the text.
>
> The project uses **Category** as its taxonomy term because the labels are
> intended to integrate with a Presidio category/entity-type extension or
> override. This repository contains the category-based generator and model
> training pipeline, but the Presidio integration itself is maintained
> elsewhere or has not yet been added here.
>
> Positive and negative contexts teach the model that a medical word alone is
> insufficient. “Has diabetes” and “he had a reaction to shellfish” are intended
> positives, while “she wrote a book about diabetes” and “had a reaction to the
> horrible cover of Van Halen's Jump” are intended negatives.
## Review of the original statements
### “The Generator uses random subject, verb, etc.”
**Mostly correct, with precision needed.**
- It shuffles values, patterns, subjects, and a tense deck with a seeded
pseudorandom generator.
- Subjects are `He`, `She`, `They`, and `Member`.
- It does not simply select arbitrary verbs. Patterns supply grammar expressions
and sentence structure; local logic and LM Studio resolve/rewrite auxiliaries
and verb phrases for subject/tense.
- `[x]` starts with the canonical value. Allowed variants are supplied to the
normalizer, rather than directly sampled as an independent random deck.
- Planning is repeatable for unchanged seed/input/configuration; the external LM
service is a remaining reproducibility variable.
### “It creates context for positive and negative tuning examples.”
**Correct.**
Every category has explicit positive and negative pattern files. The label is
stored on each generated item and carried through export, SFT conversion,
training, and evaluation.
### “Categories instead of Nodes/Edges because of Presidio.”
**Reasonable design intent, not verifiable integration in this repository.**
The implementation consistently uses categories/entity types, and that is an
appropriate vocabulary for a Presidio recognizer extension. There is no graph
model in this code, so “nodes/edges” would be misleading. However, no Presidio
package, recognizer, registry, mapping, or runtime call is checked in. Phrase
this as intended compatibility/integration, not as a current in-repository
dependency.
### “Without a subject, or if a sentence fragment, it assumes the member is the subject.”
**Correct only when qualified.**
The implementation marks patterns without `[sub]` as fragments and does not
insert `Member`. The normalizer uses an implied singular member/patient for
agreement and tense only, and must preserve a subjectless output. `Member` is
also one explicit choice in the subject deck when `[sub]` is present.
Preferred wording:
> Subjectless patterns are treated as fragments. For grammar only, the
> normalizer assumes an implied singular member/patient but does not add that
> subject to the generated text.
### Context examples
**They correctly express the intended distinction.**
The medical term or word “reaction” is not enough by itself. Actual patient
medical assertions are positive; authored/educational or idiomatic non-medical
uses are negative. These exact examples should also become automated golden
tests if they are contractual behavior.
## Category versus entity type
- **Category** is the generator/database taxonomy object.
- **Entity type** is the name emitted in model JSON.
- Both currently use the same 12 `MEDICAL_*` identifiers.
- **Keyword** is a canonical medical value assigned to a category.
- **Variant** is an allowed surface/grammatical form of a keyword.
- **Pattern** supplies contextual structure and polarity.
- **Training item** is the resulting labeled text.
## What not to claim
Avoid saying:
- the model is a keyword matcher;
- all category variants are randomly selected;
- `Member` is inserted whenever a subject is absent;
- this repository contains a working Presidio integration;
- the tracked pipeline produces GGUF;
- the current model extracts exact medical spans;
- deterministic planning guarantees identical external LM responses;
- NIH, WHO, or CDC endorse this project.

145
docs/data-and-provenance.md Normal file
View File

@ -0,0 +1,145 @@
# Data and provenance
## Provenance statement
The project owner reports that the medical-term, condition, medication, and
related terminology datasets used to assemble the category catalogs came from
the **U.S. National Institutes of Health (NIH)**, the **World Health
Organization (WHO)**, and the **U.S. Centers for Disease Control and Prevention
(CDC)**.
The repository does not currently include a source manifest proving which
catalog rows came from which organization. It also does not record source URLs,
dataset/product names, versions, retrieval dates, licenses, notices, or
transformation history. Accordingly, NIH/WHO/CDC origin is documented here as
owner-provided provenance, not independently verifiable row-level provenance.
Before redistribution or production use, create a provenance manifest and
review the terms for every underlying source. Public availability does not
automatically mean unrestricted redistribution, and naming an organization does
not imply its endorsement.
## Recommended source manifest
Track at least:
| Field | Example purpose |
|---|---|
| `source_organization` | NIH, WHO, or CDC |
| `dataset_name` | Exact source product |
| `source_url` | Stable landing/download URL |
| `version` | Release/version/date |
| `retrieved_at` | UTC retrieval timestamp |
| `license_or_terms_url` | Governing terms |
| `source_checksum` | Integrity and reproducibility |
| `category_files` | Derived files |
| `transformation` | Filtering, normalization, deduplication |
| `attribution_required` | Required notice |
| `reviewed_by` / `reviewed_at` | Governance approval |
## Generator source data
### Categories
Twelve text files define the fixed medical taxonomy. Each active row is a
pipe-separated canonical term followed by grammatical or surface variants.
These catalogs are the terminology/keyword source for generation and optional
keyword persistence.
### Patterns
Each category has a positive and negative pattern file. Patterns are project
authored/curated contextual templates; they determine whether generated text is
labeled positive or negative. Catalog membership alone does not set polarity.
### Subjects
`Subjects.txt` is required to contain exactly:
```text
He
She
They
Member
```
## SQL schema used by training export
The export reads `dbo.TrainingItem` joined to `dbo.Category` and writes:
```json
{
"id": 123,
"category": "MEDICAL_TREATMENT",
"is_positive": true,
"value": "He is receiving physical therapy."
}
```
Validation requires exactly those fields, a unique positive integer ID, one of
the 12 category names, a real JSON Boolean, and a non-empty string value.
## Split data
`create_splits.py` groups by `(category, is_positive)`, shuffles each group with
a stable derived seed, allocates rounded 10% validation and 10% development
counts, and sends the remainder to training. It then independently shuffles each
whole split and verifies uniqueness, disjointness, and complete assignment.
Small strata can have materially different proportions or no training row after
rounding. The script guarantees partition integrity, not minimum representation
in every split.
## SFT data
`build_sft.py` overwrites each split with:
```json
{
"id": 123,
"messages": [
{"role": "system", "content": "...strict JSON policy..."},
{"role": "user", "content": "He is receiving physical therapy."},
{"role": "assistant", "content": "{\"is_medical_phi\":true,...}"}
]
}
```
For positives, the expected result contains the source category and uses the
entire input for both `text` and `evidence`. For negatives, `results` is empty.
This means the current task is whole-input contextual classification rather than
precise entity-span extraction.
The conversion is destructive and non-idempotent. Recreate source-shaped splits
before rerunning it.
## Data lineage
```text
Owner-curated NIH/WHO/CDC-derived catalogs
+ project-authored polarity patterns
+ subject and tense decks
|
v
normalized generated TrainingItem rows
|
v
data/source.jsonl
|
v
source-shaped train/validation/development JSONL
|
v
SFT-shaped train/validation/development JSONL
|
v
adapter, predictions, metrics, merged model
```
## Privacy and governance
The checked-in source is synthetic/generated according to the system design,
but that assumption should be verified operationally. Do not import real patient
records without a separate privacy, consent, security, retention, and access
control review. The model is designed to detect sensitive medical context; it is
not itself a compliance determination or a substitute for human review.

View File

@ -0,0 +1,52 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
<diagram id="system-context" name="C4 Level 1 - System Context">
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1600" pageHeight="900" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="title" value="Medical PHI Model - C4 Level 1: System Context" style="text;html=1;align=center;verticalAlign=middle;fontSize=24;fontStyle=1" vertex="1" parent="1">
<mxGeometry x="350" y="20" width="900" height="40" as="geometry"/>
</mxCell>
<mxCell id="operator" value="&lt;b&gt;ML / Data Operator&lt;/b&gt;&lt;br&gt;Curates inputs, runs generation and training, reviews metrics" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=14" vertex="1" parent="1">
<mxGeometry x="70" y="310" width="240" height="110" as="geometry"/>
</mxCell>
<mxCell id="system" value="&lt;b&gt;Medical PHI Model System&lt;/b&gt;&lt;br&gt;Generates contextual examples and fine-tunes an 8B model to classify medical PHI and emit category-based JSON" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1168bd;strokeColor=#0b4884;fontColor=#ffffff;fontSize=16" vertex="1" parent="1">
<mxGeometry x="560" y="280" width="420" height="170" as="geometry"/>
</mxCell>
<mxCell id="sources" value="&lt;b&gt;NIH / WHO / CDC Sources&lt;/b&gt;&lt;br&gt;Owner-reported origins of medical terminology catalogs" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;fontSize=14" vertex="1" parent="1">
<mxGeometry x="580" y="90" width="380" height="100" as="geometry"/>
</mxCell>
<mxCell id="presidio" value="&lt;b&gt;Downstream Presidio Integration&lt;/b&gt;&lt;br&gt;Consumes compatible MEDICAL_* category/entity types; implementation is outside this repository" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=14" vertex="1" parent="1">
<mxGeometry x="1180" y="300" width="300" height="130" as="geometry"/>
</mxCell>
<mxCell id="subject" value="&lt;b&gt;Application / Reviewer&lt;/b&gt;&lt;br&gt;Submits text and consumes strict JSON classification results" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=14" vertex="1" parent="1">
<mxGeometry x="1180" y="570" width="300" height="110" as="geometry"/>
</mxCell>
<mxCell id="modelhost" value="&lt;b&gt;Model Runtime&lt;/b&gt;&lt;br&gt;Hosts the merged Hugging Face model or a separately converted GGUF artifact" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=14" vertex="1" parent="1">
<mxGeometry x="560" y="590" width="420" height="110" as="geometry"/>
</mxCell>
<mxCell id="e1" value="Operates and reviews" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="operator" target="system">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="e2" value="Curated terminology (provenance manifest needed)" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="sources" target="system">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="e3" value="Produces model artifacts" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="system" target="modelhost">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="e4" value="Category contract / model integration" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block;dashed=1" edge="1" parent="1" source="system" target="presidio">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="e5" value="Inference requests" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="subject" target="modelhost">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="e6" value="Strict JSON decisions" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="modelhost" target="subject">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="note" value="Dashed relationship is intended compatibility; Presidio code is not tracked here." style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1">
<mxGeometry x="1110" y="100" width="330" height="90" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>

View File

@ -0,0 +1,52 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
<diagram id="containers" name="C4 Level 2 - Containers">
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="1000" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="title" value="Medical PHI Model - C4 Level 2: Containers" style="text;html=1;align=center;fontSize=24;fontStyle=1" vertex="1" parent="1">
<mxGeometry x="450" y="20" width="900" height="40" as="geometry"/>
</mxCell>
<mxCell id="boundary" value="Medical PHI Model Repository" style="swimlane;html=1;startSize=35;rounded=1;fillColor=none;strokeColor=#1168bd;fontSize=16;fontStyle=1" vertex="1" parent="1">
<mxGeometry x="240" y="80" width="1320" height="780" as="geometry"/>
</mxCell>
<mxCell id="operator" value="&lt;b&gt;Operator&lt;/b&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=14" vertex="1" parent="1">
<mxGeometry x="30" y="380" width="150" height="70" as="geometry"/>
</mxCell>
<mxCell id="catalogs" value="&lt;b&gt;Catalogs and Patterns&lt;/b&gt;&lt;br&gt;Text files&lt;br&gt;12 categories, variants, subjects, positive/negative templates" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="50" y="80" width="250" height="130" as="geometry"/>
</mxCell>
<mxCell id="api" value="&lt;b&gt;Generator API&lt;/b&gt;&lt;br&gt;ASP.NET Core / .NET 10&lt;br&gt;Validates inputs, plans examples, calls LM Studio, persists ordered batches" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="380" y="80" width="280" height="140" as="geometry"/>
</mxCell>
<mxCell id="sql" value="&lt;b&gt;Training Database&lt;/b&gt;&lt;br&gt;SQL Server&lt;br&gt;Categories, keywords, generation runs, training items" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="760" y="80" width="250" height="140" as="geometry"/>
</mxCell>
<mxCell id="jsonl" value="&lt;b&gt;Training Data&lt;/b&gt;&lt;br&gt;JSONL files&lt;br&gt;Source, train, validation, development" style="shape=folder;tabWidth=60;tabHeight=20;tabPosition=left;html=1;whiteSpace=wrap;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="760" y="330" width="250" height="130" as="geometry"/>
</mxCell>
<mxCell id="pipeline" value="&lt;b&gt;Training Pipeline&lt;/b&gt;&lt;br&gt;Python 3.12 / PyTorch / TRL / PEFT&lt;br&gt;Validate, split, SFT, train, evaluate, merge" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="380" y="540" width="280" height="140" as="geometry"/>
</mxCell>
<mxCell id="artifacts" value="&lt;b&gt;Model Artifacts&lt;/b&gt;&lt;br&gt;Filesystem&lt;br&gt;LoRA adapters, metrics, merged BF16 model, untracked-process GGUF" style="shape=folder;tabWidth=60;tabHeight=20;tabPosition=left;html=1;whiteSpace=wrap;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="50" y="540" width="250" height="140" as="geometry"/>
</mxCell>
<mxCell id="lm" value="&lt;b&gt;LM Studio&lt;/b&gt;&lt;br&gt;OpenAI-compatible API&lt;br&gt;qwen/qwen3.5-9b normalization" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#999999;strokeColor=#666666;fontColor=#ffffff;fontSize=13" vertex="1" parent="1">
<mxGeometry x="1600" y="190" width="170" height="110" as="geometry"/>
</mxCell>
<mxCell id="base" value="&lt;b&gt;Base Model&lt;/b&gt;&lt;br&gt;Local Llama3-OpenBioLLM-8B" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#999999;strokeColor=#666666;fontColor=#ffffff;fontSize=13" vertex="1" parent="1">
<mxGeometry x="1600" y="600" width="170" height="100" as="geometry"/>
</mxCell>
<mxCell id="e1" value="Runs" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="operator" target="api"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e2" value="Reads" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="catalogs" target="api"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e3" value="Normalizes text" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="api" target="lm"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e4" value="Persists and checkpoints" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="api" target="sql"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e5" value="PowerShell export" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="sql" target="jsonl"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e6" value="Reads and rewrites" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="jsonl" target="pipeline"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e7" value="Loads weights/tokenizer" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="base" target="pipeline"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e8" value="Writes" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="pipeline" target="artifacts"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e9" value="Operates/reviews" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="operator" target="pipeline"><mxGeometry relative="1" as="geometry"/></mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>

View File

@ -0,0 +1,62 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
<diagram id="generator-components" name="C4 Level 3 - Generator Components">
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="1000" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="title" value="Generator API - C4 Level 3: Components" style="text;html=1;align=center;fontSize=24;fontStyle=1" vertex="1" parent="1">
<mxGeometry x="450" y="20" width="900" height="40" as="geometry"/>
</mxCell>
<mxCell id="boundary" value="RLHF.Generator.Api + RLHF.Generator" style="swimlane;html=1;startSize=35;rounded=1;fillColor=none;strokeColor=#1168bd;fontSize=16;fontStyle=1" vertex="1" parent="1">
<mxGeometry x="160" y="80" width="1450" height="780" as="geometry"/>
</mxCell>
<mxCell id="routes" value="&lt;b&gt;HTTP Endpoints&lt;/b&gt;&lt;br&gt;Program.cs&lt;br&gt;Validation, contracts, status mapping" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="50" y="90" width="240" height="120" as="geometry"/>
</mxCell>
<mxCell id="reader" value="&lt;b&gt;TrainingDataReader&lt;/b&gt;&lt;br&gt;Validates and parses subjects, categories, and patterns" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="360" y="90" width="240" height="120" as="geometry"/>
</mxCell>
<mxCell id="generation" value="&lt;b&gt;TrainingGenerationService&lt;/b&gt;&lt;br&gt;Plans, fingerprints, resumes, parallelizes, orders, validates, checkpoints" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1168bd;strokeColor=#0b4884;fontColor=#ffffff;fontSize=14" vertex="1" parent="boundary">
<mxGeometry x="680" y="280" width="300" height="160" as="geometry"/>
</mxCell>
<mxCell id="lmclient" value="&lt;b&gt;LmStudioClient&lt;/b&gt;&lt;br&gt;Local token resolution, prompt construction, HTTP completion" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="1070" y="90" width="260" height="120" as="geometry"/>
</mxCell>
<mxCell id="keyword" value="&lt;b&gt;Keyword Services&lt;/b&gt;&lt;br&gt;Load canonical values/variants; dry-run or apply unique backfill" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="50" y="550" width="260" height="130" as="geometry"/>
</mxCell>
<mxCell id="reset" value="&lt;b&gt;TrainingItemResetService&lt;/b&gt;&lt;br&gt;Clears generated rows and run checkpoints" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="380" y="550" width="250" height="130" as="geometry"/>
</mxCell>
<mxCell id="dbcontext" value="&lt;b&gt;RlhfGeneratorDbContext&lt;/b&gt;&lt;br&gt;EF Core mappings, seeds, indexes, relationships, transactions" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="720" y="550" width="270" height="130" as="geometry"/>
</mxCell>
<mxCell id="options" value="&lt;b&gt;Options&lt;/b&gt;&lt;br&gt;GenerationOptions&lt;br&gt;LmStudioOptions&lt;br&gt;SentenceTenses" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
<mxGeometry x="1070" y="550" width="260" height="130" as="geometry"/>
</mxCell>
<mxCell id="data" value="Catalog and pattern files" style="shape=folder;tabWidth=50;tabHeight=18;tabPosition=left;html=1;whiteSpace=wrap;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1">
<mxGeometry x="20" y="200" width="120" height="90" as="geometry"/>
</mxCell>
<mxCell id="lm" value="LM Studio API" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6" vertex="1" parent="1">
<mxGeometry x="1640" y="200" width="120" height="80" as="geometry"/>
</mxCell>
<mxCell id="sql" value="SQL Server" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#e1d5e7;strokeColor=#9673a6" vertex="1" parent="1">
<mxGeometry x="1640" y="640" width="120" height="90" as="geometry"/>
</mxCell>
<mxCell id="e1" value="Dispatches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="routes" target="generation"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e2" value="Reads definitions" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="generation" target="reader"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e3" value="Normalizes" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="generation" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e4" value="HTTP chat completion" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="lmclient" target="lm"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e5" value="Persists batches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="generation" target="dbcontext"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e6" value="EF Core SQL" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="dbcontext" target="sql"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e7" value="Reads" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="data" target="reader"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e8" value="Configures" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="options" target="generation"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e9" value="Configures" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="options" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e10" value="Dispatches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="routes" target="keyword"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e11" value="Dispatches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="routes" target="reset"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e12" value="Uses" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="keyword" target="dbcontext"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e13" value="Uses" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="reset" target="dbcontext"><mxGeometry relative="1" as="geometry"/></mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>

View File

@ -0,0 +1,54 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
<diagram id="generator-code" name="C4 Level 4 - Generator Code">
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="1100" math="0" shadow="0">
<root>
<mxCell id="0"/>
<mxCell id="1" parent="0"/>
<mxCell id="title" value="TrainingGenerationService - C4 Level 4: Code Relationships" style="text;html=1;align=center;fontSize=24;fontStyle=1" vertex="1" parent="1">
<mxGeometry x="400" y="20" width="1000" height="40" as="geometry"/>
</mxCell>
<mxCell id="service" value="&lt;b&gt;TrainingGenerationService&lt;/b&gt;&lt;br&gt;+ GenerateAsync()&lt;br&gt;- GenerateCategoryAsync()&lt;br&gt;- CreatePlan()&lt;br&gt;- GeneratePassAsync()&lt;br&gt;- PersistOrderedAsync()&lt;br&gt;- FindResumableRunAsync()&lt;br&gt;- ValidateGeneratedValue()" style="swimlane;html=1;startSize=30;rounded=0;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="660" y="310" width="360" height="280" as="geometry"/>
</mxCell>
<mxCell id="reader" value="&lt;b&gt;TrainingDataReader&lt;/b&gt;&lt;br&gt;+ ValidateAsync()&lt;br&gt;+ ReadCategoryValuesAsync()&lt;br&gt;+ ReadPatternsAsync()" style="swimlane;html=1;startSize=30;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="130" y="120" width="290" height="150" as="geometry"/>
</mxCell>
<mxCell id="definitions" value="&lt;b&gt;Training definitions&lt;/b&gt;&lt;br&gt;TrainingCategories&lt;br&gt;TrainingSubjects&lt;br&gt;TrainingCategoryDefinition&lt;br&gt;TrainingCategoryValue" style="swimlane;html=1;startSize=30;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="120" y="390" width="310" height="180" as="geometry"/>
</mxCell>
<mxCell id="lmclient" value="&lt;b&gt;LmStudioClient&lt;/b&gt;&lt;br&gt;+ CompleteAsync()&lt;br&gt;- ResolveFragmentTokens()&lt;br&gt;- ResolveArticles()&lt;br&gt;- SendCompletionAsync()" style="swimlane;html=1;startSize=30;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="1250" y="120" width="300" height="180" as="geometry"/>
</mxCell>
<mxCell id="options" value="&lt;b&gt;Configuration&lt;/b&gt;&lt;br&gt;GenerationOptions&lt;br&gt;LmStudioOptions&lt;br&gt;SentenceTenses" style="swimlane;html=1;startSize=30;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="1260" y="390" width="280" height="150" as="geometry"/>
</mxCell>
<mxCell id="context" value="&lt;b&gt;RlhfGeneratorDbContext&lt;/b&gt;&lt;br&gt;DbSet&lt;Category&gt;&lt;br&gt;DbSet&lt;TrainingItem&gt;&lt;br&gt;DbSet&lt;GenerationRun&gt;&lt;br&gt;DbSet&lt;Keyword&gt;&lt;br&gt;DbSet&lt;KeywordVariant&gt;" style="swimlane;html=1;startSize=30;fillColor=#f8cecc;strokeColor=#b85450;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="650" y="760" width="380" height="200" as="geometry"/>
</mxCell>
<mxCell id="run" value="&lt;b&gt;GenerationRun&lt;/b&gt;&lt;br&gt;Status, seed, fingerprints&lt;br&gt;checkpoint and counts&lt;br&gt;failure/cancellation data" style="swimlane;html=1;startSize=30;fillColor=#f8cecc;strokeColor=#b85450;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="180" y="760" width="300" height="160" as="geometry"/>
</mxCell>
<mxCell id="item" value="&lt;b&gt;TrainingItem&lt;/b&gt;&lt;br&gt;RecordKey, PlanOrdinal&lt;br&gt;Category, Keyword&lt;br&gt;IsPositive, Value" style="swimlane;html=1;startSize=30;fillColor=#f8cecc;strokeColor=#b85450;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
<mxGeometry x="1200" y="760" width="300" height="160" as="geometry"/>
</mxCell>
<mxCell id="channel" value="&lt;b&gt;Bounded Channel&lt;/b&gt;&lt;br&gt;Parallel workers write completed plan items; ordered writer consumes" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13" vertex="1" parent="1">
<mxGeometry x="690" y="120" width="300" height="110" as="geometry"/>
</mxCell>
<mxCell id="e1" value="Validates and reads inputs" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="reader"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e2" value="Returns" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="reader" target="definitions"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e3" value="Plans with" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="definitions"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e4" value="Calls concurrently" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e5" value="Binds" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="options" target="service"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e6" value="Binds" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="options" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e7" value="Coordinates producer/consumer" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="channel"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e8" value="Queries and transacts" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="context"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e9" value="Tracks" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=diamond;endFill=0" edge="1" parent="1" source="context" target="run"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e10" value="Tracks" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=diamond;endFill=0" edge="1" parent="1" source="context" target="item"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="e11" value="Owns ordered items" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=diamond;endFill=1" edge="1" parent="1" source="run" target="item"><mxGeometry relative="1" as="geometry"/></mxCell>
<mxCell id="note" value="Plan order: category positive pass, then negative pass. Workers may finish out of order; persistence waits for the next contiguous ordinal and commits rows plus checkpoint atomically." style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=12" vertex="1" parent="1">
<mxGeometry x="1080" y="560" width="430" height="120" as="geometry"/>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>

201
docs/generator.md Normal file
View File

@ -0,0 +1,201 @@
# Generator
## Purpose
`PHI.Data.Generator` creates labeled contextual medical-language examples. It
does not generate a graph of nodes and edges. Its central domain concepts are
categories, category values/keywords, patterns, subjects, and training items.
## Projects
| Project | Role |
|---|---|
| `RLHF.Generator` | Catalog parsing, planning, LM client, persistence, generation, keyword services |
| `RLHF.Generator.Api` | ASP.NET Core host and HTTP endpoints |
| `RLHF.Generator.Tests` | xUnit unit/integration tests using SQLite for test persistence |
All three target .NET 10. Production persistence uses EF Core SQL Server.
## Taxonomy
The fixed categories are:
1. `MEDICAL_ALLERGY`
2. `MEDICAL_DIAGNOSIS`
3. `MEDICAL_FAMILY_HISTORY`
4. `MEDICAL_MEDICATION`
5. `MEDICAL_MENTAL_HEALTH`
6. `MEDICAL_OTHER`
7. `MEDICAL_PHYSICAL_HEALTH`
8. `MEDICAL_PROCEDURE`
9. `MEDICAL_SUBSTANCE_USE`
10. `MEDICAL_SYMPTOM`
11. `MEDICAL_TEST`
12. `MEDICAL_TREATMENT`
Lookup is case-insensitive; stored/returned names use these exact values.
## Input layout and grammar
```text
RLHF.Generator/Data/
Subjects.txt
Categories/
MEDICAL_ALLERGY.txt
...
Patterns/
MEDICAL_ALLERGY.positive.txt
MEDICAL_ALLERGY.negative.txt
...
```
Blank lines and lines beginning with `#` are ignored. A category line is:
```text
canonical form|allowed variant|another allowed variant
```
Forms must be non-empty and case-insensitively unique within a category. The
first is canonical. The remainder are alternative forms available to the
normalization model.
The exact subjects are `He`, `She`, `They`, and `Member`.
Pattern tokens include:
| Token | Meaning |
|---|---|
| `[x]` | Canonical category value; allowed variants are supplied to LM Studio |
| `[sub]` | A subject selected from the shuffled subject deck |
| `[is\|are\|was\|were]` | Copula selected/rephrased for subject and tense |
| `[has\|have\|had]` | Possession/auxiliary selected for subject and tense |
| `[a\|an]` | Article resolved locally from the following word |
The legacy `[s]` placeholder is rejected.
## Positive and negative examples
Every category has an independent positive and negative pattern set. A positive
pattern describes real medical context; a negative pattern uses the same domain
language without asserting medical information about the person being analyzed.
Representative intended behavior:
| Input | Expected label | Why |
|---|---:|---|
| `Has diabetes` | Positive | Subjectless clinical fragment with an implied member/patient |
| `He had a reaction to shellfish` | Positive | Actual allergy/reaction assertion |
| `She wrote a book about diabetes` | Negative | Creative/educational context, not her diagnosis |
| `Had a reaction to the horrible cover of Van Halen's Jump` | Negative | Idiomatic non-medical reaction |
Those examples describe the intended learned behavior; they are not hard-coded
rules or keyword matchers in the inference pipeline.
## Plan construction
For each category, the service runs the positive pass and then the negative
pass. It partitions patterns into:
- **value-bound:** contains `[x]`;
- **generic:** does not contain `[x]`.
It rejects values without any value-bound pattern and value-bound patterns when
the category has no values. It then:
1. shuffles values, value-bound patterns, and generic patterns;
2. creates shuffled/cycling subject and tense decks;
3. emits every generic pattern once;
4. emits `max(value-bound pattern count, value count)` value-bound plans;
5. reshuffles a deck whenever it is exhausted.
The tense deck is `null`, `past`, `present`, and `future`. The canonical category
value is placed in `[x]`; variants are not independently randomly sampled.
Because each polarity is planned from its own pattern/value counts, positive and
negative totals are not guaranteed to be equal.
## Subject and fragment behavior
The precise behavior is:
- A pattern containing `[sub]` gets one shuffled explicit subject.
- A pattern without `[sub]` is marked as a fragment and gets no inserted
subject.
- `Member` is one of the four explicit subject choices; it is not a fallback
value.
- For a fragment, the LM system prompt uses an implied singular member/patient
only to choose grammar. It explicitly forbids returning or inventing that
subject.
- Some fragments are completed locally without an LM call when no unresolved
choice, protected-form decision, or meaningful tense rewrite remains.
Therefore the safe summary is: **subjectless patterns remain subjectless, while
grammar assumes an implied singular member/patient.**
## LM Studio normalization
Default settings:
| Setting | Value |
|---|---|
| Endpoint | `http://localhost:1234/v1/chat/completions` |
| Model | `qwen/qwen3.5-9b` |
| Temperature | `0.0` |
| Top-p | `1.0` |
| Max tokens | `96` |
| Seed | `42` |
| Timeout | `300` seconds |
Each request carries fragment status, optional protected subject, protected
medical term and allowed forms, requested tense, and the sentence/template. The
system prompt requires exactly one unwrapped line, preserves meaning and
polarity, resolves all bracket expressions, and forbids invented fragment
predicates.
Generation rejects empty output, newlines, unresolved square brackets,
quotation/JSON/code-fence wrappers, and common response labels.
## Concurrency and durability
Defaults are seed 42, parallelism 8, save batches of 24, and progress logs every
5 seconds.
Workers generate concurrently into a bounded channel. A single persistence
writer places results into a sorted buffer and commits only the next contiguous
plan ordinals. Each transaction inserts the batch and advances the generation
checkpoint together.
A run can resume only when its seed, input fingerprint, configuration
fingerprint, expected count, committed row count, and last ordinal agree.
Changed inputs/configuration or inconsistent persisted state are rejected.
## Keyword catalog
Keyword loading uses each category file's canonical value and variants. Loading
is additive, case-insensitive, and idempotent. It does not update existing
training items. Backfill is a separate operation:
- dry-run is available;
- only unique matches are assigned;
- unmatched or ambiguous rows remain without a keyword.
## Database migrations
The migrations progress through:
1. initial `Category` and `TrainingItem` schema;
2. durable generation-run/checkpoint support;
3. `Keyword`, `KeywordVariant`, and training-item keyword linkage.
`AddKeywordCatalog.sql` provides a SQL alternative for the keyword phase.
## Configuration changes that invalidate resume
Treat these as generation identity:
- category, pattern, or subject file contents/paths;
- random seed;
- concurrency and save/progress options included in the fingerprint;
- LM endpoint/model/prompt and generation settings.
Start a clean run rather than bypassing fingerprint checks.

254
docs/runbook.md Normal file
View File

@ -0,0 +1,254 @@
# End-to-end runbook
This procedure starts with catalogs and ends with a merged model. Run commands
from the repository root unless stated otherwise.
## 1. Record the run
Capture:
- Git revision and clean/dirty state;
- operator and UTC start time;
- catalog/pattern checksums and provenance manifest;
- SQL Server target;
- LM Studio model artifact/version and settings;
- base model path/revision/checksum;
- GPU, driver, CUDA, Python, and dependency versions;
- expected output locations and acceptance thresholds.
## 2. Prerequisite checks
Install/prepare:
- .NET 10 SDK;
- SQL Server reachable with schema-creation privileges;
- LM Studio with `qwen/qwen3.5-9b` (or an intentionally approved replacement);
- Windows PowerShell for SQL export;
- WSL/Linux, Git, build tools, and `uv`;
- Python 3.12;
- NVIDIA CUDA GPU with BF16 and sufficient VRAM;
- local `Llama3-OpenBioLLM-8B` base model;
- enough CPU RAM/disk for an unquantized 8B BF16 merge.
Correct the legacy `/d/ai/training/openbiollm-phi` path in every shell scaffold,
or use the direct commands below from the real checkout.
## 3. Validate generator inputs
Confirm:
- exactly 12 category files;
- one positive and one negative pattern file per category;
- subjects exactly `He`, `She`, `They`, `Member`;
- no legacy `[s]` tokens;
- canonical/variant forms are unique;
- each category has values and compatible `[x]` patterns.
The generator also enforces these checks when a run starts.
## 4. Configure and migrate SQL Server
Review the `RLHFGenerator` connection string in
`PHI.Data.Generator/RLHF.Generator.Api/appsettings.json`. Prefer environment
configuration or user secrets for deployment-specific values.
From `PHI.Data.Generator`:
```powershell
dotnet restore .\RLHF.Generator.slnx
dotnet ef database update `
--project .\RLHF.Generator\RLHF.Generator.csproj `
--startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj
```
Back up an existing database before migration or reset.
## 5. Test the generator
```powershell
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
```
Stop if any test fails.
## 6. Start LM Studio
1. Load the configured normalization model.
2. Start the local OpenAI-compatible server.
3. Confirm `http://localhost:1234/v1/chat/completions` is reachable.
4. Keep temperature, seed, token limit, and model name aligned with generator
configuration.
## 7. Start the generator API
```powershell
dotnet run --project `
.\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj
```
Note the actual listening URL. In Development, inspect Swagger/Scalar. Call
`GET /` as a basic process check.
## 8. Load keywords and generate
In another PowerShell window:
```powershell
$base = "http://localhost:<port>"
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
```
The generation request remains open until completion. Monitor API logs and SQL
`GenerationRun`. On cancellation/failure, correct the cause and call the same
endpoint to resume only if the service accepts the matching fingerprints.
Do not edit catalogs, patterns, generation settings, or LM settings mid-run.
Optional keyword linkage:
```powershell
Invoke-RestMethod -Method Post `
-Uri "$base/api/training-items/backfill-keywords?dryRun=true"
Invoke-RestMethod -Method Post `
-Uri "$base/api/training-items/backfill-keywords?dryRun=false"
```
## 9. Export SQL data
Review SQL server/database values in `config/pipeline.json`, then run:
```powershell
.\scaffolding\03-export-sql.ps1
```
Confirm `data/source.jsonl` exists and record positive/negative totals.
## 10. Prepare the Python environment in WSL
From the actual WSL path of the repository:
```bash
uv python install 3.12.13
uv venv --python 3.12.13
uv lock
uv sync --frozen
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"
```
Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the
environment and contain `git add`/`git commit` commands.
## 11. Review pipeline configuration
Update `config/pipeline.json` deliberately:
- SQL connection metadata;
- actual WSL base-model path;
- split proportions/seed;
- sequence length;
- LoRA and training hyperparameters;
- output directories.
Archive the exact configuration used.
## 12. Validate and split
```bash
uv run python scripts/validate_source.py
uv run python scripts/create_splits.py
```
Review counts per category/polarity/split. Pay special attention to small strata.
## 13. Build SFT conversations
```bash
uv run python scripts/build_sft.py
```
This overwrites all split files. If repeated, run `create_splits.py` first.
Inspect a positive and negative example from every category.
## 14. Smoke-test the base model
```bash
uv run python scripts/model_smoke_test.py
```
Confirm successful loading, finite logits, and expected GPU memory use.
## 15. Run the pilot
```bash
uv run python scripts/train_pilot.py
```
Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss
before full training.
## 16. Train the full adapter
```bash
uv run python scripts/train.py
```
This replaces `outputs/final-adapter`. Preserve any prior accepted adapter first.
Monitor training/evaluation loss, checkpoints, utilization, and disk.
## 17. Evaluate and rescore
```bash
uv run python scripts/evaluate.py
uv run python scripts/rescore_evaluation.py
```
Compare `outputs/development-evaluation/summary.json` with acceptance thresholds.
Review every error, particularly false negatives, false positives, category
errors, malformed JSON, and contextual-negative failures.
## 18. Verify stopping
```bash
uv run python scripts/verify_generation_stop.py
```
All five sampled outputs must be exact parseable JSON.
## 19. Merge the adapter
```bash
uv run python scripts/merge_model.py
```
This replaces `outputs/Invited-PHI-8B` atomically after verification. It can
require substantial CPU RAM and disk.
## 20. Validate and publish artifacts
Before release:
1. Add/run a merged-model inference smoke test.
2. Confirm tokenizer, chat template, system prompt, configs, and all weight
shards are present.
3. Record checksums and total sizes.
4. Preserve adapter, merged model, metrics, predictions, config, source/split
checksums, Git revision, environment lock, and provenance manifest.
5. If GGUF is required, use an approved conversion/quantization process and
record it; no such process is tracked here.
6. Run downstream Presidio integration and end-user acceptance tests.
## Recovery
| Failure | Action |
|---|---|
| LM Studio unavailable/timeout | Restore service/model, then resume unchanged run |
| Generator invalid output | Inspect request/model/prompt; do not bypass validation |
| Fingerprint mismatch | Restore original inputs/settings or intentionally reset/start clean |
| SQL batch failure | Correct database cause; checkpointed committed batches remain |
| Split/SFT rerun failure | Recreate splits from `source.jsonl`, then rebuild SFT |
| CUDA OOM | Reduce approved batch/sequence settings, rerun pilot, document change |
| Training output exists | Archive accepted output; scripts intentionally replace output dirs |
| Evaluation failure | Preserve raw logs/artifacts, fix root cause, rerun evaluation |
| Merge resource failure | Increase CPU RAM/disk or use an approved merge host |

View File

@ -0,0 +1,90 @@
# Testing and validation
## .NET automated tests
Run:
```powershell
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
```
The xUnit suite covers:
| Test area | Coverage |
|---|---|
| LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming |
| Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output |
| Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs |
| Endpoint behavior | Generation endpoint waits for actual generation and commit |
| Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs |
| Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity |
| Provider isolation | SQL Server production DI and SQLite-only test setup |
Tests use SQLite for integration persistence while explicitly verifying that the
production service registers SQL Server.
## Python verification stages
There is no Python unit-test suite. Verification is operational:
1. `validate_source.py` validates exported schema and taxonomy.
2. `create_splits.py` checks uniqueness, disjointness, and complete assignment.
3. `build_sft.py` checks message shape and parses generated assistant JSON.
4. `model_smoke_test.py` loads the base model and rejects non-finite logits.
5. `train_pilot.py` verifies a small QLoRA training path.
6. `evaluate.py` validates generated JSON/schema and computes metrics.
7. `verify_generation_stop.py` checks exact JSON parsing on five examples.
8. `merge_model.py` verifies merged weights and configuration files.
## Current tracked evaluation snapshot
`outputs/development-evaluation/summary.json` currently reports:
| Metric | Value |
|---|---:|
| Development records | 2,027 |
| Valid JSON | 100% |
| Valid schema | 100% |
| Classification accuracy | 99.852% |
| True positives / false negatives | 996 / 0 |
| True negatives / false positives | 1,028 / 3 |
| Precision | 99.700% |
| Recall | 100% |
| F1 | 99.850% |
| Entity category accuracy | 98.394% |
| Entity text accuracy | 100% |
| Evidence accuracy | 100% |
These values describe the checked-in development artifact, not an independent
benchmark or prospective production guarantee.
## Recommended acceptance gates
Before publishing a new model:
- all .NET tests pass;
- source validation and split integrity pass;
- the pilot finishes with finite loss;
- full training selects/saves a valid adapter;
- JSON and schema validity remain 100%;
- classification/category metrics meet a pre-agreed threshold;
- false positives and false negatives are manually reviewed;
- generation-stop verification passes;
- merged-model loading/inference is added and passes;
- provenance and license manifest is complete;
- a Presidio integration test is run in the downstream repository;
- checksums, configuration, Git revision, base model revision, and environment
versions are archived.
## Test gaps
- No direct Python unit tests for schema, splitting edge cases, SFT conversion,
JSON stopping, scoring, or merge logic.
- No inference smoke test for the merged model.
- No tracked GGUF conversion/quantization verification.
- No end-to-end test from generator API through SQL export to training data.
- No checked-in Presidio integration or category mapping test.
- No fixed golden tests for the four representative contextual sentences.
Adding those contextual examples as regression tests would make the intended
positive/negative distinction executable rather than documentary.

152
docs/training-pipeline.md Normal file
View File

@ -0,0 +1,152 @@
# Training pipeline
## Runtime
- Python `3.12.*` (the setup script installs 3.12.13)
- `uv` for interpreter/dependency management
- PyTorch 2.11.0 from the CUDA 12.8 index
- Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1
- bitsandbytes 0.49.2 for 4-bit NF4 loading
- CUDA-capable NVIDIA GPU with BF16 support
- WSL/Linux for shell wrappers
- substantial CPU RAM/disk for BF16 merge
The canonical configuration is `config/pipeline.json`.
## Pipeline stages
### 00 - Audit
`scaffolding/00-audit.sh` checks Linux/WSL, Python, Git, compiler/build tools,
download tools, NVIDIA utilities, memory, disk, base model, and repository state.
### 01 - Create environment
`scaffolding/01-create-venv.sh` installs `uv`, installs Python 3.12.13, and
recreates `.venv`. It also attempts to stage and commit files, so inspect/remove
those Git commands before using it in an automated or shared worktree.
### 02 - Install dependencies
`scaffolding/02-install-dependencies.sh` locks/synchronizes dependencies and
fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime
information. It also contains Git commit commands.
### 03 - Export SQL
`scaffolding/03-export-sql.ps1` uses Windows integrated authentication, queries
SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative
counts.
### 04 - Validate source
`scripts/validate_source.py` enforces the four-field schema, IDs, taxonomy,
Boolean labels, non-empty values, and ID uniqueness.
### 05 - Create splits
`scripts/create_splits.py` creates deterministic stratified 80/10/10 partitions
using seed `20260726` and verifies that every input ID appears exactly once.
### 06 - Build SFT
`scripts/build_sft.py` loads `config/system_prompt.txt`, builds the exact target
JSON, and overwrites all three split files with chat conversations. Always run
stage 05 immediately before rerunning stage 06.
### 07 - Base-model smoke test
`scripts/model_smoke_test.py` loads one training example, tokenizer/template, and
the 4-bit base model; performs a forward pass; and rejects non-finite logits.
It does not test the adapter or merged model.
### 08 - Pilot training
`scripts/train_pilot.py` selects the first up-to-64 training records and runs 10
steps. It deletes/recreates `outputs/pilot-adapter`.
### 09 - Full training
`scripts/train.py` uses:
- NF4 4-bit base weights with double quantization and BF16 compute;
- LoRA rank 16, alpha 32, dropout 0.05, target `all-linear`;
- completion-only loss over the assistant response;
- gradient checkpointing;
- batch size 2 and accumulation 8;
- 2 epochs, learning rate 0.0002, 100 warmup steps;
- epoch evaluation and checkpointing;
- lowest validation loss as the best saved checkpoint.
It deletes/recreates `outputs/final-adapter`. There is no early-stopping
callback.
### 10 - Evaluate
`scripts/evaluate.py` loads the base model plus final adapter and generates all
development outputs in deterministic batches of eight. Generation stops at EOS,
end-of-turn, or a complete balanced JSON object. Results are atomically written
to `outputs/development-evaluation`.
Metrics cover JSON/schema validity, classification confusion matrix and
precision/recall/F1, result count, category, text, and evidence.
### Rescore without inference
`scripts/rescore_evaluation.py` reparses existing raw outputs, annotates the
prediction records, and rewrites predictions/summary without rerunning a model.
### 11 - Verify generation stopping
`scripts/verify_generation_stop.py` generates the first five development
examples and requires each complete output to parse with `json.loads`. Balanced
braces trigger stopping; schema validation remains a later responsibility.
### 12 - Merge
`scripts/merge_model.py` loads unquantized BF16 base weights on CPU, merges the
final adapter, saves safe-serialized 5 GB shards to a temporary directory,
verifies key files, and atomically promotes the directory to
`outputs/Invited-PHI-8B`.
This stage creates a standalone Hugging Face model, not GGUF.
## Chat contract
`config/chat_template.jinja` uses Llama-style role headers and end-of-turn
tokens. `config/system_prompt.txt` requires exactly one JSON object:
```json
{
"is_medical_phi": true,
"score": 1.0,
"reason": "...",
"results": [
{
"entity_type": "MEDICAL_DIAGNOSIS",
"text": "...",
"score": 1.0,
"evidence": "..."
}
]
}
```
No Markdown, extra prose, inferred facts, or non-`MEDICAL_*` types are allowed.
## Evaluation caveats
- A valid JSON prefix followed by trailing text can count as parseable in the
evaluator/rescorer because `raw_decode` is used.
- Entity comparisons use the first expected and predicted result.
- Entity accuracy denominators are all positive records.
- Exact whole-input text/evidence reproduction is measured, not span overlap.
- Result-count correctness is reported separately from schema validity.
- Evaluation covers base-plus-adapter, not the merged standalone model.
## Path caveat
All `.sh` wrappers use `/d/ai/training/openbiollm-phi`, while this checkout may
be elsewhere (for example `/d/ai/training/Models-Phi-8B`). Fix the `PROJECT_ROOT`
line in every wrapper or invoke the underlying `uv run python` commands from the
actual repository root.