updated docs
This commit is contained in:
parent
2f5686c3a1
commit
42ee4c2605
55
README.md
55
README.md
|
|
@ -1,3 +1,54 @@
|
|||
# Model for flagging and redacting PHI
|
||||
# Medical PHI detection model
|
||||
|
||||
Simple, repeatable pipeline for exporting labeled "Generator" created SQL Server data, fine-tuning / extending Llama3-OpenBioLLM-8B, evaluating it, and producing a GGUF model.
|
||||
This repository contains two connected systems:
|
||||
|
||||
1. A .NET generator that combines curated medical terminology with positive and
|
||||
negative language patterns, normalizes the results through an
|
||||
OpenAI-compatible LM Studio endpoint, and persists labeled examples to SQL
|
||||
Server.
|
||||
2. A Python/WSL training pipeline that exports those examples, validates and
|
||||
splits them, converts them to supervised fine-tuning (SFT) conversations,
|
||||
trains and evaluates a QLoRA adapter for `Llama3-OpenBioLLM-8B`, and merges
|
||||
the adapter into a standalone Hugging Face model.
|
||||
|
||||
The repository also contains generated data, trained adapters, a merged model,
|
||||
evaluation results, and GGUF artifacts. The tracked automation creates the
|
||||
Hugging Face model; a GGUF conversion command is not currently tracked.
|
||||
|
||||
## Documentation
|
||||
|
||||
Start with [the documentation index](docs/README.md).
|
||||
|
||||
- [Architecture](docs/architecture.md)
|
||||
- [Generator](docs/generator.md)
|
||||
- [Data and provenance](docs/data-and-provenance.md)
|
||||
- [Training pipeline](docs/training-pipeline.md)
|
||||
- [API reference](docs/api-reference.md)
|
||||
- [Testing and validation](docs/testing-and-validation.md)
|
||||
- [Step-by-step runbook](docs/runbook.md)
|
||||
- [Terminology and verified claims](docs/claims-and-terminology.md)
|
||||
- [Artifacts and repository map](docs/artifacts.md)
|
||||
- [Editable C4 diagrams](docs/diagrams/)
|
||||
|
||||
## Important semantics
|
||||
|
||||
- `Category` is the label/taxonomy term used throughout this repository. The
|
||||
names are compatible with the intended Presidio extension/override, but this
|
||||
repository does not contain the Presidio integration itself.
|
||||
- A pattern with `[sub]` receives a shuffled subject (`He`, `She`, `They`, or
|
||||
`Member`). A pattern without `[sub]` is a subjectless fragment. The LM prompt
|
||||
treats that fragment as having an implied singular member/patient for grammar
|
||||
only; it must not insert that subject into the output.
|
||||
- Positive and negative patterns teach contextual classification. Medical words
|
||||
can therefore appear in both classes: a real medical assertion is positive,
|
||||
while educational, fictional, historical, quoted, or unrelated use can be
|
||||
negative.
|
||||
- The medical terminology catalogs were assembled from NIH, WHO, and CDC data,
|
||||
according to the project owner. See the provenance documentation for the
|
||||
verification and licensing work still required.
|
||||
|
||||
## Quick start
|
||||
|
||||
The complete, ordered procedure is in [the runbook](docs/runbook.md). Do not run
|
||||
the numbered shell wrappers without first correcting their legacy hard-coded
|
||||
project path.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,64 @@
|
|||
# Documentation index
|
||||
|
||||
## System overview
|
||||
|
||||
This project creates a medical-context PHI classifier in two stages:
|
||||
|
||||
```text
|
||||
NIH/WHO/CDC-derived terminology + curated patterns
|
||||
|
|
||||
v
|
||||
.NET generator
|
||||
|
|
||||
v
|
||||
SQL Server TrainingItem
|
||||
|
|
||||
v
|
||||
export -> validate -> split -> SFT
|
||||
|
|
||||
v
|
||||
QLoRA train and evaluate
|
||||
|
|
||||
v
|
||||
adapter -> merged Hugging Face model
|
||||
```
|
||||
|
||||
The generator creates labeled natural-language examples. The Python pipeline
|
||||
uses those examples to teach a model to return a strict JSON decision describing
|
||||
whether the input contains medical PHI and, for positives, its category.
|
||||
|
||||
## Documents
|
||||
|
||||
| Document | Purpose |
|
||||
|---|---|
|
||||
| [Architecture](architecture.md) | System boundaries, data flow, persistence, and design decisions |
|
||||
| [Generator](generator.md) | Catalogs, patterns, planning, randomization, LM normalization, durability |
|
||||
| [Data and provenance](data-and-provenance.md) | Schemas, categories, terminology sources, lineage, and governance |
|
||||
| [Training pipeline](training-pipeline.md) | Every Python script and scaffold step |
|
||||
| [API reference](api-reference.md) | Generator HTTP endpoints, contracts, and failure behavior |
|
||||
| [Testing and validation](testing-and-validation.md) | .NET tests, Python verification, evaluation semantics, current metrics |
|
||||
| [Runbook](runbook.md) | Prerequisites and complete operator procedure |
|
||||
| [Terminology and verified claims](claims-and-terminology.md) | Corrections and precise wording for the project explanation |
|
||||
| [Artifacts and repository map](artifacts.md) | Source tree, inputs, outputs, and reproducibility boundaries |
|
||||
|
||||
## C4 diagrams
|
||||
|
||||
The editable draw.io documents are:
|
||||
|
||||
- [Level 1: System context](diagrams/01-system-context.drawio)
|
||||
- [Level 2: Containers](diagrams/02-containers.drawio)
|
||||
- [Level 3: Components](diagrams/03-generator-components.drawio)
|
||||
- [Level 4: Code](diagrams/04-generator-code.drawio)
|
||||
|
||||
Open the files with diagrams.net, the draw.io desktop application, or the VS
|
||||
Code Draw.io Integration extension.
|
||||
|
||||
## Documentation conventions
|
||||
|
||||
- **Verified** means behavior is directly represented by tracked code, tests, or
|
||||
configuration.
|
||||
- **Owner-provided provenance** means the fact was supplied by the project owner
|
||||
but source URLs, versions, retrieval dates, and licenses are not tracked here.
|
||||
- Paths in commands are examples. The checked-in shell wrappers still use the
|
||||
legacy WSL path `/d/ai/training/openbiollm-phi`; update them before use if the
|
||||
checkout is elsewhere.
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
# Generator API reference
|
||||
|
||||
## Hosting
|
||||
|
||||
Run the ASP.NET Core .NET 10 project
|
||||
`PHI.Data.Generator/RLHF.Generator.Api`. Production requires:
|
||||
|
||||
- the `RLHFGenerator` SQL Server connection string;
|
||||
- migrated database schema;
|
||||
- copied generator `Data` files;
|
||||
- LM Studio listening at the configured OpenAI-compatible endpoint.
|
||||
|
||||
Swagger and Scalar are enabled only in the Development environment.
|
||||
|
||||
## Endpoints
|
||||
|
||||
### `GET /`
|
||||
|
||||
Returns a small service status/identity response. Use it as a basic process
|
||||
health check; it does not prove database or LM Studio readiness.
|
||||
|
||||
### `POST /api/lm-studio/complete`
|
||||
|
||||
Normalizes one supplied sentence or fragment without persistence.
|
||||
|
||||
The request contract supports:
|
||||
|
||||
- text/sentence;
|
||||
- required fragment flag;
|
||||
- optional protected subject;
|
||||
- optional protected medical term;
|
||||
- optional allowed medical forms;
|
||||
- optional requested tense.
|
||||
|
||||
Validation rejects empty input, missing fragment status, invalid tense,
|
||||
inconsistent protected-term/forms metadata, and duplicate allowed forms.
|
||||
Successful output is the normalized plain-text line.
|
||||
|
||||
### `POST /api/keywords/load`
|
||||
|
||||
Loads the category catalogs into `Keyword`/`KeywordVariant`.
|
||||
|
||||
- additive;
|
||||
- case-insensitive;
|
||||
- idempotent;
|
||||
- does not backfill existing training items.
|
||||
|
||||
### `POST /api/training-items/backfill-keywords`
|
||||
|
||||
Matches existing generated text to keywords.
|
||||
|
||||
Query behavior includes a `dryRun` option and bounded batch processing. Only a
|
||||
unique match is assigned. Ambiguous and unmatched records remain null.
|
||||
|
||||
### `POST /api/generation/run`
|
||||
|
||||
Starts a new compatible generation run or resumes an incomplete one. The HTTP
|
||||
request waits for generation and all persistence commits to complete; it does
|
||||
not merely enqueue background work.
|
||||
|
||||
The response reports run identity/status and generation counts. Failures and
|
||||
cancellation update durable run state before surfacing the error.
|
||||
|
||||
### `POST /api/training-items/reset`
|
||||
|
||||
Clears generated training items and generation checkpoints while preserving the
|
||||
seeded category rows. Treat this as destructive and restrict it operationally.
|
||||
|
||||
## Suggested invocation sequence
|
||||
|
||||
```powershell
|
||||
$base = "http://localhost:<port>"
|
||||
|
||||
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
|
||||
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
|
||||
Invoke-RestMethod -Method Post -Uri "$base/api/training-items/backfill-keywords?dryRun=true"
|
||||
Invoke-RestMethod -Method Post -Uri "$base/api/training-items/backfill-keywords?dryRun=false"
|
||||
```
|
||||
|
||||
Use the actual port shown by `dotnet run`/launch settings.
|
||||
|
||||
## Operational controls
|
||||
|
||||
- Put authentication/authorization and network controls in front of the API;
|
||||
the tracked host does not define an application authorization policy.
|
||||
- Restrict reset and generation endpoints to operators.
|
||||
- Do not expose the configured SQL Server connection details or LM endpoint.
|
||||
- Monitor generation progress, LM failures, request duration, committed counts,
|
||||
and generation-run status.
|
||||
- Back up the database before reset or migration.
|
||||
|
|
@ -0,0 +1,113 @@
|
|||
# Architecture
|
||||
|
||||
## Goals
|
||||
|
||||
The system produces a domain-adapted model that classifies contextual medical
|
||||
PHI and returns a strict JSON response. Its principal design goals are:
|
||||
|
||||
- create reproducible positive and negative training examples;
|
||||
- preserve a controlled 12-category medical taxonomy;
|
||||
- use contextual negatives so keywords alone do not determine classification;
|
||||
- durably generate large datasets with resumable SQL persistence;
|
||||
- fine-tune an 8B model on commodity GPU hardware through QLoRA;
|
||||
- evaluate JSON validity, classification, and category extraction separately;
|
||||
- publish both an adapter and a standalone merged model.
|
||||
|
||||
## System boundary
|
||||
|
||||
Inside the repository are:
|
||||
|
||||
- the .NET 10 generator library, API, EF Core migrations, catalogs, and tests;
|
||||
- curated category values, grammar variants, subjects, and positive/negative
|
||||
patterns;
|
||||
- Python 3.12 data preparation, training, evaluation, and merge scripts;
|
||||
- PowerShell/Bash orchestration scripts;
|
||||
- generated JSONL data and model/evaluation artifacts.
|
||||
|
||||
External dependencies are:
|
||||
|
||||
- SQL Server (`RLHFGenerator`);
|
||||
- LM Studio exposing an OpenAI-compatible chat completions endpoint;
|
||||
- the `qwen/qwen3.5-9b` normalization model configured in the API;
|
||||
- the local `Llama3-OpenBioLLM-8B` base model;
|
||||
- WSL/Linux, CUDA 12.8-compatible PyTorch, and a BF16-capable NVIDIA GPU;
|
||||
- NIH, WHO, and CDC as owner-reported origins of the terminology catalogs.
|
||||
|
||||
Microsoft Presidio is an intended downstream integration, not a runtime
|
||||
component of this repository.
|
||||
|
||||
## End-to-end data flow
|
||||
|
||||
1. The generator validates 12 category files, 24 polarity-specific pattern
|
||||
files, and the exact four-subject catalog.
|
||||
2. For each category and polarity, it deterministically shuffles values,
|
||||
patterns, subjects, and tenses using the configured random seed.
|
||||
3. It substitutes protected medical values, resolves some grammar locally, and
|
||||
asks LM Studio to normalize remaining expressions without changing meaning or
|
||||
polarity.
|
||||
4. It validates one-line output and writes ordered `TrainingItem` rows to SQL
|
||||
Server in atomic batches, updating a durable generation checkpoint in the
|
||||
same transaction.
|
||||
5. PowerShell exports `TrainingItem` plus `Category` to `data/source.jsonl`.
|
||||
6. Python validates the schema and taxonomy, then deterministically stratifies
|
||||
by `(category, is_positive)` into train, validation, and development sets.
|
||||
7. The SFT builder replaces each split record with a three-message conversation
|
||||
containing the system policy, input text, and exact target JSON.
|
||||
8. Pilot and full training load the base model in 4-bit NF4 and train LoRA
|
||||
adapters using completion-only loss.
|
||||
9. Evaluation loads the base model plus final adapter and generates development
|
||||
predictions deterministically.
|
||||
10. The merge step loads the unquantized base model on CPU, merges the adapter,
|
||||
and saves a BF16 Hugging Face model.
|
||||
|
||||
## Persistence model
|
||||
|
||||
| Entity | Responsibility |
|
||||
|---|---|
|
||||
| `Category` | Stable seeded category ID/name |
|
||||
| `Keyword` | Canonical category-specific medical value |
|
||||
| `KeywordVariant` | Alternate grammatical/surface form |
|
||||
| `GenerationRun` | Seed, fingerprints, status, checkpoint, counts, errors |
|
||||
| `TrainingItem` | Generated text, label, category, optional keyword, plan ordinal |
|
||||
|
||||
Important constraints include unique category names, unique canonical values
|
||||
within a category, unique variants within a keyword, unique training record
|
||||
keys, and unique `(GenerationRunId, PlanOrdinal)`.
|
||||
|
||||
## Reliability model
|
||||
|
||||
- **Deterministic plan:** `System.Random` plus seed 42 creates a repeatable plan
|
||||
for unchanged inputs and configuration.
|
||||
- **External-model caveat:** repeatable planning does not guarantee byte-identical
|
||||
LM Studio responses if model files, runtime, or service behavior change.
|
||||
- **Fingerprinting:** generation hashes source files and effective settings;
|
||||
changed inputs/settings reject an unsafe resume.
|
||||
- **Ordered parallelism:** generation is concurrent, but a single ordered writer
|
||||
buffers out-of-order results.
|
||||
- **Atomic checkpoints:** inserted rows and the checkpoint advance commit in the
|
||||
same database transaction.
|
||||
- **Fail closed:** invalid LM output, request failures, persistence failures,
|
||||
cancellation, mismatched durable state, or ambiguous incomplete runs fail the
|
||||
operation rather than silently producing success-shaped output.
|
||||
|
||||
## Classification model
|
||||
|
||||
The model is trained as a binary classifier plus category reporter:
|
||||
|
||||
- A positive SFT target has `is_medical_phi: true`, score `1.0`, a single result,
|
||||
and an entity type equal to one of the 12 categories.
|
||||
- A negative SFT target has `is_medical_phi: false`, score `0.0`, and no results.
|
||||
- Positive targets currently mark the entire input as `text` and `evidence`.
|
||||
This is classification with whole-input evidence, not span extraction.
|
||||
|
||||
## Known architecture gaps
|
||||
|
||||
- No tracked Presidio recognizer, registry, mapping, or deployment integration.
|
||||
- No tracked GGUF conversion or quantization workflow, although GGUF files are
|
||||
present.
|
||||
- Shell scaffolds have a legacy hard-coded checkout path.
|
||||
- The SFT conversion overwrites split files and is not idempotent.
|
||||
- The generator depends on an external normalization model whose exact artifact
|
||||
version is not fingerprinted.
|
||||
- Terminology source URLs, versions, retrieval dates, transformations, and
|
||||
licenses are not captured in a machine-readable manifest.
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
# Artifacts and repository map
|
||||
|
||||
## Source tree
|
||||
|
||||
```text
|
||||
.
|
||||
|-- PHI.Data.Generator/
|
||||
| |-- RLHF.Generator/ Core .NET generator and Data catalogs
|
||||
| |-- RLHF.Generator.Api/ ASP.NET Core API
|
||||
| |-- RLHF.Generator.Tests/ xUnit tests
|
||||
| |-- AddKeywordCatalog.sql
|
||||
| `-- RLHF.Generator.slnx
|
||||
|-- config/ Training, prompt, and chat-template config
|
||||
|-- data/ Exported and transformed JSONL
|
||||
|-- scaffolding/ Ordered operational wrappers
|
||||
|-- scripts/ Python pipeline implementation
|
||||
|-- outputs/ Adapters, evaluation, merged model, GGUF files
|
||||
|-- docs/ Project documentation and draw.io diagrams
|
||||
|-- pyproject.toml
|
||||
`-- uv.lock
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `PHI.Data.Generator/RLHF.Generator.Api/appsettings.json` | SQL, LM Studio, generation defaults |
|
||||
| `config/pipeline.json` | SQL export metadata, paths, splits, base model, LoRA/training/merge |
|
||||
| `config/system_prompt.txt` | Inference/SFT output contract |
|
||||
| `config/chat_template.jinja` | Llama-style chat serialization |
|
||||
| `pyproject.toml` / `uv.lock` | Python requirements and resolved environment |
|
||||
|
||||
## Data artifacts
|
||||
|
||||
| Path | State |
|
||||
|---|---|
|
||||
| `data/source.jsonl` | Source-shaped SQL export |
|
||||
| `data/train.jsonl` | Source-shaped after split; SFT-shaped after build |
|
||||
| `data/validation.jsonl` | Source-shaped after split; SFT-shaped after build |
|
||||
| `data/development.jsonl` | Source-shaped after split; SFT-shaped after build |
|
||||
|
||||
Because the same split paths change schema in place, record stage/checksums when
|
||||
archiving or automating the pipeline.
|
||||
|
||||
## Model artifacts
|
||||
|
||||
| Path | Meaning |
|
||||
|---|---|
|
||||
| `outputs/pilot-adapter` | Small pilot LoRA adapter |
|
||||
| `outputs/final-adapter` | Full LoRA adapter and retained checkpoints |
|
||||
| `outputs/development-evaluation` | Predictions and summary metrics |
|
||||
| `outputs/Invited-PHI-8B` | Standalone merged BF16 Hugging Face model |
|
||||
| `outputs/gguf` | BF16/Q4_K_M/Q5_K_M GGUF files |
|
||||
|
||||
The GGUF artifacts have no corresponding tracked conversion script, flags,
|
||||
tool revision, or verification step. They cannot be reproduced from repository
|
||||
automation alone.
|
||||
|
||||
## Reproducibility boundary
|
||||
|
||||
Tracked:
|
||||
|
||||
- source code, catalogs, patterns, seeds, prompts, configurations;
|
||||
- Python lock file;
|
||||
- generator tests;
|
||||
- current data/model/evaluation artifacts.
|
||||
|
||||
Not fully tracked:
|
||||
|
||||
- exact NIH/WHO/CDC source datasets and transformation history;
|
||||
- LM Studio model checksum/runtime build;
|
||||
- base model checksum/revision;
|
||||
- SQL Server contents before export;
|
||||
- host drivers/CUDA/toolchain snapshot;
|
||||
- GGUF conversion/quantization commands;
|
||||
- downstream Presidio implementation.
|
||||
|
||||
## Large files and source control
|
||||
|
||||
The repository tracks multi-gigabyte model/training artifacts. For sustainable
|
||||
operation, consider an artifact registry or Git LFS with immutable versioning,
|
||||
checksums, retention, access control, and release metadata. Keep source,
|
||||
configuration, metrics, and artifact manifests in Git even if weights move to
|
||||
external storage.
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
# Terminology and verified claims
|
||||
|
||||
## Recommended project explanation
|
||||
|
||||
> The generator combines medical category values with curated positive and
|
||||
> negative contextual patterns. It deterministically shuffles category values,
|
||||
> patterns, explicit subjects, and requested tenses, then uses LM Studio to
|
||||
> normalize grammar while preserving the protected medical concept and the
|
||||
> example's polarity. Patterns without an explicit subject remain subjectless
|
||||
> fragments; the normalizer assumes an implied singular member/patient only for
|
||||
> grammar and must not add that subject to the text.
|
||||
>
|
||||
> The project uses **Category** as its taxonomy term because the labels are
|
||||
> intended to integrate with a Presidio category/entity-type extension or
|
||||
> override. This repository contains the category-based generator and model
|
||||
> training pipeline, but the Presidio integration itself is maintained
|
||||
> elsewhere or has not yet been added here.
|
||||
>
|
||||
> Positive and negative contexts teach the model that a medical word alone is
|
||||
> insufficient. “Has diabetes” and “he had a reaction to shellfish” are intended
|
||||
> positives, while “she wrote a book about diabetes” and “had a reaction to the
|
||||
> horrible cover of Van Halen's Jump” are intended negatives.
|
||||
|
||||
## Review of the original statements
|
||||
|
||||
### “The Generator uses random subject, verb, etc.”
|
||||
|
||||
**Mostly correct, with precision needed.**
|
||||
|
||||
- It shuffles values, patterns, subjects, and a tense deck with a seeded
|
||||
pseudorandom generator.
|
||||
- Subjects are `He`, `She`, `They`, and `Member`.
|
||||
- It does not simply select arbitrary verbs. Patterns supply grammar expressions
|
||||
and sentence structure; local logic and LM Studio resolve/rewrite auxiliaries
|
||||
and verb phrases for subject/tense.
|
||||
- `[x]` starts with the canonical value. Allowed variants are supplied to the
|
||||
normalizer, rather than directly sampled as an independent random deck.
|
||||
- Planning is repeatable for unchanged seed/input/configuration; the external LM
|
||||
service is a remaining reproducibility variable.
|
||||
|
||||
### “It creates context for positive and negative tuning examples.”
|
||||
|
||||
**Correct.**
|
||||
|
||||
Every category has explicit positive and negative pattern files. The label is
|
||||
stored on each generated item and carried through export, SFT conversion,
|
||||
training, and evaluation.
|
||||
|
||||
### “Categories instead of Nodes/Edges because of Presidio.”
|
||||
|
||||
**Reasonable design intent, not verifiable integration in this repository.**
|
||||
|
||||
The implementation consistently uses categories/entity types, and that is an
|
||||
appropriate vocabulary for a Presidio recognizer extension. There is no graph
|
||||
model in this code, so “nodes/edges” would be misleading. However, no Presidio
|
||||
package, recognizer, registry, mapping, or runtime call is checked in. Phrase
|
||||
this as intended compatibility/integration, not as a current in-repository
|
||||
dependency.
|
||||
|
||||
### “Without a subject, or if a sentence fragment, it assumes the member is the subject.”
|
||||
|
||||
**Correct only when qualified.**
|
||||
|
||||
The implementation marks patterns without `[sub]` as fragments and does not
|
||||
insert `Member`. The normalizer uses an implied singular member/patient for
|
||||
agreement and tense only, and must preserve a subjectless output. `Member` is
|
||||
also one explicit choice in the subject deck when `[sub]` is present.
|
||||
|
||||
Preferred wording:
|
||||
|
||||
> Subjectless patterns are treated as fragments. For grammar only, the
|
||||
> normalizer assumes an implied singular member/patient but does not add that
|
||||
> subject to the generated text.
|
||||
|
||||
### Context examples
|
||||
|
||||
**They correctly express the intended distinction.**
|
||||
|
||||
The medical term or word “reaction” is not enough by itself. Actual patient
|
||||
medical assertions are positive; authored/educational or idiomatic non-medical
|
||||
uses are negative. These exact examples should also become automated golden
|
||||
tests if they are contractual behavior.
|
||||
|
||||
## Category versus entity type
|
||||
|
||||
- **Category** is the generator/database taxonomy object.
|
||||
- **Entity type** is the name emitted in model JSON.
|
||||
- Both currently use the same 12 `MEDICAL_*` identifiers.
|
||||
- **Keyword** is a canonical medical value assigned to a category.
|
||||
- **Variant** is an allowed surface/grammatical form of a keyword.
|
||||
- **Pattern** supplies contextual structure and polarity.
|
||||
- **Training item** is the resulting labeled text.
|
||||
|
||||
## What not to claim
|
||||
|
||||
Avoid saying:
|
||||
|
||||
- the model is a keyword matcher;
|
||||
- all category variants are randomly selected;
|
||||
- `Member` is inserted whenever a subject is absent;
|
||||
- this repository contains a working Presidio integration;
|
||||
- the tracked pipeline produces GGUF;
|
||||
- the current model extracts exact medical spans;
|
||||
- deterministic planning guarantees identical external LM responses;
|
||||
- NIH, WHO, or CDC endorse this project.
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
# Data and provenance
|
||||
|
||||
## Provenance statement
|
||||
|
||||
The project owner reports that the medical-term, condition, medication, and
|
||||
related terminology datasets used to assemble the category catalogs came from
|
||||
the **U.S. National Institutes of Health (NIH)**, the **World Health
|
||||
Organization (WHO)**, and the **U.S. Centers for Disease Control and Prevention
|
||||
(CDC)**.
|
||||
|
||||
The repository does not currently include a source manifest proving which
|
||||
catalog rows came from which organization. It also does not record source URLs,
|
||||
dataset/product names, versions, retrieval dates, licenses, notices, or
|
||||
transformation history. Accordingly, NIH/WHO/CDC origin is documented here as
|
||||
owner-provided provenance, not independently verifiable row-level provenance.
|
||||
|
||||
Before redistribution or production use, create a provenance manifest and
|
||||
review the terms for every underlying source. Public availability does not
|
||||
automatically mean unrestricted redistribution, and naming an organization does
|
||||
not imply its endorsement.
|
||||
|
||||
## Recommended source manifest
|
||||
|
||||
Track at least:
|
||||
|
||||
| Field | Example purpose |
|
||||
|---|---|
|
||||
| `source_organization` | NIH, WHO, or CDC |
|
||||
| `dataset_name` | Exact source product |
|
||||
| `source_url` | Stable landing/download URL |
|
||||
| `version` | Release/version/date |
|
||||
| `retrieved_at` | UTC retrieval timestamp |
|
||||
| `license_or_terms_url` | Governing terms |
|
||||
| `source_checksum` | Integrity and reproducibility |
|
||||
| `category_files` | Derived files |
|
||||
| `transformation` | Filtering, normalization, deduplication |
|
||||
| `attribution_required` | Required notice |
|
||||
| `reviewed_by` / `reviewed_at` | Governance approval |
|
||||
|
||||
## Generator source data
|
||||
|
||||
### Categories
|
||||
|
||||
Twelve text files define the fixed medical taxonomy. Each active row is a
|
||||
pipe-separated canonical term followed by grammatical or surface variants.
|
||||
These catalogs are the terminology/keyword source for generation and optional
|
||||
keyword persistence.
|
||||
|
||||
### Patterns
|
||||
|
||||
Each category has a positive and negative pattern file. Patterns are project
|
||||
authored/curated contextual templates; they determine whether generated text is
|
||||
labeled positive or negative. Catalog membership alone does not set polarity.
|
||||
|
||||
### Subjects
|
||||
|
||||
`Subjects.txt` is required to contain exactly:
|
||||
|
||||
```text
|
||||
He
|
||||
She
|
||||
They
|
||||
Member
|
||||
```
|
||||
|
||||
## SQL schema used by training export
|
||||
|
||||
The export reads `dbo.TrainingItem` joined to `dbo.Category` and writes:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 123,
|
||||
"category": "MEDICAL_TREATMENT",
|
||||
"is_positive": true,
|
||||
"value": "He is receiving physical therapy."
|
||||
}
|
||||
```
|
||||
|
||||
Validation requires exactly those fields, a unique positive integer ID, one of
|
||||
the 12 category names, a real JSON Boolean, and a non-empty string value.
|
||||
|
||||
## Split data
|
||||
|
||||
`create_splits.py` groups by `(category, is_positive)`, shuffles each group with
|
||||
a stable derived seed, allocates rounded 10% validation and 10% development
|
||||
counts, and sends the remainder to training. It then independently shuffles each
|
||||
whole split and verifies uniqueness, disjointness, and complete assignment.
|
||||
|
||||
Small strata can have materially different proportions or no training row after
|
||||
rounding. The script guarantees partition integrity, not minimum representation
|
||||
in every split.
|
||||
|
||||
## SFT data
|
||||
|
||||
`build_sft.py` overwrites each split with:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 123,
|
||||
"messages": [
|
||||
{"role": "system", "content": "...strict JSON policy..."},
|
||||
{"role": "user", "content": "He is receiving physical therapy."},
|
||||
{"role": "assistant", "content": "{\"is_medical_phi\":true,...}"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
For positives, the expected result contains the source category and uses the
|
||||
entire input for both `text` and `evidence`. For negatives, `results` is empty.
|
||||
This means the current task is whole-input contextual classification rather than
|
||||
precise entity-span extraction.
|
||||
|
||||
The conversion is destructive and non-idempotent. Recreate source-shaped splits
|
||||
before rerunning it.
|
||||
|
||||
## Data lineage
|
||||
|
||||
```text
|
||||
Owner-curated NIH/WHO/CDC-derived catalogs
|
||||
+ project-authored polarity patterns
|
||||
+ subject and tense decks
|
||||
|
|
||||
v
|
||||
normalized generated TrainingItem rows
|
||||
|
|
||||
v
|
||||
data/source.jsonl
|
||||
|
|
||||
v
|
||||
source-shaped train/validation/development JSONL
|
||||
|
|
||||
v
|
||||
SFT-shaped train/validation/development JSONL
|
||||
|
|
||||
v
|
||||
adapter, predictions, metrics, merged model
|
||||
```
|
||||
|
||||
## Privacy and governance
|
||||
|
||||
The checked-in source is synthetic/generated according to the system design,
|
||||
but that assumption should be verified operationally. Do not import real patient
|
||||
records without a separate privacy, consent, security, retention, and access
|
||||
control review. The model is designed to detect sensitive medical context; it is
|
||||
not itself a compliance determination or a substitute for human review.
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
|
||||
<diagram id="system-context" name="C4 Level 1 - System Context">
|
||||
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1600" pageHeight="900" math="0" shadow="0">
|
||||
<root>
|
||||
<mxCell id="0"/>
|
||||
<mxCell id="1" parent="0"/>
|
||||
<mxCell id="title" value="Medical PHI Model - C4 Level 1: System Context" style="text;html=1;align=center;verticalAlign=middle;fontSize=24;fontStyle=1" vertex="1" parent="1">
|
||||
<mxGeometry x="350" y="20" width="900" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="operator" value="<b>ML / Data Operator</b><br>Curates inputs, runs generation and training, reviews metrics" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=14" vertex="1" parent="1">
|
||||
<mxGeometry x="70" y="310" width="240" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="system" value="<b>Medical PHI Model System</b><br>Generates contextual examples and fine-tunes an 8B model to classify medical PHI and emit category-based JSON" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1168bd;strokeColor=#0b4884;fontColor=#ffffff;fontSize=16" vertex="1" parent="1">
|
||||
<mxGeometry x="560" y="280" width="420" height="170" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="sources" value="<b>NIH / WHO / CDC Sources</b><br>Owner-reported origins of medical terminology catalogs" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;fontSize=14" vertex="1" parent="1">
|
||||
<mxGeometry x="580" y="90" width="380" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="presidio" value="<b>Downstream Presidio Integration</b><br>Consumes compatible MEDICAL_* category/entity types; implementation is outside this repository" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=14" vertex="1" parent="1">
|
||||
<mxGeometry x="1180" y="300" width="300" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="subject" value="<b>Application / Reviewer</b><br>Submits text and consumes strict JSON classification results" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=14" vertex="1" parent="1">
|
||||
<mxGeometry x="1180" y="570" width="300" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="modelhost" value="<b>Model Runtime</b><br>Hosts the merged Hugging Face model or a separately converted GGUF artifact" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=14" vertex="1" parent="1">
|
||||
<mxGeometry x="560" y="590" width="420" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e1" value="Operates and reviews" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="operator" target="system">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e2" value="Curated terminology (provenance manifest needed)" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="sources" target="system">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e3" value="Produces model artifacts" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="system" target="modelhost">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e4" value="Category contract / model integration" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block;dashed=1" edge="1" parent="1" source="system" target="presidio">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e5" value="Inference requests" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="subject" target="modelhost">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e6" value="Strict JSON decisions" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=block" edge="1" parent="1" source="modelhost" target="subject">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="note" value="Dashed relationship is intended compatibility; Presidio code is not tracked here." style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1">
|
||||
<mxGeometry x="1110" y="100" width="330" height="90" as="geometry"/>
|
||||
</mxCell>
|
||||
</root>
|
||||
</mxGraphModel>
|
||||
</diagram>
|
||||
</mxfile>
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
|
||||
<diagram id="containers" name="C4 Level 2 - Containers">
|
||||
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="1000" math="0" shadow="0">
|
||||
<root>
|
||||
<mxCell id="0"/>
|
||||
<mxCell id="1" parent="0"/>
|
||||
<mxCell id="title" value="Medical PHI Model - C4 Level 2: Containers" style="text;html=1;align=center;fontSize=24;fontStyle=1" vertex="1" parent="1">
|
||||
<mxGeometry x="450" y="20" width="900" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="boundary" value="Medical PHI Model Repository" style="swimlane;html=1;startSize=35;rounded=1;fillColor=none;strokeColor=#1168bd;fontSize=16;fontStyle=1" vertex="1" parent="1">
|
||||
<mxGeometry x="240" y="80" width="1320" height="780" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="operator" value="<b>Operator</b>" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=14" vertex="1" parent="1">
|
||||
<mxGeometry x="30" y="380" width="150" height="70" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="catalogs" value="<b>Catalogs and Patterns</b><br>Text files<br>12 categories, variants, subjects, positive/negative templates" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="50" y="80" width="250" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="api" value="<b>Generator API</b><br>ASP.NET Core / .NET 10<br>Validates inputs, plans examples, calls LM Studio, persists ordered batches" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="380" y="80" width="280" height="140" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="sql" value="<b>Training Database</b><br>SQL Server<br>Categories, keywords, generation runs, training items" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="760" y="80" width="250" height="140" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="jsonl" value="<b>Training Data</b><br>JSONL files<br>Source, train, validation, development" style="shape=folder;tabWidth=60;tabHeight=20;tabPosition=left;html=1;whiteSpace=wrap;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="760" y="330" width="250" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="pipeline" value="<b>Training Pipeline</b><br>Python 3.12 / PyTorch / TRL / PEFT<br>Validate, split, SFT, train, evaluate, merge" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="380" y="540" width="280" height="140" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="artifacts" value="<b>Model Artifacts</b><br>Filesystem<br>LoRA adapters, metrics, merged BF16 model, untracked-process GGUF" style="shape=folder;tabWidth=60;tabHeight=20;tabPosition=left;html=1;whiteSpace=wrap;fillColor=#438dd5;strokeColor=#0b4884;fontColor=#ffffff;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="50" y="540" width="250" height="140" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="lm" value="<b>LM Studio</b><br>OpenAI-compatible API<br>qwen/qwen3.5-9b normalization" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#999999;strokeColor=#666666;fontColor=#ffffff;fontSize=13" vertex="1" parent="1">
|
||||
<mxGeometry x="1600" y="190" width="170" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="base" value="<b>Base Model</b><br>Local Llama3-OpenBioLLM-8B" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#999999;strokeColor=#666666;fontColor=#ffffff;fontSize=13" vertex="1" parent="1">
|
||||
<mxGeometry x="1600" y="600" width="170" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e1" value="Runs" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="operator" target="api"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e2" value="Reads" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="catalogs" target="api"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e3" value="Normalizes text" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="api" target="lm"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e4" value="Persists and checkpoints" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="api" target="sql"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e5" value="PowerShell export" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="sql" target="jsonl"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e6" value="Reads and rewrites" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="jsonl" target="pipeline"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e7" value="Loads weights/tokenizer" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="base" target="pipeline"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e8" value="Writes" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="pipeline" target="artifacts"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e9" value="Operates/reviews" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="operator" target="pipeline"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
</root>
|
||||
</mxGraphModel>
|
||||
</diagram>
|
||||
</mxfile>
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
|
||||
<diagram id="generator-components" name="C4 Level 3 - Generator Components">
|
||||
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="1000" math="0" shadow="0">
|
||||
<root>
|
||||
<mxCell id="0"/>
|
||||
<mxCell id="1" parent="0"/>
|
||||
<mxCell id="title" value="Generator API - C4 Level 3: Components" style="text;html=1;align=center;fontSize=24;fontStyle=1" vertex="1" parent="1">
|
||||
<mxGeometry x="450" y="20" width="900" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="boundary" value="RLHF.Generator.Api + RLHF.Generator" style="swimlane;html=1;startSize=35;rounded=1;fillColor=none;strokeColor=#1168bd;fontSize=16;fontStyle=1" vertex="1" parent="1">
|
||||
<mxGeometry x="160" y="80" width="1450" height="780" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="routes" value="<b>HTTP Endpoints</b><br>Program.cs<br>Validation, contracts, status mapping" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="50" y="90" width="240" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="reader" value="<b>TrainingDataReader</b><br>Validates and parses subjects, categories, and patterns" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="360" y="90" width="240" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="generation" value="<b>TrainingGenerationService</b><br>Plans, fingerprints, resumes, parallelizes, orders, validates, checkpoints" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1168bd;strokeColor=#0b4884;fontColor=#ffffff;fontSize=14" vertex="1" parent="boundary">
|
||||
<mxGeometry x="680" y="280" width="300" height="160" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="lmclient" value="<b>LmStudioClient</b><br>Local token resolution, prompt construction, HTTP completion" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="1070" y="90" width="260" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="keyword" value="<b>Keyword Services</b><br>Load canonical values/variants; dry-run or apply unique backfill" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="50" y="550" width="260" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="reset" value="<b>TrainingItemResetService</b><br>Clears generated rows and run checkpoints" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="380" y="550" width="250" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dbcontext" value="<b>RlhfGeneratorDbContext</b><br>EF Core mappings, seeds, indexes, relationships, transactions" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="720" y="550" width="270" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="options" value="<b>Options</b><br>GenerationOptions<br>LmStudioOptions<br>SentenceTenses" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#85bbf0;strokeColor=#0b4884;fontSize=13" vertex="1" parent="boundary">
|
||||
<mxGeometry x="1070" y="550" width="260" height="130" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="data" value="Catalog and pattern files" style="shape=folder;tabWidth=50;tabHeight=18;tabPosition=left;html=1;whiteSpace=wrap;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1">
|
||||
<mxGeometry x="20" y="200" width="120" height="90" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="lm" value="LM Studio API" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6" vertex="1" parent="1">
|
||||
<mxGeometry x="1640" y="200" width="120" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="sql" value="SQL Server" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#e1d5e7;strokeColor=#9673a6" vertex="1" parent="1">
|
||||
<mxGeometry x="1640" y="640" width="120" height="90" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e1" value="Dispatches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="routes" target="generation"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e2" value="Reads definitions" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="generation" target="reader"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e3" value="Normalizes" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="generation" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e4" value="HTTP chat completion" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="lmclient" target="lm"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e5" value="Persists batches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="generation" target="dbcontext"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e6" value="EF Core SQL" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="dbcontext" target="sql"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e7" value="Reads" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="data" target="reader"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e8" value="Configures" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="options" target="generation"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e9" value="Configures" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="options" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e10" value="Dispatches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="routes" target="keyword"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e11" value="Dispatches" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="routes" target="reset"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e12" value="Uses" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="keyword" target="dbcontext"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e13" value="Uses" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=block" edge="1" parent="1" source="reset" target="dbcontext"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
</root>
|
||||
</mxGraphModel>
|
||||
</diagram>
|
||||
</mxfile>
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T15:00:00.000Z" agent="Copilot" version="24.7.17">
|
||||
<diagram id="generator-code" name="C4 Level 4 - Generator Code">
|
||||
<mxGraphModel dx="1422" dy="794" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="1100" math="0" shadow="0">
|
||||
<root>
|
||||
<mxCell id="0"/>
|
||||
<mxCell id="1" parent="0"/>
|
||||
<mxCell id="title" value="TrainingGenerationService - C4 Level 4: Code Relationships" style="text;html=1;align=center;fontSize=24;fontStyle=1" vertex="1" parent="1">
|
||||
<mxGeometry x="400" y="20" width="1000" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="service" value="<b>TrainingGenerationService</b><br>+ GenerateAsync()<br>- GenerateCategoryAsync()<br>- CreatePlan()<br>- GeneratePassAsync()<br>- PersistOrderedAsync()<br>- FindResumableRunAsync()<br>- ValidateGeneratedValue()" style="swimlane;html=1;startSize=30;rounded=0;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="660" y="310" width="360" height="280" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="reader" value="<b>TrainingDataReader</b><br>+ ValidateAsync()<br>+ ReadCategoryValuesAsync()<br>+ ReadPatternsAsync()" style="swimlane;html=1;startSize=30;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="130" y="120" width="290" height="150" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="definitions" value="<b>Training definitions</b><br>TrainingCategories<br>TrainingSubjects<br>TrainingCategoryDefinition<br>TrainingCategoryValue" style="swimlane;html=1;startSize=30;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="120" y="390" width="310" height="180" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="lmclient" value="<b>LmStudioClient</b><br>+ CompleteAsync()<br>- ResolveFragmentTokens()<br>- ResolveArticles()<br>- SendCompletionAsync()" style="swimlane;html=1;startSize=30;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="1250" y="120" width="300" height="180" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="options" value="<b>Configuration</b><br>GenerationOptions<br>LmStudioOptions<br>SentenceTenses" style="swimlane;html=1;startSize=30;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="1260" y="390" width="280" height="150" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="context" value="<b>RlhfGeneratorDbContext</b><br>DbSet<Category><br>DbSet<TrainingItem><br>DbSet<GenerationRun><br>DbSet<Keyword><br>DbSet<KeywordVariant>" style="swimlane;html=1;startSize=30;fillColor=#f8cecc;strokeColor=#b85450;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="650" y="760" width="380" height="200" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="run" value="<b>GenerationRun</b><br>Status, seed, fingerprints<br>checkpoint and counts<br>failure/cancellation data" style="swimlane;html=1;startSize=30;fillColor=#f8cecc;strokeColor=#b85450;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="180" y="760" width="300" height="160" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="item" value="<b>TrainingItem</b><br>RecordKey, PlanOrdinal<br>Category, Keyword<br>IsPositive, Value" style="swimlane;html=1;startSize=30;fillColor=#f8cecc;strokeColor=#b85450;fontSize=13;align=left;spacingLeft=8" vertex="1" parent="1">
|
||||
<mxGeometry x="1200" y="760" width="300" height="160" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="channel" value="<b>Bounded Channel</b><br>Parallel workers write completed plan items; ordered writer consumes" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13" vertex="1" parent="1">
|
||||
<mxGeometry x="690" y="120" width="300" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e1" value="Validates and reads inputs" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="reader"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e2" value="Returns" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="reader" target="definitions"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e3" value="Plans with" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="definitions"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e4" value="Calls concurrently" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e5" value="Binds" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="options" target="service"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e6" value="Binds" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="options" target="lmclient"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e7" value="Coordinates producer/consumer" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="channel"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e8" value="Queries and transacts" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=open" edge="1" parent="1" source="service" target="context"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e9" value="Tracks" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=diamond;endFill=0" edge="1" parent="1" source="context" target="run"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e10" value="Tracks" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=diamond;endFill=0" edge="1" parent="1" source="context" target="item"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="e11" value="Owns ordered items" style="edgeStyle=orthogonalEdgeStyle;html=1;endArrow=diamond;endFill=1" edge="1" parent="1" source="run" target="item"><mxGeometry relative="1" as="geometry"/></mxCell>
|
||||
<mxCell id="note" value="Plan order: category positive pass, then negative pass. Workers may finish out of order; persistence waits for the next contiguous ordinal and commits rows plus checkpoint atomically." style="shape=note;whiteSpace=wrap;html=1;backgroundOutline=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=12" vertex="1" parent="1">
|
||||
<mxGeometry x="1080" y="560" width="430" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
</root>
|
||||
</mxGraphModel>
|
||||
</diagram>
|
||||
</mxfile>
|
||||
|
|
@ -0,0 +1,201 @@
|
|||
# Generator
|
||||
|
||||
## Purpose
|
||||
|
||||
`PHI.Data.Generator` creates labeled contextual medical-language examples. It
|
||||
does not generate a graph of nodes and edges. Its central domain concepts are
|
||||
categories, category values/keywords, patterns, subjects, and training items.
|
||||
|
||||
## Projects
|
||||
|
||||
| Project | Role |
|
||||
|---|---|
|
||||
| `RLHF.Generator` | Catalog parsing, planning, LM client, persistence, generation, keyword services |
|
||||
| `RLHF.Generator.Api` | ASP.NET Core host and HTTP endpoints |
|
||||
| `RLHF.Generator.Tests` | xUnit unit/integration tests using SQLite for test persistence |
|
||||
|
||||
All three target .NET 10. Production persistence uses EF Core SQL Server.
|
||||
|
||||
## Taxonomy
|
||||
|
||||
The fixed categories are:
|
||||
|
||||
1. `MEDICAL_ALLERGY`
|
||||
2. `MEDICAL_DIAGNOSIS`
|
||||
3. `MEDICAL_FAMILY_HISTORY`
|
||||
4. `MEDICAL_MEDICATION`
|
||||
5. `MEDICAL_MENTAL_HEALTH`
|
||||
6. `MEDICAL_OTHER`
|
||||
7. `MEDICAL_PHYSICAL_HEALTH`
|
||||
8. `MEDICAL_PROCEDURE`
|
||||
9. `MEDICAL_SUBSTANCE_USE`
|
||||
10. `MEDICAL_SYMPTOM`
|
||||
11. `MEDICAL_TEST`
|
||||
12. `MEDICAL_TREATMENT`
|
||||
|
||||
Lookup is case-insensitive; stored/returned names use these exact values.
|
||||
|
||||
## Input layout and grammar
|
||||
|
||||
```text
|
||||
RLHF.Generator/Data/
|
||||
Subjects.txt
|
||||
Categories/
|
||||
MEDICAL_ALLERGY.txt
|
||||
...
|
||||
Patterns/
|
||||
MEDICAL_ALLERGY.positive.txt
|
||||
MEDICAL_ALLERGY.negative.txt
|
||||
...
|
||||
```
|
||||
|
||||
Blank lines and lines beginning with `#` are ignored. A category line is:
|
||||
|
||||
```text
|
||||
canonical form|allowed variant|another allowed variant
|
||||
```
|
||||
|
||||
Forms must be non-empty and case-insensitively unique within a category. The
|
||||
first is canonical. The remainder are alternative forms available to the
|
||||
normalization model.
|
||||
|
||||
The exact subjects are `He`, `She`, `They`, and `Member`.
|
||||
|
||||
Pattern tokens include:
|
||||
|
||||
| Token | Meaning |
|
||||
|---|---|
|
||||
| `[x]` | Canonical category value; allowed variants are supplied to LM Studio |
|
||||
| `[sub]` | A subject selected from the shuffled subject deck |
|
||||
| `[is\|are\|was\|were]` | Copula selected/rephrased for subject and tense |
|
||||
| `[has\|have\|had]` | Possession/auxiliary selected for subject and tense |
|
||||
| `[a\|an]` | Article resolved locally from the following word |
|
||||
|
||||
The legacy `[s]` placeholder is rejected.
|
||||
|
||||
## Positive and negative examples
|
||||
|
||||
Every category has an independent positive and negative pattern set. A positive
|
||||
pattern describes real medical context; a negative pattern uses the same domain
|
||||
language without asserting medical information about the person being analyzed.
|
||||
|
||||
Representative intended behavior:
|
||||
|
||||
| Input | Expected label | Why |
|
||||
|---|---:|---|
|
||||
| `Has diabetes` | Positive | Subjectless clinical fragment with an implied member/patient |
|
||||
| `He had a reaction to shellfish` | Positive | Actual allergy/reaction assertion |
|
||||
| `She wrote a book about diabetes` | Negative | Creative/educational context, not her diagnosis |
|
||||
| `Had a reaction to the horrible cover of Van Halen's Jump` | Negative | Idiomatic non-medical reaction |
|
||||
|
||||
Those examples describe the intended learned behavior; they are not hard-coded
|
||||
rules or keyword matchers in the inference pipeline.
|
||||
|
||||
## Plan construction
|
||||
|
||||
For each category, the service runs the positive pass and then the negative
|
||||
pass. It partitions patterns into:
|
||||
|
||||
- **value-bound:** contains `[x]`;
|
||||
- **generic:** does not contain `[x]`.
|
||||
|
||||
It rejects values without any value-bound pattern and value-bound patterns when
|
||||
the category has no values. It then:
|
||||
|
||||
1. shuffles values, value-bound patterns, and generic patterns;
|
||||
2. creates shuffled/cycling subject and tense decks;
|
||||
3. emits every generic pattern once;
|
||||
4. emits `max(value-bound pattern count, value count)` value-bound plans;
|
||||
5. reshuffles a deck whenever it is exhausted.
|
||||
|
||||
The tense deck is `null`, `past`, `present`, and `future`. The canonical category
|
||||
value is placed in `[x]`; variants are not independently randomly sampled.
|
||||
|
||||
Because each polarity is planned from its own pattern/value counts, positive and
|
||||
negative totals are not guaranteed to be equal.
|
||||
|
||||
## Subject and fragment behavior
|
||||
|
||||
The precise behavior is:
|
||||
|
||||
- A pattern containing `[sub]` gets one shuffled explicit subject.
|
||||
- A pattern without `[sub]` is marked as a fragment and gets no inserted
|
||||
subject.
|
||||
- `Member` is one of the four explicit subject choices; it is not a fallback
|
||||
value.
|
||||
- For a fragment, the LM system prompt uses an implied singular member/patient
|
||||
only to choose grammar. It explicitly forbids returning or inventing that
|
||||
subject.
|
||||
- Some fragments are completed locally without an LM call when no unresolved
|
||||
choice, protected-form decision, or meaningful tense rewrite remains.
|
||||
|
||||
Therefore the safe summary is: **subjectless patterns remain subjectless, while
|
||||
grammar assumes an implied singular member/patient.**
|
||||
|
||||
## LM Studio normalization
|
||||
|
||||
Default settings:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Endpoint | `http://localhost:1234/v1/chat/completions` |
|
||||
| Model | `qwen/qwen3.5-9b` |
|
||||
| Temperature | `0.0` |
|
||||
| Top-p | `1.0` |
|
||||
| Max tokens | `96` |
|
||||
| Seed | `42` |
|
||||
| Timeout | `300` seconds |
|
||||
|
||||
Each request carries fragment status, optional protected subject, protected
|
||||
medical term and allowed forms, requested tense, and the sentence/template. The
|
||||
system prompt requires exactly one unwrapped line, preserves meaning and
|
||||
polarity, resolves all bracket expressions, and forbids invented fragment
|
||||
predicates.
|
||||
|
||||
Generation rejects empty output, newlines, unresolved square brackets,
|
||||
quotation/JSON/code-fence wrappers, and common response labels.
|
||||
|
||||
## Concurrency and durability
|
||||
|
||||
Defaults are seed 42, parallelism 8, save batches of 24, and progress logs every
|
||||
5 seconds.
|
||||
|
||||
Workers generate concurrently into a bounded channel. A single persistence
|
||||
writer places results into a sorted buffer and commits only the next contiguous
|
||||
plan ordinals. Each transaction inserts the batch and advances the generation
|
||||
checkpoint together.
|
||||
|
||||
A run can resume only when its seed, input fingerprint, configuration
|
||||
fingerprint, expected count, committed row count, and last ordinal agree.
|
||||
Changed inputs/configuration or inconsistent persisted state are rejected.
|
||||
|
||||
## Keyword catalog
|
||||
|
||||
Keyword loading uses each category file's canonical value and variants. Loading
|
||||
is additive, case-insensitive, and idempotent. It does not update existing
|
||||
training items. Backfill is a separate operation:
|
||||
|
||||
- dry-run is available;
|
||||
- only unique matches are assigned;
|
||||
- unmatched or ambiguous rows remain without a keyword.
|
||||
|
||||
## Database migrations
|
||||
|
||||
The migrations progress through:
|
||||
|
||||
1. initial `Category` and `TrainingItem` schema;
|
||||
2. durable generation-run/checkpoint support;
|
||||
3. `Keyword`, `KeywordVariant`, and training-item keyword linkage.
|
||||
|
||||
`AddKeywordCatalog.sql` provides a SQL alternative for the keyword phase.
|
||||
|
||||
## Configuration changes that invalidate resume
|
||||
|
||||
Treat these as generation identity:
|
||||
|
||||
- category, pattern, or subject file contents/paths;
|
||||
- random seed;
|
||||
- concurrency and save/progress options included in the fingerprint;
|
||||
- LM endpoint/model/prompt and generation settings.
|
||||
|
||||
Start a clean run rather than bypassing fingerprint checks.
|
||||
|
|
@ -0,0 +1,254 @@
|
|||
# End-to-end runbook
|
||||
|
||||
This procedure starts with catalogs and ends with a merged model. Run commands
|
||||
from the repository root unless stated otherwise.
|
||||
|
||||
## 1. Record the run
|
||||
|
||||
Capture:
|
||||
|
||||
- Git revision and clean/dirty state;
|
||||
- operator and UTC start time;
|
||||
- catalog/pattern checksums and provenance manifest;
|
||||
- SQL Server target;
|
||||
- LM Studio model artifact/version and settings;
|
||||
- base model path/revision/checksum;
|
||||
- GPU, driver, CUDA, Python, and dependency versions;
|
||||
- expected output locations and acceptance thresholds.
|
||||
|
||||
## 2. Prerequisite checks
|
||||
|
||||
Install/prepare:
|
||||
|
||||
- .NET 10 SDK;
|
||||
- SQL Server reachable with schema-creation privileges;
|
||||
- LM Studio with `qwen/qwen3.5-9b` (or an intentionally approved replacement);
|
||||
- Windows PowerShell for SQL export;
|
||||
- WSL/Linux, Git, build tools, and `uv`;
|
||||
- Python 3.12;
|
||||
- NVIDIA CUDA GPU with BF16 and sufficient VRAM;
|
||||
- local `Llama3-OpenBioLLM-8B` base model;
|
||||
- enough CPU RAM/disk for an unquantized 8B BF16 merge.
|
||||
|
||||
Correct the legacy `/d/ai/training/openbiollm-phi` path in every shell scaffold,
|
||||
or use the direct commands below from the real checkout.
|
||||
|
||||
## 3. Validate generator inputs
|
||||
|
||||
Confirm:
|
||||
|
||||
- exactly 12 category files;
|
||||
- one positive and one negative pattern file per category;
|
||||
- subjects exactly `He`, `She`, `They`, `Member`;
|
||||
- no legacy `[s]` tokens;
|
||||
- canonical/variant forms are unique;
|
||||
- each category has values and compatible `[x]` patterns.
|
||||
|
||||
The generator also enforces these checks when a run starts.
|
||||
|
||||
## 4. Configure and migrate SQL Server
|
||||
|
||||
Review the `RLHFGenerator` connection string in
|
||||
`PHI.Data.Generator/RLHF.Generator.Api/appsettings.json`. Prefer environment
|
||||
configuration or user secrets for deployment-specific values.
|
||||
|
||||
From `PHI.Data.Generator`:
|
||||
|
||||
```powershell
|
||||
dotnet restore .\RLHF.Generator.slnx
|
||||
dotnet ef database update `
|
||||
--project .\RLHF.Generator\RLHF.Generator.csproj `
|
||||
--startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj
|
||||
```
|
||||
|
||||
Back up an existing database before migration or reset.
|
||||
|
||||
## 5. Test the generator
|
||||
|
||||
```powershell
|
||||
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
|
||||
```
|
||||
|
||||
Stop if any test fails.
|
||||
|
||||
## 6. Start LM Studio
|
||||
|
||||
1. Load the configured normalization model.
|
||||
2. Start the local OpenAI-compatible server.
|
||||
3. Confirm `http://localhost:1234/v1/chat/completions` is reachable.
|
||||
4. Keep temperature, seed, token limit, and model name aligned with generator
|
||||
configuration.
|
||||
|
||||
## 7. Start the generator API
|
||||
|
||||
```powershell
|
||||
dotnet run --project `
|
||||
.\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj
|
||||
```
|
||||
|
||||
Note the actual listening URL. In Development, inspect Swagger/Scalar. Call
|
||||
`GET /` as a basic process check.
|
||||
|
||||
## 8. Load keywords and generate
|
||||
|
||||
In another PowerShell window:
|
||||
|
||||
```powershell
|
||||
$base = "http://localhost:<port>"
|
||||
|
||||
Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load"
|
||||
Invoke-RestMethod -Method Post -Uri "$base/api/generation/run"
|
||||
```
|
||||
|
||||
The generation request remains open until completion. Monitor API logs and SQL
|
||||
`GenerationRun`. On cancellation/failure, correct the cause and call the same
|
||||
endpoint to resume only if the service accepts the matching fingerprints.
|
||||
|
||||
Do not edit catalogs, patterns, generation settings, or LM settings mid-run.
|
||||
|
||||
Optional keyword linkage:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod -Method Post `
|
||||
-Uri "$base/api/training-items/backfill-keywords?dryRun=true"
|
||||
Invoke-RestMethod -Method Post `
|
||||
-Uri "$base/api/training-items/backfill-keywords?dryRun=false"
|
||||
```
|
||||
|
||||
## 9. Export SQL data
|
||||
|
||||
Review SQL server/database values in `config/pipeline.json`, then run:
|
||||
|
||||
```powershell
|
||||
.\scaffolding\03-export-sql.ps1
|
||||
```
|
||||
|
||||
Confirm `data/source.jsonl` exists and record positive/negative totals.
|
||||
|
||||
## 10. Prepare the Python environment in WSL
|
||||
|
||||
From the actual WSL path of the repository:
|
||||
|
||||
```bash
|
||||
uv python install 3.12.13
|
||||
uv venv --python 3.12.13
|
||||
uv lock
|
||||
uv sync --frozen
|
||||
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"
|
||||
```
|
||||
|
||||
Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the
|
||||
environment and contain `git add`/`git commit` commands.
|
||||
|
||||
## 11. Review pipeline configuration
|
||||
|
||||
Update `config/pipeline.json` deliberately:
|
||||
|
||||
- SQL connection metadata;
|
||||
- actual WSL base-model path;
|
||||
- split proportions/seed;
|
||||
- sequence length;
|
||||
- LoRA and training hyperparameters;
|
||||
- output directories.
|
||||
|
||||
Archive the exact configuration used.
|
||||
|
||||
## 12. Validate and split
|
||||
|
||||
```bash
|
||||
uv run python scripts/validate_source.py
|
||||
uv run python scripts/create_splits.py
|
||||
```
|
||||
|
||||
Review counts per category/polarity/split. Pay special attention to small strata.
|
||||
|
||||
## 13. Build SFT conversations
|
||||
|
||||
```bash
|
||||
uv run python scripts/build_sft.py
|
||||
```
|
||||
|
||||
This overwrites all split files. If repeated, run `create_splits.py` first.
|
||||
Inspect a positive and negative example from every category.
|
||||
|
||||
## 14. Smoke-test the base model
|
||||
|
||||
```bash
|
||||
uv run python scripts/model_smoke_test.py
|
||||
```
|
||||
|
||||
Confirm successful loading, finite logits, and expected GPU memory use.
|
||||
|
||||
## 15. Run the pilot
|
||||
|
||||
```bash
|
||||
uv run python scripts/train_pilot.py
|
||||
```
|
||||
|
||||
Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss
|
||||
before full training.
|
||||
|
||||
## 16. Train the full adapter
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py
|
||||
```
|
||||
|
||||
This replaces `outputs/final-adapter`. Preserve any prior accepted adapter first.
|
||||
Monitor training/evaluation loss, checkpoints, utilization, and disk.
|
||||
|
||||
## 17. Evaluate and rescore
|
||||
|
||||
```bash
|
||||
uv run python scripts/evaluate.py
|
||||
uv run python scripts/rescore_evaluation.py
|
||||
```
|
||||
|
||||
Compare `outputs/development-evaluation/summary.json` with acceptance thresholds.
|
||||
Review every error, particularly false negatives, false positives, category
|
||||
errors, malformed JSON, and contextual-negative failures.
|
||||
|
||||
## 18. Verify stopping
|
||||
|
||||
```bash
|
||||
uv run python scripts/verify_generation_stop.py
|
||||
```
|
||||
|
||||
All five sampled outputs must be exact parseable JSON.
|
||||
|
||||
## 19. Merge the adapter
|
||||
|
||||
```bash
|
||||
uv run python scripts/merge_model.py
|
||||
```
|
||||
|
||||
This replaces `outputs/Invited-PHI-8B` atomically after verification. It can
|
||||
require substantial CPU RAM and disk.
|
||||
|
||||
## 20. Validate and publish artifacts
|
||||
|
||||
Before release:
|
||||
|
||||
1. Add/run a merged-model inference smoke test.
|
||||
2. Confirm tokenizer, chat template, system prompt, configs, and all weight
|
||||
shards are present.
|
||||
3. Record checksums and total sizes.
|
||||
4. Preserve adapter, merged model, metrics, predictions, config, source/split
|
||||
checksums, Git revision, environment lock, and provenance manifest.
|
||||
5. If GGUF is required, use an approved conversion/quantization process and
|
||||
record it; no such process is tracked here.
|
||||
6. Run downstream Presidio integration and end-user acceptance tests.
|
||||
|
||||
## Recovery
|
||||
|
||||
| Failure | Action |
|
||||
|---|---|
|
||||
| LM Studio unavailable/timeout | Restore service/model, then resume unchanged run |
|
||||
| Generator invalid output | Inspect request/model/prompt; do not bypass validation |
|
||||
| Fingerprint mismatch | Restore original inputs/settings or intentionally reset/start clean |
|
||||
| SQL batch failure | Correct database cause; checkpointed committed batches remain |
|
||||
| Split/SFT rerun failure | Recreate splits from `source.jsonl`, then rebuild SFT |
|
||||
| CUDA OOM | Reduce approved batch/sequence settings, rerun pilot, document change |
|
||||
| Training output exists | Archive accepted output; scripts intentionally replace output dirs |
|
||||
| Evaluation failure | Preserve raw logs/artifacts, fix root cause, rerun evaluation |
|
||||
| Merge resource failure | Increase CPU RAM/disk or use an approved merge host |
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
# Testing and validation
|
||||
|
||||
## .NET automated tests
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
|
||||
```
|
||||
|
||||
The xUnit suite covers:
|
||||
|
||||
| Test area | Coverage |
|
||||
|---|---|
|
||||
| LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming |
|
||||
| Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output |
|
||||
| Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs |
|
||||
| Endpoint behavior | Generation endpoint waits for actual generation and commit |
|
||||
| Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs |
|
||||
| Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity |
|
||||
| Provider isolation | SQL Server production DI and SQLite-only test setup |
|
||||
|
||||
Tests use SQLite for integration persistence while explicitly verifying that the
|
||||
production service registers SQL Server.
|
||||
|
||||
## Python verification stages
|
||||
|
||||
There is no Python unit-test suite. Verification is operational:
|
||||
|
||||
1. `validate_source.py` validates exported schema and taxonomy.
|
||||
2. `create_splits.py` checks uniqueness, disjointness, and complete assignment.
|
||||
3. `build_sft.py` checks message shape and parses generated assistant JSON.
|
||||
4. `model_smoke_test.py` loads the base model and rejects non-finite logits.
|
||||
5. `train_pilot.py` verifies a small QLoRA training path.
|
||||
6. `evaluate.py` validates generated JSON/schema and computes metrics.
|
||||
7. `verify_generation_stop.py` checks exact JSON parsing on five examples.
|
||||
8. `merge_model.py` verifies merged weights and configuration files.
|
||||
|
||||
## Current tracked evaluation snapshot
|
||||
|
||||
`outputs/development-evaluation/summary.json` currently reports:
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| Development records | 2,027 |
|
||||
| Valid JSON | 100% |
|
||||
| Valid schema | 100% |
|
||||
| Classification accuracy | 99.852% |
|
||||
| True positives / false negatives | 996 / 0 |
|
||||
| True negatives / false positives | 1,028 / 3 |
|
||||
| Precision | 99.700% |
|
||||
| Recall | 100% |
|
||||
| F1 | 99.850% |
|
||||
| Entity category accuracy | 98.394% |
|
||||
| Entity text accuracy | 100% |
|
||||
| Evidence accuracy | 100% |
|
||||
|
||||
These values describe the checked-in development artifact, not an independent
|
||||
benchmark or prospective production guarantee.
|
||||
|
||||
## Recommended acceptance gates
|
||||
|
||||
Before publishing a new model:
|
||||
|
||||
- all .NET tests pass;
|
||||
- source validation and split integrity pass;
|
||||
- the pilot finishes with finite loss;
|
||||
- full training selects/saves a valid adapter;
|
||||
- JSON and schema validity remain 100%;
|
||||
- classification/category metrics meet a pre-agreed threshold;
|
||||
- false positives and false negatives are manually reviewed;
|
||||
- generation-stop verification passes;
|
||||
- merged-model loading/inference is added and passes;
|
||||
- provenance and license manifest is complete;
|
||||
- a Presidio integration test is run in the downstream repository;
|
||||
- checksums, configuration, Git revision, base model revision, and environment
|
||||
versions are archived.
|
||||
|
||||
## Test gaps
|
||||
|
||||
- No direct Python unit tests for schema, splitting edge cases, SFT conversion,
|
||||
JSON stopping, scoring, or merge logic.
|
||||
- No inference smoke test for the merged model.
|
||||
- No tracked GGUF conversion/quantization verification.
|
||||
- No end-to-end test from generator API through SQL export to training data.
|
||||
- No checked-in Presidio integration or category mapping test.
|
||||
- No fixed golden tests for the four representative contextual sentences.
|
||||
|
||||
Adding those contextual examples as regression tests would make the intended
|
||||
positive/negative distinction executable rather than documentary.
|
||||
|
|
@ -0,0 +1,152 @@
|
|||
# Training pipeline
|
||||
|
||||
## Runtime
|
||||
|
||||
- Python `3.12.*` (the setup script installs 3.12.13)
|
||||
- `uv` for interpreter/dependency management
|
||||
- PyTorch 2.11.0 from the CUDA 12.8 index
|
||||
- Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1
|
||||
- bitsandbytes 0.49.2 for 4-bit NF4 loading
|
||||
- CUDA-capable NVIDIA GPU with BF16 support
|
||||
- WSL/Linux for shell wrappers
|
||||
- substantial CPU RAM/disk for BF16 merge
|
||||
|
||||
The canonical configuration is `config/pipeline.json`.
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
### 00 - Audit
|
||||
|
||||
`scaffolding/00-audit.sh` checks Linux/WSL, Python, Git, compiler/build tools,
|
||||
download tools, NVIDIA utilities, memory, disk, base model, and repository state.
|
||||
|
||||
### 01 - Create environment
|
||||
|
||||
`scaffolding/01-create-venv.sh` installs `uv`, installs Python 3.12.13, and
|
||||
recreates `.venv`. It also attempts to stage and commit files, so inspect/remove
|
||||
those Git commands before using it in an automated or shared worktree.
|
||||
|
||||
### 02 - Install dependencies
|
||||
|
||||
`scaffolding/02-install-dependencies.sh` locks/synchronizes dependencies and
|
||||
fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime
|
||||
information. It also contains Git commit commands.
|
||||
|
||||
### 03 - Export SQL
|
||||
|
||||
`scaffolding/03-export-sql.ps1` uses Windows integrated authentication, queries
|
||||
SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative
|
||||
counts.
|
||||
|
||||
### 04 - Validate source
|
||||
|
||||
`scripts/validate_source.py` enforces the four-field schema, IDs, taxonomy,
|
||||
Boolean labels, non-empty values, and ID uniqueness.
|
||||
|
||||
### 05 - Create splits
|
||||
|
||||
`scripts/create_splits.py` creates deterministic stratified 80/10/10 partitions
|
||||
using seed `20260726` and verifies that every input ID appears exactly once.
|
||||
|
||||
### 06 - Build SFT
|
||||
|
||||
`scripts/build_sft.py` loads `config/system_prompt.txt`, builds the exact target
|
||||
JSON, and overwrites all three split files with chat conversations. Always run
|
||||
stage 05 immediately before rerunning stage 06.
|
||||
|
||||
### 07 - Base-model smoke test
|
||||
|
||||
`scripts/model_smoke_test.py` loads one training example, tokenizer/template, and
|
||||
the 4-bit base model; performs a forward pass; and rejects non-finite logits.
|
||||
It does not test the adapter or merged model.
|
||||
|
||||
### 08 - Pilot training
|
||||
|
||||
`scripts/train_pilot.py` selects the first up-to-64 training records and runs 10
|
||||
steps. It deletes/recreates `outputs/pilot-adapter`.
|
||||
|
||||
### 09 - Full training
|
||||
|
||||
`scripts/train.py` uses:
|
||||
|
||||
- NF4 4-bit base weights with double quantization and BF16 compute;
|
||||
- LoRA rank 16, alpha 32, dropout 0.05, target `all-linear`;
|
||||
- completion-only loss over the assistant response;
|
||||
- gradient checkpointing;
|
||||
- batch size 2 and accumulation 8;
|
||||
- 2 epochs, learning rate 0.0002, 100 warmup steps;
|
||||
- epoch evaluation and checkpointing;
|
||||
- lowest validation loss as the best saved checkpoint.
|
||||
|
||||
It deletes/recreates `outputs/final-adapter`. There is no early-stopping
|
||||
callback.
|
||||
|
||||
### 10 - Evaluate
|
||||
|
||||
`scripts/evaluate.py` loads the base model plus final adapter and generates all
|
||||
development outputs in deterministic batches of eight. Generation stops at EOS,
|
||||
end-of-turn, or a complete balanced JSON object. Results are atomically written
|
||||
to `outputs/development-evaluation`.
|
||||
|
||||
Metrics cover JSON/schema validity, classification confusion matrix and
|
||||
precision/recall/F1, result count, category, text, and evidence.
|
||||
|
||||
### Rescore without inference
|
||||
|
||||
`scripts/rescore_evaluation.py` reparses existing raw outputs, annotates the
|
||||
prediction records, and rewrites predictions/summary without rerunning a model.
|
||||
|
||||
### 11 - Verify generation stopping
|
||||
|
||||
`scripts/verify_generation_stop.py` generates the first five development
|
||||
examples and requires each complete output to parse with `json.loads`. Balanced
|
||||
braces trigger stopping; schema validation remains a later responsibility.
|
||||
|
||||
### 12 - Merge
|
||||
|
||||
`scripts/merge_model.py` loads unquantized BF16 base weights on CPU, merges the
|
||||
final adapter, saves safe-serialized 5 GB shards to a temporary directory,
|
||||
verifies key files, and atomically promotes the directory to
|
||||
`outputs/Invited-PHI-8B`.
|
||||
|
||||
This stage creates a standalone Hugging Face model, not GGUF.
|
||||
|
||||
## Chat contract
|
||||
|
||||
`config/chat_template.jinja` uses Llama-style role headers and end-of-turn
|
||||
tokens. `config/system_prompt.txt` requires exactly one JSON object:
|
||||
|
||||
```json
|
||||
{
|
||||
"is_medical_phi": true,
|
||||
"score": 1.0,
|
||||
"reason": "...",
|
||||
"results": [
|
||||
{
|
||||
"entity_type": "MEDICAL_DIAGNOSIS",
|
||||
"text": "...",
|
||||
"score": 1.0,
|
||||
"evidence": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
No Markdown, extra prose, inferred facts, or non-`MEDICAL_*` types are allowed.
|
||||
|
||||
## Evaluation caveats
|
||||
|
||||
- A valid JSON prefix followed by trailing text can count as parseable in the
|
||||
evaluator/rescorer because `raw_decode` is used.
|
||||
- Entity comparisons use the first expected and predicted result.
|
||||
- Entity accuracy denominators are all positive records.
|
||||
- Exact whole-input text/evidence reproduction is measured, not span overlap.
|
||||
- Result-count correctness is reported separately from schema validity.
|
||||
- Evaluation covers base-plus-adapter, not the merged standalone model.
|
||||
|
||||
## Path caveat
|
||||
|
||||
All `.sh` wrappers use `/d/ai/training/openbiollm-phi`, while this checkout may
|
||||
be elsewhere (for example `/d/ai/training/Models-Phi-8B`). Fix the `PROJECT_ROOT`
|
||||
line in every wrapper or invoke the underlying `uv run python` commands from the
|
||||
actual repository root.
|
||||
Loading…
Reference in New Issue