From 42ee4c260535a430a3ea1f743ea5293d59b00ecb Mon Sep 17 00:00:00 2001 From: GAJ3T Date: Thu, 15 Oct 2026 10:32:34 -0400 Subject: [PATCH] updated docs --- README.md | 55 +++- docs/README.md | 64 +++++ docs/api-reference.md | 90 +++++++ docs/architecture.md | 113 +++++++++ docs/artifacts.md | 84 ++++++ docs/claims-and-terminology.md | 105 ++++++++ docs/data-and-provenance.md | 145 +++++++++++ docs/diagrams/01-system-context.drawio | 52 ++++ docs/diagrams/02-containers.drawio | 52 ++++ docs/diagrams/03-generator-components.drawio | 62 +++++ docs/diagrams/04-generator-code.drawio | 54 ++++ docs/generator.md | 201 +++++++++++++++ docs/runbook.md | 254 +++++++++++++++++++ docs/testing-and-validation.md | 90 +++++++ docs/training-pipeline.md | 152 +++++++++++ 15 files changed, 1571 insertions(+), 2 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/api-reference.md create mode 100644 docs/architecture.md create mode 100644 docs/artifacts.md create mode 100644 docs/claims-and-terminology.md create mode 100644 docs/data-and-provenance.md create mode 100644 docs/diagrams/01-system-context.drawio create mode 100644 docs/diagrams/02-containers.drawio create mode 100644 docs/diagrams/03-generator-components.drawio create mode 100644 docs/diagrams/04-generator-code.drawio create mode 100644 docs/generator.md create mode 100644 docs/runbook.md create mode 100644 docs/testing-and-validation.md create mode 100644 docs/training-pipeline.md diff --git a/README.md b/README.md index 3b82c21..a9e3190 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,54 @@ -# Model for flagging and redacting PHI +# Medical PHI detection model -Simple, repeatable pipeline for exporting labeled "Generator" created SQL Server data, fine-tuning / extending Llama3-OpenBioLLM-8B, evaluating it, and producing a GGUF model. +This repository contains two connected systems: + +1. A .NET generator that combines curated medical terminology with positive and + negative language patterns, normalizes the results through an + OpenAI-compatible LM Studio endpoint, and persists labeled examples to SQL + Server. +2. A Python/WSL training pipeline that exports those examples, validates and + splits them, converts them to supervised fine-tuning (SFT) conversations, + trains and evaluates a QLoRA adapter for `Llama3-OpenBioLLM-8B`, and merges + the adapter into a standalone Hugging Face model. + +The repository also contains generated data, trained adapters, a merged model, +evaluation results, and GGUF artifacts. The tracked automation creates the +Hugging Face model; a GGUF conversion command is not currently tracked. + +## Documentation + +Start with [the documentation index](docs/README.md). + +- [Architecture](docs/architecture.md) +- [Generator](docs/generator.md) +- [Data and provenance](docs/data-and-provenance.md) +- [Training pipeline](docs/training-pipeline.md) +- [API reference](docs/api-reference.md) +- [Testing and validation](docs/testing-and-validation.md) +- [Step-by-step runbook](docs/runbook.md) +- [Terminology and verified claims](docs/claims-and-terminology.md) +- [Artifacts and repository map](docs/artifacts.md) +- [Editable C4 diagrams](docs/diagrams/) + +## Important semantics + +- `Category` is the label/taxonomy term used throughout this repository. The + names are compatible with the intended Presidio extension/override, but this + repository does not contain the Presidio integration itself. +- A pattern with `[sub]` receives a shuffled subject (`He`, `She`, `They`, or + `Member`). A pattern without `[sub]` is a subjectless fragment. The LM prompt + treats that fragment as having an implied singular member/patient for grammar + only; it must not insert that subject into the output. +- Positive and negative patterns teach contextual classification. Medical words + can therefore appear in both classes: a real medical assertion is positive, + while educational, fictional, historical, quoted, or unrelated use can be + negative. +- The medical terminology catalogs were assembled from NIH, WHO, and CDC data, + according to the project owner. See the provenance documentation for the + verification and licensing work still required. + +## Quick start + +The complete, ordered procedure is in [the runbook](docs/runbook.md). Do not run +the numbered shell wrappers without first correcting their legacy hard-coded +project path. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..419b6c3 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,64 @@ +# Documentation index + +## System overview + +This project creates a medical-context PHI classifier in two stages: + +```text +NIH/WHO/CDC-derived terminology + curated patterns + | + v + .NET generator + | + v + SQL Server TrainingItem + | + v + export -> validate -> split -> SFT + | + v + QLoRA train and evaluate + | + v + adapter -> merged Hugging Face model +``` + +The generator creates labeled natural-language examples. The Python pipeline +uses those examples to teach a model to return a strict JSON decision describing +whether the input contains medical PHI and, for positives, its category. + +## Documents + +| Document | Purpose | +|---|---| +| [Architecture](architecture.md) | System boundaries, data flow, persistence, and design decisions | +| [Generator](generator.md) | Catalogs, patterns, planning, randomization, LM normalization, durability | +| [Data and provenance](data-and-provenance.md) | Schemas, categories, terminology sources, lineage, and governance | +| [Training pipeline](training-pipeline.md) | Every Python script and scaffold step | +| [API reference](api-reference.md) | Generator HTTP endpoints, contracts, and failure behavior | +| [Testing and validation](testing-and-validation.md) | .NET tests, Python verification, evaluation semantics, current metrics | +| [Runbook](runbook.md) | Prerequisites and complete operator procedure | +| [Terminology and verified claims](claims-and-terminology.md) | Corrections and precise wording for the project explanation | +| [Artifacts and repository map](artifacts.md) | Source tree, inputs, outputs, and reproducibility boundaries | + +## C4 diagrams + +The editable draw.io documents are: + +- [Level 1: System context](diagrams/01-system-context.drawio) +- [Level 2: Containers](diagrams/02-containers.drawio) +- [Level 3: Components](diagrams/03-generator-components.drawio) +- [Level 4: Code](diagrams/04-generator-code.drawio) + +Open the files with diagrams.net, the draw.io desktop application, or the VS +Code Draw.io Integration extension. + +## Documentation conventions + +- **Verified** means behavior is directly represented by tracked code, tests, or + configuration. +- **Owner-provided provenance** means the fact was supplied by the project owner + but source URLs, versions, retrieval dates, and licenses are not tracked here. +- Paths in commands are examples. The checked-in shell wrappers still use the + legacy WSL path `/d/ai/training/openbiollm-phi`; update them before use if the + checkout is elsewhere. diff --git a/docs/api-reference.md b/docs/api-reference.md new file mode 100644 index 0000000..48b8c4f --- /dev/null +++ b/docs/api-reference.md @@ -0,0 +1,90 @@ +# Generator API reference + +## Hosting + +Run the ASP.NET Core .NET 10 project +`PHI.Data.Generator/RLHF.Generator.Api`. Production requires: + +- the `RLHFGenerator` SQL Server connection string; +- migrated database schema; +- copied generator `Data` files; +- LM Studio listening at the configured OpenAI-compatible endpoint. + +Swagger and Scalar are enabled only in the Development environment. + +## Endpoints + +### `GET /` + +Returns a small service status/identity response. Use it as a basic process +health check; it does not prove database or LM Studio readiness. + +### `POST /api/lm-studio/complete` + +Normalizes one supplied sentence or fragment without persistence. + +The request contract supports: + +- text/sentence; +- required fragment flag; +- optional protected subject; +- optional protected medical term; +- optional allowed medical forms; +- optional requested tense. + +Validation rejects empty input, missing fragment status, invalid tense, +inconsistent protected-term/forms metadata, and duplicate allowed forms. +Successful output is the normalized plain-text line. + +### `POST /api/keywords/load` + +Loads the category catalogs into `Keyword`/`KeywordVariant`. + +- additive; +- case-insensitive; +- idempotent; +- does not backfill existing training items. + +### `POST /api/training-items/backfill-keywords` + +Matches existing generated text to keywords. + +Query behavior includes a `dryRun` option and bounded batch processing. Only a +unique match is assigned. Ambiguous and unmatched records remain null. + +### `POST /api/generation/run` + +Starts a new compatible generation run or resumes an incomplete one. The HTTP +request waits for generation and all persistence commits to complete; it does +not merely enqueue background work. + +The response reports run identity/status and generation counts. Failures and +cancellation update durable run state before surfacing the error. + +### `POST /api/training-items/reset` + +Clears generated training items and generation checkpoints while preserving the +seeded category rows. Treat this as destructive and restrict it operationally. + +## Suggested invocation sequence + +```powershell +$base = "http://localhost:" + +Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load" +Invoke-RestMethod -Method Post -Uri "$base/api/generation/run" +Invoke-RestMethod -Method Post -Uri "$base/api/training-items/backfill-keywords?dryRun=true" +Invoke-RestMethod -Method Post -Uri "$base/api/training-items/backfill-keywords?dryRun=false" +``` + +Use the actual port shown by `dotnet run`/launch settings. + +## Operational controls + +- Put authentication/authorization and network controls in front of the API; + the tracked host does not define an application authorization policy. +- Restrict reset and generation endpoints to operators. +- Do not expose the configured SQL Server connection details or LM endpoint. +- Monitor generation progress, LM failures, request duration, committed counts, + and generation-run status. +- Back up the database before reset or migration. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..81cc508 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,113 @@ +# Architecture + +## Goals + +The system produces a domain-adapted model that classifies contextual medical +PHI and returns a strict JSON response. Its principal design goals are: + +- create reproducible positive and negative training examples; +- preserve a controlled 12-category medical taxonomy; +- use contextual negatives so keywords alone do not determine classification; +- durably generate large datasets with resumable SQL persistence; +- fine-tune an 8B model on commodity GPU hardware through QLoRA; +- evaluate JSON validity, classification, and category extraction separately; +- publish both an adapter and a standalone merged model. + +## System boundary + +Inside the repository are: + +- the .NET 10 generator library, API, EF Core migrations, catalogs, and tests; +- curated category values, grammar variants, subjects, and positive/negative + patterns; +- Python 3.12 data preparation, training, evaluation, and merge scripts; +- PowerShell/Bash orchestration scripts; +- generated JSONL data and model/evaluation artifacts. + +External dependencies are: + +- SQL Server (`RLHFGenerator`); +- LM Studio exposing an OpenAI-compatible chat completions endpoint; +- the `qwen/qwen3.5-9b` normalization model configured in the API; +- the local `Llama3-OpenBioLLM-8B` base model; +- WSL/Linux, CUDA 12.8-compatible PyTorch, and a BF16-capable NVIDIA GPU; +- NIH, WHO, and CDC as owner-reported origins of the terminology catalogs. + +Microsoft Presidio is an intended downstream integration, not a runtime +component of this repository. + +## End-to-end data flow + +1. The generator validates 12 category files, 24 polarity-specific pattern + files, and the exact four-subject catalog. +2. For each category and polarity, it deterministically shuffles values, + patterns, subjects, and tenses using the configured random seed. +3. It substitutes protected medical values, resolves some grammar locally, and + asks LM Studio to normalize remaining expressions without changing meaning or + polarity. +4. It validates one-line output and writes ordered `TrainingItem` rows to SQL + Server in atomic batches, updating a durable generation checkpoint in the + same transaction. +5. PowerShell exports `TrainingItem` plus `Category` to `data/source.jsonl`. +6. Python validates the schema and taxonomy, then deterministically stratifies + by `(category, is_positive)` into train, validation, and development sets. +7. The SFT builder replaces each split record with a three-message conversation + containing the system policy, input text, and exact target JSON. +8. Pilot and full training load the base model in 4-bit NF4 and train LoRA + adapters using completion-only loss. +9. Evaluation loads the base model plus final adapter and generates development + predictions deterministically. +10. The merge step loads the unquantized base model on CPU, merges the adapter, + and saves a BF16 Hugging Face model. + +## Persistence model + +| Entity | Responsibility | +|---|---| +| `Category` | Stable seeded category ID/name | +| `Keyword` | Canonical category-specific medical value | +| `KeywordVariant` | Alternate grammatical/surface form | +| `GenerationRun` | Seed, fingerprints, status, checkpoint, counts, errors | +| `TrainingItem` | Generated text, label, category, optional keyword, plan ordinal | + +Important constraints include unique category names, unique canonical values +within a category, unique variants within a keyword, unique training record +keys, and unique `(GenerationRunId, PlanOrdinal)`. + +## Reliability model + +- **Deterministic plan:** `System.Random` plus seed 42 creates a repeatable plan + for unchanged inputs and configuration. +- **External-model caveat:** repeatable planning does not guarantee byte-identical + LM Studio responses if model files, runtime, or service behavior change. +- **Fingerprinting:** generation hashes source files and effective settings; + changed inputs/settings reject an unsafe resume. +- **Ordered parallelism:** generation is concurrent, but a single ordered writer + buffers out-of-order results. +- **Atomic checkpoints:** inserted rows and the checkpoint advance commit in the + same database transaction. +- **Fail closed:** invalid LM output, request failures, persistence failures, + cancellation, mismatched durable state, or ambiguous incomplete runs fail the + operation rather than silently producing success-shaped output. + +## Classification model + +The model is trained as a binary classifier plus category reporter: + +- A positive SFT target has `is_medical_phi: true`, score `1.0`, a single result, + and an entity type equal to one of the 12 categories. +- A negative SFT target has `is_medical_phi: false`, score `0.0`, and no results. +- Positive targets currently mark the entire input as `text` and `evidence`. + This is classification with whole-input evidence, not span extraction. + +## Known architecture gaps + +- No tracked Presidio recognizer, registry, mapping, or deployment integration. +- No tracked GGUF conversion or quantization workflow, although GGUF files are + present. +- Shell scaffolds have a legacy hard-coded checkout path. +- The SFT conversion overwrites split files and is not idempotent. +- The generator depends on an external normalization model whose exact artifact + version is not fingerprinted. +- Terminology source URLs, versions, retrieval dates, transformations, and + licenses are not captured in a machine-readable manifest. diff --git a/docs/artifacts.md b/docs/artifacts.md new file mode 100644 index 0000000..298bb78 --- /dev/null +++ b/docs/artifacts.md @@ -0,0 +1,84 @@ +# Artifacts and repository map + +## Source tree + +```text +. +|-- PHI.Data.Generator/ +| |-- RLHF.Generator/ Core .NET generator and Data catalogs +| |-- RLHF.Generator.Api/ ASP.NET Core API +| |-- RLHF.Generator.Tests/ xUnit tests +| |-- AddKeywordCatalog.sql +| `-- RLHF.Generator.slnx +|-- config/ Training, prompt, and chat-template config +|-- data/ Exported and transformed JSONL +|-- scaffolding/ Ordered operational wrappers +|-- scripts/ Python pipeline implementation +|-- outputs/ Adapters, evaluation, merged model, GGUF files +|-- docs/ Project documentation and draw.io diagrams +|-- pyproject.toml +`-- uv.lock +``` + +## Configuration + +| Path | Purpose | +|---|---| +| `PHI.Data.Generator/RLHF.Generator.Api/appsettings.json` | SQL, LM Studio, generation defaults | +| `config/pipeline.json` | SQL export metadata, paths, splits, base model, LoRA/training/merge | +| `config/system_prompt.txt` | Inference/SFT output contract | +| `config/chat_template.jinja` | Llama-style chat serialization | +| `pyproject.toml` / `uv.lock` | Python requirements and resolved environment | + +## Data artifacts + +| Path | State | +|---|---| +| `data/source.jsonl` | Source-shaped SQL export | +| `data/train.jsonl` | Source-shaped after split; SFT-shaped after build | +| `data/validation.jsonl` | Source-shaped after split; SFT-shaped after build | +| `data/development.jsonl` | Source-shaped after split; SFT-shaped after build | + +Because the same split paths change schema in place, record stage/checksums when +archiving or automating the pipeline. + +## Model artifacts + +| Path | Meaning | +|---|---| +| `outputs/pilot-adapter` | Small pilot LoRA adapter | +| `outputs/final-adapter` | Full LoRA adapter and retained checkpoints | +| `outputs/development-evaluation` | Predictions and summary metrics | +| `outputs/Invited-PHI-8B` | Standalone merged BF16 Hugging Face model | +| `outputs/gguf` | BF16/Q4_K_M/Q5_K_M GGUF files | + +The GGUF artifacts have no corresponding tracked conversion script, flags, +tool revision, or verification step. They cannot be reproduced from repository +automation alone. + +## Reproducibility boundary + +Tracked: + +- source code, catalogs, patterns, seeds, prompts, configurations; +- Python lock file; +- generator tests; +- current data/model/evaluation artifacts. + +Not fully tracked: + +- exact NIH/WHO/CDC source datasets and transformation history; +- LM Studio model checksum/runtime build; +- base model checksum/revision; +- SQL Server contents before export; +- host drivers/CUDA/toolchain snapshot; +- GGUF conversion/quantization commands; +- downstream Presidio implementation. + +## Large files and source control + +The repository tracks multi-gigabyte model/training artifacts. For sustainable +operation, consider an artifact registry or Git LFS with immutable versioning, +checksums, retention, access control, and release metadata. Keep source, +configuration, metrics, and artifact manifests in Git even if weights move to +external storage. diff --git a/docs/claims-and-terminology.md b/docs/claims-and-terminology.md new file mode 100644 index 0000000..44d6495 --- /dev/null +++ b/docs/claims-and-terminology.md @@ -0,0 +1,105 @@ +# Terminology and verified claims + +## Recommended project explanation + +> The generator combines medical category values with curated positive and +> negative contextual patterns. It deterministically shuffles category values, +> patterns, explicit subjects, and requested tenses, then uses LM Studio to +> normalize grammar while preserving the protected medical concept and the +> example's polarity. Patterns without an explicit subject remain subjectless +> fragments; the normalizer assumes an implied singular member/patient only for +> grammar and must not add that subject to the text. +> +> The project uses **Category** as its taxonomy term because the labels are +> intended to integrate with a Presidio category/entity-type extension or +> override. This repository contains the category-based generator and model +> training pipeline, but the Presidio integration itself is maintained +> elsewhere or has not yet been added here. +> +> Positive and negative contexts teach the model that a medical word alone is +> insufficient. “Has diabetes” and “he had a reaction to shellfish” are intended +> positives, while “she wrote a book about diabetes” and “had a reaction to the +> horrible cover of Van Halen's Jump” are intended negatives. + +## Review of the original statements + +### “The Generator uses random subject, verb, etc.” + +**Mostly correct, with precision needed.** + +- It shuffles values, patterns, subjects, and a tense deck with a seeded + pseudorandom generator. +- Subjects are `He`, `She`, `They`, and `Member`. +- It does not simply select arbitrary verbs. Patterns supply grammar expressions + and sentence structure; local logic and LM Studio resolve/rewrite auxiliaries + and verb phrases for subject/tense. +- `[x]` starts with the canonical value. Allowed variants are supplied to the + normalizer, rather than directly sampled as an independent random deck. +- Planning is repeatable for unchanged seed/input/configuration; the external LM + service is a remaining reproducibility variable. + +### “It creates context for positive and negative tuning examples.” + +**Correct.** + +Every category has explicit positive and negative pattern files. The label is +stored on each generated item and carried through export, SFT conversion, +training, and evaluation. + +### “Categories instead of Nodes/Edges because of Presidio.” + +**Reasonable design intent, not verifiable integration in this repository.** + +The implementation consistently uses categories/entity types, and that is an +appropriate vocabulary for a Presidio recognizer extension. There is no graph +model in this code, so “nodes/edges” would be misleading. However, no Presidio +package, recognizer, registry, mapping, or runtime call is checked in. Phrase +this as intended compatibility/integration, not as a current in-repository +dependency. + +### “Without a subject, or if a sentence fragment, it assumes the member is the subject.” + +**Correct only when qualified.** + +The implementation marks patterns without `[sub]` as fragments and does not +insert `Member`. The normalizer uses an implied singular member/patient for +agreement and tense only, and must preserve a subjectless output. `Member` is +also one explicit choice in the subject deck when `[sub]` is present. + +Preferred wording: + +> Subjectless patterns are treated as fragments. For grammar only, the +> normalizer assumes an implied singular member/patient but does not add that +> subject to the generated text. + +### Context examples + +**They correctly express the intended distinction.** + +The medical term or word “reaction” is not enough by itself. Actual patient +medical assertions are positive; authored/educational or idiomatic non-medical +uses are negative. These exact examples should also become automated golden +tests if they are contractual behavior. + +## Category versus entity type + +- **Category** is the generator/database taxonomy object. +- **Entity type** is the name emitted in model JSON. +- Both currently use the same 12 `MEDICAL_*` identifiers. +- **Keyword** is a canonical medical value assigned to a category. +- **Variant** is an allowed surface/grammatical form of a keyword. +- **Pattern** supplies contextual structure and polarity. +- **Training item** is the resulting labeled text. + +## What not to claim + +Avoid saying: + +- the model is a keyword matcher; +- all category variants are randomly selected; +- `Member` is inserted whenever a subject is absent; +- this repository contains a working Presidio integration; +- the tracked pipeline produces GGUF; +- the current model extracts exact medical spans; +- deterministic planning guarantees identical external LM responses; +- NIH, WHO, or CDC endorse this project. diff --git a/docs/data-and-provenance.md b/docs/data-and-provenance.md new file mode 100644 index 0000000..541520f --- /dev/null +++ b/docs/data-and-provenance.md @@ -0,0 +1,145 @@ +# Data and provenance + +## Provenance statement + +The project owner reports that the medical-term, condition, medication, and +related terminology datasets used to assemble the category catalogs came from +the **U.S. National Institutes of Health (NIH)**, the **World Health +Organization (WHO)**, and the **U.S. Centers for Disease Control and Prevention +(CDC)**. + +The repository does not currently include a source manifest proving which +catalog rows came from which organization. It also does not record source URLs, +dataset/product names, versions, retrieval dates, licenses, notices, or +transformation history. Accordingly, NIH/WHO/CDC origin is documented here as +owner-provided provenance, not independently verifiable row-level provenance. + +Before redistribution or production use, create a provenance manifest and +review the terms for every underlying source. Public availability does not +automatically mean unrestricted redistribution, and naming an organization does +not imply its endorsement. + +## Recommended source manifest + +Track at least: + +| Field | Example purpose | +|---|---| +| `source_organization` | NIH, WHO, or CDC | +| `dataset_name` | Exact source product | +| `source_url` | Stable landing/download URL | +| `version` | Release/version/date | +| `retrieved_at` | UTC retrieval timestamp | +| `license_or_terms_url` | Governing terms | +| `source_checksum` | Integrity and reproducibility | +| `category_files` | Derived files | +| `transformation` | Filtering, normalization, deduplication | +| `attribution_required` | Required notice | +| `reviewed_by` / `reviewed_at` | Governance approval | + +## Generator source data + +### Categories + +Twelve text files define the fixed medical taxonomy. Each active row is a +pipe-separated canonical term followed by grammatical or surface variants. +These catalogs are the terminology/keyword source for generation and optional +keyword persistence. + +### Patterns + +Each category has a positive and negative pattern file. Patterns are project +authored/curated contextual templates; they determine whether generated text is +labeled positive or negative. Catalog membership alone does not set polarity. + +### Subjects + +`Subjects.txt` is required to contain exactly: + +```text +He +She +They +Member +``` + +## SQL schema used by training export + +The export reads `dbo.TrainingItem` joined to `dbo.Category` and writes: + +```json +{ + "id": 123, + "category": "MEDICAL_TREATMENT", + "is_positive": true, + "value": "He is receiving physical therapy." +} +``` + +Validation requires exactly those fields, a unique positive integer ID, one of +the 12 category names, a real JSON Boolean, and a non-empty string value. + +## Split data + +`create_splits.py` groups by `(category, is_positive)`, shuffles each group with +a stable derived seed, allocates rounded 10% validation and 10% development +counts, and sends the remainder to training. It then independently shuffles each +whole split and verifies uniqueness, disjointness, and complete assignment. + +Small strata can have materially different proportions or no training row after +rounding. The script guarantees partition integrity, not minimum representation +in every split. + +## SFT data + +`build_sft.py` overwrites each split with: + +```json +{ + "id": 123, + "messages": [ + {"role": "system", "content": "...strict JSON policy..."}, + {"role": "user", "content": "He is receiving physical therapy."}, + {"role": "assistant", "content": "{\"is_medical_phi\":true,...}"} + ] +} +``` + +For positives, the expected result contains the source category and uses the +entire input for both `text` and `evidence`. For negatives, `results` is empty. +This means the current task is whole-input contextual classification rather than +precise entity-span extraction. + +The conversion is destructive and non-idempotent. Recreate source-shaped splits +before rerunning it. + +## Data lineage + +```text +Owner-curated NIH/WHO/CDC-derived catalogs + + project-authored polarity patterns + + subject and tense decks + | + v + normalized generated TrainingItem rows + | + v + data/source.jsonl + | + v + source-shaped train/validation/development JSONL + | + v + SFT-shaped train/validation/development JSONL + | + v + adapter, predictions, metrics, merged model +``` + +## Privacy and governance + +The checked-in source is synthetic/generated according to the system design, +but that assumption should be verified operationally. Do not import real patient +records without a separate privacy, consent, security, retention, and access +control review. The model is designed to detect sensitive medical context; it is +not itself a compliance determination or a substitute for human review. diff --git a/docs/diagrams/01-system-context.drawio b/docs/diagrams/01-system-context.drawio new file mode 100644 index 0000000..469a636 --- /dev/null +++ b/docs/diagrams/01-system-context.drawio @@ -0,0 +1,52 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/diagrams/02-containers.drawio b/docs/diagrams/02-containers.drawio new file mode 100644 index 0000000..8712bd4 --- /dev/null +++ b/docs/diagrams/02-containers.drawio @@ -0,0 +1,52 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/diagrams/03-generator-components.drawio b/docs/diagrams/03-generator-components.drawio new file mode 100644 index 0000000..fbde6f2 --- /dev/null +++ b/docs/diagrams/03-generator-components.drawio @@ -0,0 +1,62 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/diagrams/04-generator-code.drawio b/docs/diagrams/04-generator-code.drawio new file mode 100644 index 0000000..dcd8cd1 --- /dev/null +++ b/docs/diagrams/04-generator-code.drawio @@ -0,0 +1,54 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/generator.md b/docs/generator.md new file mode 100644 index 0000000..00c831c --- /dev/null +++ b/docs/generator.md @@ -0,0 +1,201 @@ +# Generator + +## Purpose + +`PHI.Data.Generator` creates labeled contextual medical-language examples. It +does not generate a graph of nodes and edges. Its central domain concepts are +categories, category values/keywords, patterns, subjects, and training items. + +## Projects + +| Project | Role | +|---|---| +| `RLHF.Generator` | Catalog parsing, planning, LM client, persistence, generation, keyword services | +| `RLHF.Generator.Api` | ASP.NET Core host and HTTP endpoints | +| `RLHF.Generator.Tests` | xUnit unit/integration tests using SQLite for test persistence | + +All three target .NET 10. Production persistence uses EF Core SQL Server. + +## Taxonomy + +The fixed categories are: + +1. `MEDICAL_ALLERGY` +2. `MEDICAL_DIAGNOSIS` +3. `MEDICAL_FAMILY_HISTORY` +4. `MEDICAL_MEDICATION` +5. `MEDICAL_MENTAL_HEALTH` +6. `MEDICAL_OTHER` +7. `MEDICAL_PHYSICAL_HEALTH` +8. `MEDICAL_PROCEDURE` +9. `MEDICAL_SUBSTANCE_USE` +10. `MEDICAL_SYMPTOM` +11. `MEDICAL_TEST` +12. `MEDICAL_TREATMENT` + +Lookup is case-insensitive; stored/returned names use these exact values. + +## Input layout and grammar + +```text +RLHF.Generator/Data/ + Subjects.txt + Categories/ + MEDICAL_ALLERGY.txt + ... + Patterns/ + MEDICAL_ALLERGY.positive.txt + MEDICAL_ALLERGY.negative.txt + ... +``` + +Blank lines and lines beginning with `#` are ignored. A category line is: + +```text +canonical form|allowed variant|another allowed variant +``` + +Forms must be non-empty and case-insensitively unique within a category. The +first is canonical. The remainder are alternative forms available to the +normalization model. + +The exact subjects are `He`, `She`, `They`, and `Member`. + +Pattern tokens include: + +| Token | Meaning | +|---|---| +| `[x]` | Canonical category value; allowed variants are supplied to LM Studio | +| `[sub]` | A subject selected from the shuffled subject deck | +| `[is\|are\|was\|were]` | Copula selected/rephrased for subject and tense | +| `[has\|have\|had]` | Possession/auxiliary selected for subject and tense | +| `[a\|an]` | Article resolved locally from the following word | + +The legacy `[s]` placeholder is rejected. + +## Positive and negative examples + +Every category has an independent positive and negative pattern set. A positive +pattern describes real medical context; a negative pattern uses the same domain +language without asserting medical information about the person being analyzed. + +Representative intended behavior: + +| Input | Expected label | Why | +|---|---:|---| +| `Has diabetes` | Positive | Subjectless clinical fragment with an implied member/patient | +| `He had a reaction to shellfish` | Positive | Actual allergy/reaction assertion | +| `She wrote a book about diabetes` | Negative | Creative/educational context, not her diagnosis | +| `Had a reaction to the horrible cover of Van Halen's Jump` | Negative | Idiomatic non-medical reaction | + +Those examples describe the intended learned behavior; they are not hard-coded +rules or keyword matchers in the inference pipeline. + +## Plan construction + +For each category, the service runs the positive pass and then the negative +pass. It partitions patterns into: + +- **value-bound:** contains `[x]`; +- **generic:** does not contain `[x]`. + +It rejects values without any value-bound pattern and value-bound patterns when +the category has no values. It then: + +1. shuffles values, value-bound patterns, and generic patterns; +2. creates shuffled/cycling subject and tense decks; +3. emits every generic pattern once; +4. emits `max(value-bound pattern count, value count)` value-bound plans; +5. reshuffles a deck whenever it is exhausted. + +The tense deck is `null`, `past`, `present`, and `future`. The canonical category +value is placed in `[x]`; variants are not independently randomly sampled. + +Because each polarity is planned from its own pattern/value counts, positive and +negative totals are not guaranteed to be equal. + +## Subject and fragment behavior + +The precise behavior is: + +- A pattern containing `[sub]` gets one shuffled explicit subject. +- A pattern without `[sub]` is marked as a fragment and gets no inserted + subject. +- `Member` is one of the four explicit subject choices; it is not a fallback + value. +- For a fragment, the LM system prompt uses an implied singular member/patient + only to choose grammar. It explicitly forbids returning or inventing that + subject. +- Some fragments are completed locally without an LM call when no unresolved + choice, protected-form decision, or meaningful tense rewrite remains. + +Therefore the safe summary is: **subjectless patterns remain subjectless, while +grammar assumes an implied singular member/patient.** + +## LM Studio normalization + +Default settings: + +| Setting | Value | +|---|---| +| Endpoint | `http://localhost:1234/v1/chat/completions` | +| Model | `qwen/qwen3.5-9b` | +| Temperature | `0.0` | +| Top-p | `1.0` | +| Max tokens | `96` | +| Seed | `42` | +| Timeout | `300` seconds | + +Each request carries fragment status, optional protected subject, protected +medical term and allowed forms, requested tense, and the sentence/template. The +system prompt requires exactly one unwrapped line, preserves meaning and +polarity, resolves all bracket expressions, and forbids invented fragment +predicates. + +Generation rejects empty output, newlines, unresolved square brackets, +quotation/JSON/code-fence wrappers, and common response labels. + +## Concurrency and durability + +Defaults are seed 42, parallelism 8, save batches of 24, and progress logs every +5 seconds. + +Workers generate concurrently into a bounded channel. A single persistence +writer places results into a sorted buffer and commits only the next contiguous +plan ordinals. Each transaction inserts the batch and advances the generation +checkpoint together. + +A run can resume only when its seed, input fingerprint, configuration +fingerprint, expected count, committed row count, and last ordinal agree. +Changed inputs/configuration or inconsistent persisted state are rejected. + +## Keyword catalog + +Keyword loading uses each category file's canonical value and variants. Loading +is additive, case-insensitive, and idempotent. It does not update existing +training items. Backfill is a separate operation: + +- dry-run is available; +- only unique matches are assigned; +- unmatched or ambiguous rows remain without a keyword. + +## Database migrations + +The migrations progress through: + +1. initial `Category` and `TrainingItem` schema; +2. durable generation-run/checkpoint support; +3. `Keyword`, `KeywordVariant`, and training-item keyword linkage. + +`AddKeywordCatalog.sql` provides a SQL alternative for the keyword phase. + +## Configuration changes that invalidate resume + +Treat these as generation identity: + +- category, pattern, or subject file contents/paths; +- random seed; +- concurrency and save/progress options included in the fingerprint; +- LM endpoint/model/prompt and generation settings. + +Start a clean run rather than bypassing fingerprint checks. diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 0000000..9d63ae4 --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,254 @@ +# End-to-end runbook + +This procedure starts with catalogs and ends with a merged model. Run commands +from the repository root unless stated otherwise. + +## 1. Record the run + +Capture: + +- Git revision and clean/dirty state; +- operator and UTC start time; +- catalog/pattern checksums and provenance manifest; +- SQL Server target; +- LM Studio model artifact/version and settings; +- base model path/revision/checksum; +- GPU, driver, CUDA, Python, and dependency versions; +- expected output locations and acceptance thresholds. + +## 2. Prerequisite checks + +Install/prepare: + +- .NET 10 SDK; +- SQL Server reachable with schema-creation privileges; +- LM Studio with `qwen/qwen3.5-9b` (or an intentionally approved replacement); +- Windows PowerShell for SQL export; +- WSL/Linux, Git, build tools, and `uv`; +- Python 3.12; +- NVIDIA CUDA GPU with BF16 and sufficient VRAM; +- local `Llama3-OpenBioLLM-8B` base model; +- enough CPU RAM/disk for an unquantized 8B BF16 merge. + +Correct the legacy `/d/ai/training/openbiollm-phi` path in every shell scaffold, +or use the direct commands below from the real checkout. + +## 3. Validate generator inputs + +Confirm: + +- exactly 12 category files; +- one positive and one negative pattern file per category; +- subjects exactly `He`, `She`, `They`, `Member`; +- no legacy `[s]` tokens; +- canonical/variant forms are unique; +- each category has values and compatible `[x]` patterns. + +The generator also enforces these checks when a run starts. + +## 4. Configure and migrate SQL Server + +Review the `RLHFGenerator` connection string in +`PHI.Data.Generator/RLHF.Generator.Api/appsettings.json`. Prefer environment +configuration or user secrets for deployment-specific values. + +From `PHI.Data.Generator`: + +```powershell +dotnet restore .\RLHF.Generator.slnx +dotnet ef database update ` + --project .\RLHF.Generator\RLHF.Generator.csproj ` + --startup-project .\RLHF.Generator.Api\RLHF.Generator.Api.csproj +``` + +Back up an existing database before migration or reset. + +## 5. Test the generator + +```powershell +dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx +``` + +Stop if any test fails. + +## 6. Start LM Studio + +1. Load the configured normalization model. +2. Start the local OpenAI-compatible server. +3. Confirm `http://localhost:1234/v1/chat/completions` is reachable. +4. Keep temperature, seed, token limit, and model name aligned with generator + configuration. + +## 7. Start the generator API + +```powershell +dotnet run --project ` + .\PHI.Data.Generator\RLHF.Generator.Api\RLHF.Generator.Api.csproj +``` + +Note the actual listening URL. In Development, inspect Swagger/Scalar. Call +`GET /` as a basic process check. + +## 8. Load keywords and generate + +In another PowerShell window: + +```powershell +$base = "http://localhost:" + +Invoke-RestMethod -Method Post -Uri "$base/api/keywords/load" +Invoke-RestMethod -Method Post -Uri "$base/api/generation/run" +``` + +The generation request remains open until completion. Monitor API logs and SQL +`GenerationRun`. On cancellation/failure, correct the cause and call the same +endpoint to resume only if the service accepts the matching fingerprints. + +Do not edit catalogs, patterns, generation settings, or LM settings mid-run. + +Optional keyword linkage: + +```powershell +Invoke-RestMethod -Method Post ` + -Uri "$base/api/training-items/backfill-keywords?dryRun=true" +Invoke-RestMethod -Method Post ` + -Uri "$base/api/training-items/backfill-keywords?dryRun=false" +``` + +## 9. Export SQL data + +Review SQL server/database values in `config/pipeline.json`, then run: + +```powershell +.\scaffolding\03-export-sql.ps1 +``` + +Confirm `data/source.jsonl` exists and record positive/negative totals. + +## 10. Prepare the Python environment in WSL + +From the actual WSL path of the repository: + +```bash +uv python install 3.12.13 +uv venv --python 3.12.13 +uv lock +uv sync --frozen +uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.is_bf16_supported())" +``` + +Do not blindly run scaffold steps 01/02 in a shared worktree: they recreate the +environment and contain `git add`/`git commit` commands. + +## 11. Review pipeline configuration + +Update `config/pipeline.json` deliberately: + +- SQL connection metadata; +- actual WSL base-model path; +- split proportions/seed; +- sequence length; +- LoRA and training hyperparameters; +- output directories. + +Archive the exact configuration used. + +## 12. Validate and split + +```bash +uv run python scripts/validate_source.py +uv run python scripts/create_splits.py +``` + +Review counts per category/polarity/split. Pay special attention to small strata. + +## 13. Build SFT conversations + +```bash +uv run python scripts/build_sft.py +``` + +This overwrites all split files. If repeated, run `create_splits.py` first. +Inspect a positive and negative example from every category. + +## 14. Smoke-test the base model + +```bash +uv run python scripts/model_smoke_test.py +``` + +Confirm successful loading, finite logits, and expected GPU memory use. + +## 15. Run the pilot + +```bash +uv run python scripts/train_pilot.py +``` + +Review loss, warnings, throughput, and peak memory. Resolve OOM/non-finite loss +before full training. + +## 16. Train the full adapter + +```bash +uv run python scripts/train.py +``` + +This replaces `outputs/final-adapter`. Preserve any prior accepted adapter first. +Monitor training/evaluation loss, checkpoints, utilization, and disk. + +## 17. Evaluate and rescore + +```bash +uv run python scripts/evaluate.py +uv run python scripts/rescore_evaluation.py +``` + +Compare `outputs/development-evaluation/summary.json` with acceptance thresholds. +Review every error, particularly false negatives, false positives, category +errors, malformed JSON, and contextual-negative failures. + +## 18. Verify stopping + +```bash +uv run python scripts/verify_generation_stop.py +``` + +All five sampled outputs must be exact parseable JSON. + +## 19. Merge the adapter + +```bash +uv run python scripts/merge_model.py +``` + +This replaces `outputs/Invited-PHI-8B` atomically after verification. It can +require substantial CPU RAM and disk. + +## 20. Validate and publish artifacts + +Before release: + +1. Add/run a merged-model inference smoke test. +2. Confirm tokenizer, chat template, system prompt, configs, and all weight + shards are present. +3. Record checksums and total sizes. +4. Preserve adapter, merged model, metrics, predictions, config, source/split + checksums, Git revision, environment lock, and provenance manifest. +5. If GGUF is required, use an approved conversion/quantization process and + record it; no such process is tracked here. +6. Run downstream Presidio integration and end-user acceptance tests. + +## Recovery + +| Failure | Action | +|---|---| +| LM Studio unavailable/timeout | Restore service/model, then resume unchanged run | +| Generator invalid output | Inspect request/model/prompt; do not bypass validation | +| Fingerprint mismatch | Restore original inputs/settings or intentionally reset/start clean | +| SQL batch failure | Correct database cause; checkpointed committed batches remain | +| Split/SFT rerun failure | Recreate splits from `source.jsonl`, then rebuild SFT | +| CUDA OOM | Reduce approved batch/sequence settings, rerun pilot, document change | +| Training output exists | Archive accepted output; scripts intentionally replace output dirs | +| Evaluation failure | Preserve raw logs/artifacts, fix root cause, rerun evaluation | +| Merge resource failure | Increase CPU RAM/disk or use an approved merge host | diff --git a/docs/testing-and-validation.md b/docs/testing-and-validation.md new file mode 100644 index 0000000..575716a --- /dev/null +++ b/docs/testing-and-validation.md @@ -0,0 +1,90 @@ +# Testing and validation + +## .NET automated tests + +Run: + +```powershell +dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx +``` + +The xUnit suite covers: + +| Test area | Coverage | +|---|---| +| LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming | +| Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output | +| Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs | +| Endpoint behavior | Generation endpoint waits for actual generation and commit | +| Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs | +| Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity | +| Provider isolation | SQL Server production DI and SQLite-only test setup | + +Tests use SQLite for integration persistence while explicitly verifying that the +production service registers SQL Server. + +## Python verification stages + +There is no Python unit-test suite. Verification is operational: + +1. `validate_source.py` validates exported schema and taxonomy. +2. `create_splits.py` checks uniqueness, disjointness, and complete assignment. +3. `build_sft.py` checks message shape and parses generated assistant JSON. +4. `model_smoke_test.py` loads the base model and rejects non-finite logits. +5. `train_pilot.py` verifies a small QLoRA training path. +6. `evaluate.py` validates generated JSON/schema and computes metrics. +7. `verify_generation_stop.py` checks exact JSON parsing on five examples. +8. `merge_model.py` verifies merged weights and configuration files. + +## Current tracked evaluation snapshot + +`outputs/development-evaluation/summary.json` currently reports: + +| Metric | Value | +|---|---:| +| Development records | 2,027 | +| Valid JSON | 100% | +| Valid schema | 100% | +| Classification accuracy | 99.852% | +| True positives / false negatives | 996 / 0 | +| True negatives / false positives | 1,028 / 3 | +| Precision | 99.700% | +| Recall | 100% | +| F1 | 99.850% | +| Entity category accuracy | 98.394% | +| Entity text accuracy | 100% | +| Evidence accuracy | 100% | + +These values describe the checked-in development artifact, not an independent +benchmark or prospective production guarantee. + +## Recommended acceptance gates + +Before publishing a new model: + +- all .NET tests pass; +- source validation and split integrity pass; +- the pilot finishes with finite loss; +- full training selects/saves a valid adapter; +- JSON and schema validity remain 100%; +- classification/category metrics meet a pre-agreed threshold; +- false positives and false negatives are manually reviewed; +- generation-stop verification passes; +- merged-model loading/inference is added and passes; +- provenance and license manifest is complete; +- a Presidio integration test is run in the downstream repository; +- checksums, configuration, Git revision, base model revision, and environment + versions are archived. + +## Test gaps + +- No direct Python unit tests for schema, splitting edge cases, SFT conversion, + JSON stopping, scoring, or merge logic. +- No inference smoke test for the merged model. +- No tracked GGUF conversion/quantization verification. +- No end-to-end test from generator API through SQL export to training data. +- No checked-in Presidio integration or category mapping test. +- No fixed golden tests for the four representative contextual sentences. + +Adding those contextual examples as regression tests would make the intended +positive/negative distinction executable rather than documentary. diff --git a/docs/training-pipeline.md b/docs/training-pipeline.md new file mode 100644 index 0000000..4347f9e --- /dev/null +++ b/docs/training-pipeline.md @@ -0,0 +1,152 @@ +# Training pipeline + +## Runtime + +- Python `3.12.*` (the setup script installs 3.12.13) +- `uv` for interpreter/dependency management +- PyTorch 2.11.0 from the CUDA 12.8 index +- Transformers 5.14.1, TRL 1.9.1, PEFT 0.19.1 +- bitsandbytes 0.49.2 for 4-bit NF4 loading +- CUDA-capable NVIDIA GPU with BF16 support +- WSL/Linux for shell wrappers +- substantial CPU RAM/disk for BF16 merge + +The canonical configuration is `config/pipeline.json`. + +## Pipeline stages + +### 00 - Audit + +`scaffolding/00-audit.sh` checks Linux/WSL, Python, Git, compiler/build tools, +download tools, NVIDIA utilities, memory, disk, base model, and repository state. + +### 01 - Create environment + +`scaffolding/01-create-venv.sh` installs `uv`, installs Python 3.12.13, and +recreates `.venv`. It also attempts to stage and commit files, so inspect/remove +those Git commands before using it in an automated or shared worktree. + +### 02 - Install dependencies + +`scaffolding/02-install-dependencies.sh` locks/synchronizes dependencies and +fails if CUDA is not available. It prints GPU, BF16, CUDA, and package/runtime +information. It also contains Git commit commands. + +### 03 - Export SQL + +`scaffolding/03-export-sql.ps1` uses Windows integrated authentication, queries +SQL Server, and writes ordered UTF-8 JSONL. It reports total/positive/negative +counts. + +### 04 - Validate source + +`scripts/validate_source.py` enforces the four-field schema, IDs, taxonomy, +Boolean labels, non-empty values, and ID uniqueness. + +### 05 - Create splits + +`scripts/create_splits.py` creates deterministic stratified 80/10/10 partitions +using seed `20260726` and verifies that every input ID appears exactly once. + +### 06 - Build SFT + +`scripts/build_sft.py` loads `config/system_prompt.txt`, builds the exact target +JSON, and overwrites all three split files with chat conversations. Always run +stage 05 immediately before rerunning stage 06. + +### 07 - Base-model smoke test + +`scripts/model_smoke_test.py` loads one training example, tokenizer/template, and +the 4-bit base model; performs a forward pass; and rejects non-finite logits. +It does not test the adapter or merged model. + +### 08 - Pilot training + +`scripts/train_pilot.py` selects the first up-to-64 training records and runs 10 +steps. It deletes/recreates `outputs/pilot-adapter`. + +### 09 - Full training + +`scripts/train.py` uses: + +- NF4 4-bit base weights with double quantization and BF16 compute; +- LoRA rank 16, alpha 32, dropout 0.05, target `all-linear`; +- completion-only loss over the assistant response; +- gradient checkpointing; +- batch size 2 and accumulation 8; +- 2 epochs, learning rate 0.0002, 100 warmup steps; +- epoch evaluation and checkpointing; +- lowest validation loss as the best saved checkpoint. + +It deletes/recreates `outputs/final-adapter`. There is no early-stopping +callback. + +### 10 - Evaluate + +`scripts/evaluate.py` loads the base model plus final adapter and generates all +development outputs in deterministic batches of eight. Generation stops at EOS, +end-of-turn, or a complete balanced JSON object. Results are atomically written +to `outputs/development-evaluation`. + +Metrics cover JSON/schema validity, classification confusion matrix and +precision/recall/F1, result count, category, text, and evidence. + +### Rescore without inference + +`scripts/rescore_evaluation.py` reparses existing raw outputs, annotates the +prediction records, and rewrites predictions/summary without rerunning a model. + +### 11 - Verify generation stopping + +`scripts/verify_generation_stop.py` generates the first five development +examples and requires each complete output to parse with `json.loads`. Balanced +braces trigger stopping; schema validation remains a later responsibility. + +### 12 - Merge + +`scripts/merge_model.py` loads unquantized BF16 base weights on CPU, merges the +final adapter, saves safe-serialized 5 GB shards to a temporary directory, +verifies key files, and atomically promotes the directory to +`outputs/Invited-PHI-8B`. + +This stage creates a standalone Hugging Face model, not GGUF. + +## Chat contract + +`config/chat_template.jinja` uses Llama-style role headers and end-of-turn +tokens. `config/system_prompt.txt` requires exactly one JSON object: + +```json +{ + "is_medical_phi": true, + "score": 1.0, + "reason": "...", + "results": [ + { + "entity_type": "MEDICAL_DIAGNOSIS", + "text": "...", + "score": 1.0, + "evidence": "..." + } + ] +} +``` + +No Markdown, extra prose, inferred facts, or non-`MEDICAL_*` types are allowed. + +## Evaluation caveats + +- A valid JSON prefix followed by trailing text can count as parseable in the + evaluator/rescorer because `raw_decode` is used. +- Entity comparisons use the first expected and predicted result. +- Entity accuracy denominators are all positive records. +- Exact whole-input text/evidence reproduction is measured, not span overlap. +- Result-count correctness is reported separately from schema validity. +- Evaluation covers base-plus-adapter, not the merged standalone model. + +## Path caveat + +All `.sh` wrappers use `/d/ai/training/openbiollm-phi`, while this checkout may +be elsewhere (for example `/d/ai/training/Models-Phi-8B`). Fix the `PROJECT_ROOT` +line in every wrapper or invoke the underlying `uv run python` commands from the +actual repository root.