Models-Phi-8B/docs/generator.md

202 lines
6.7 KiB
Markdown

# Generator
## Purpose
`PHI.Data.Generator` creates labeled contextual medical-language examples. It
does not generate a graph of nodes and edges. Its central domain concepts are
categories, category values/keywords, patterns, subjects, and training items.
## Projects
| Project | Role |
|---|---|
| `RLHF.Generator` | Catalog parsing, planning, LM client, persistence, generation, keyword services |
| `RLHF.Generator.Api` | ASP.NET Core host and HTTP endpoints |
| `RLHF.Generator.Tests` | xUnit unit/integration tests using SQLite for test persistence |
All three target .NET 10. Production persistence uses EF Core SQL Server.
## Taxonomy
The fixed categories are:
1. `MEDICAL_ALLERGY`
2. `MEDICAL_DIAGNOSIS`
3. `MEDICAL_FAMILY_HISTORY`
4. `MEDICAL_MEDICATION`
5. `MEDICAL_MENTAL_HEALTH`
6. `MEDICAL_OTHER`
7. `MEDICAL_PHYSICAL_HEALTH`
8. `MEDICAL_PROCEDURE`
9. `MEDICAL_SUBSTANCE_USE`
10. `MEDICAL_SYMPTOM`
11. `MEDICAL_TEST`
12. `MEDICAL_TREATMENT`
Lookup is case-insensitive; stored/returned names use these exact values.
## Input layout and grammar
```text
RLHF.Generator/Data/
Subjects.txt
Categories/
MEDICAL_ALLERGY.txt
...
Patterns/
MEDICAL_ALLERGY.positive.txt
MEDICAL_ALLERGY.negative.txt
...
```
Blank lines and lines beginning with `#` are ignored. A category line is:
```text
canonical form|allowed variant|another allowed variant
```
Forms must be non-empty and case-insensitively unique within a category. The
first is canonical. The remainder are alternative forms available to the
normalization model.
The exact subjects are `He`, `She`, `They`, and `Member`.
Pattern tokens include:
| Token | Meaning |
|---|---|
| `[x]` | Canonical category value; allowed variants are supplied to LM Studio |
| `[sub]` | A subject selected from the shuffled subject deck |
| `[is\|are\|was\|were]` | Copula selected/rephrased for subject and tense |
| `[has\|have\|had]` | Possession/auxiliary selected for subject and tense |
| `[a\|an]` | Article resolved locally from the following word |
The legacy `[s]` placeholder is rejected.
## Positive and negative examples
Every category has an independent positive and negative pattern set. A positive
pattern describes real medical context; a negative pattern uses the same domain
language without asserting medical information about the person being analyzed.
Representative intended behavior:
| Input | Expected label | Why |
|---|---:|---|
| `Has diabetes` | Positive | Subjectless clinical fragment with an implied member/patient |
| `He had a reaction to shellfish` | Positive | Actual allergy/reaction assertion |
| `She wrote a book about diabetes` | Negative | Creative/educational context, not her diagnosis |
| `Had a reaction to the horrible cover of Van Halen's Jump` | Negative | Idiomatic non-medical reaction |
Those examples describe the intended learned behavior; they are not hard-coded
rules or keyword matchers in the inference pipeline.
## Plan construction
For each category, the service runs the positive pass and then the negative
pass. It partitions patterns into:
- **value-bound:** contains `[x]`;
- **generic:** does not contain `[x]`.
It rejects values without any value-bound pattern and value-bound patterns when
the category has no values. It then:
1. shuffles values, value-bound patterns, and generic patterns;
2. creates shuffled/cycling subject and tense decks;
3. emits every generic pattern once;
4. emits `max(value-bound pattern count, value count)` value-bound plans;
5. reshuffles a deck whenever it is exhausted.
The tense deck is `null`, `past`, `present`, and `future`. The canonical category
value is placed in `[x]`; variants are not independently randomly sampled.
Because each polarity is planned from its own pattern/value counts, positive and
negative totals are not guaranteed to be equal.
## Subject and fragment behavior
The precise behavior is:
- A pattern containing `[sub]` gets one shuffled explicit subject.
- A pattern without `[sub]` is marked as a fragment and gets no inserted
subject.
- `Member` is one of the four explicit subject choices; it is not a fallback
value.
- For a fragment, the LM system prompt uses an implied singular member/patient
only to choose grammar. It explicitly forbids returning or inventing that
subject.
- Some fragments are completed locally without an LM call when no unresolved
choice, protected-form decision, or meaningful tense rewrite remains.
Therefore the safe summary is: **subjectless patterns remain subjectless, while
grammar assumes an implied singular member/patient.**
## LM Studio normalization
Default settings:
| Setting | Value |
|---|---|
| Endpoint | `http://localhost:1234/v1/chat/completions` |
| Model | `qwen/qwen3.5-9b` |
| Temperature | `0.0` |
| Top-p | `1.0` |
| Max tokens | `96` |
| Seed | `42` |
| Timeout | `300` seconds |
Each request carries fragment status, optional protected subject, protected
medical term and allowed forms, requested tense, and the sentence/template. The
system prompt requires exactly one unwrapped line, preserves meaning and
polarity, resolves all bracket expressions, and forbids invented fragment
predicates.
Generation rejects empty output, newlines, unresolved square brackets,
quotation/JSON/code-fence wrappers, and common response labels.
## Concurrency and durability
Defaults are seed 42, parallelism 8, save batches of 24, and progress logs every
5 seconds.
Workers generate concurrently into a bounded channel. A single persistence
writer places results into a sorted buffer and commits only the next contiguous
plan ordinals. Each transaction inserts the batch and advances the generation
checkpoint together.
A run can resume only when its seed, input fingerprint, configuration
fingerprint, expected count, committed row count, and last ordinal agree.
Changed inputs/configuration or inconsistent persisted state are rejected.
## Keyword catalog
Keyword loading uses each category file's canonical value and variants. Loading
is additive, case-insensitive, and idempotent. It does not update existing
training items. Backfill is a separate operation:
- dry-run is available;
- only unique matches are assigned;
- unmatched or ambiguous rows remain without a keyword.
## Database migrations
The migrations progress through:
1. initial `Category` and `TrainingItem` schema;
2. durable generation-run/checkpoint support;
3. `Keyword`, `KeywordVariant`, and training-item keyword linkage.
`AddKeywordCatalog.sql` provides a SQL alternative for the keyword phase.
## Configuration changes that invalidate resume
Treat these as generation identity:
- category, pattern, or subject file contents/paths;
- random seed;
- concurrency and save/progress options included in the fingerprint;
- LM endpoint/model/prompt and generation settings.
Start a clean run rather than bypassing fingerprint checks.