202 lines
6.7 KiB
Markdown
202 lines
6.7 KiB
Markdown
# Generator
|
|
|
|
## Purpose
|
|
|
|
`PHI.Data.Generator` creates labeled contextual medical-language examples. It
|
|
does not generate a graph of nodes and edges. Its central domain concepts are
|
|
categories, category values/keywords, patterns, subjects, and training items.
|
|
|
|
## Projects
|
|
|
|
| Project | Role |
|
|
|---|---|
|
|
| `RLHF.Generator` | Catalog parsing, planning, LM client, persistence, generation, keyword services |
|
|
| `RLHF.Generator.Api` | ASP.NET Core host and HTTP endpoints |
|
|
| `RLHF.Generator.Tests` | xUnit unit/integration tests using SQLite for test persistence |
|
|
|
|
All three target .NET 10. Production persistence uses EF Core SQL Server.
|
|
|
|
## Taxonomy
|
|
|
|
The fixed categories are:
|
|
|
|
1. `MEDICAL_ALLERGY`
|
|
2. `MEDICAL_DIAGNOSIS`
|
|
3. `MEDICAL_FAMILY_HISTORY`
|
|
4. `MEDICAL_MEDICATION`
|
|
5. `MEDICAL_MENTAL_HEALTH`
|
|
6. `MEDICAL_OTHER`
|
|
7. `MEDICAL_PHYSICAL_HEALTH`
|
|
8. `MEDICAL_PROCEDURE`
|
|
9. `MEDICAL_SUBSTANCE_USE`
|
|
10. `MEDICAL_SYMPTOM`
|
|
11. `MEDICAL_TEST`
|
|
12. `MEDICAL_TREATMENT`
|
|
|
|
Lookup is case-insensitive; stored/returned names use these exact values.
|
|
|
|
## Input layout and grammar
|
|
|
|
```text
|
|
RLHF.Generator/Data/
|
|
Subjects.txt
|
|
Categories/
|
|
MEDICAL_ALLERGY.txt
|
|
...
|
|
Patterns/
|
|
MEDICAL_ALLERGY.positive.txt
|
|
MEDICAL_ALLERGY.negative.txt
|
|
...
|
|
```
|
|
|
|
Blank lines and lines beginning with `#` are ignored. A category line is:
|
|
|
|
```text
|
|
canonical form|allowed variant|another allowed variant
|
|
```
|
|
|
|
Forms must be non-empty and case-insensitively unique within a category. The
|
|
first is canonical. The remainder are alternative forms available to the
|
|
normalization model.
|
|
|
|
The exact subjects are `He`, `She`, `They`, and `Member`.
|
|
|
|
Pattern tokens include:
|
|
|
|
| Token | Meaning |
|
|
|---|---|
|
|
| `[x]` | Canonical category value; allowed variants are supplied to LM Studio |
|
|
| `[sub]` | A subject selected from the shuffled subject deck |
|
|
| `[is\|are\|was\|were]` | Copula selected/rephrased for subject and tense |
|
|
| `[has\|have\|had]` | Possession/auxiliary selected for subject and tense |
|
|
| `[a\|an]` | Article resolved locally from the following word |
|
|
|
|
The legacy `[s]` placeholder is rejected.
|
|
|
|
## Positive and negative examples
|
|
|
|
Every category has an independent positive and negative pattern set. A positive
|
|
pattern describes real medical context; a negative pattern uses the same domain
|
|
language without asserting medical information about the person being analyzed.
|
|
|
|
Representative intended behavior:
|
|
|
|
| Input | Expected label | Why |
|
|
|---|---:|---|
|
|
| `Has diabetes` | Positive | Subjectless clinical fragment with an implied member/patient |
|
|
| `He had a reaction to shellfish` | Positive | Actual allergy/reaction assertion |
|
|
| `She wrote a book about diabetes` | Negative | Creative/educational context, not her diagnosis |
|
|
| `Had a reaction to the horrible cover of Van Halen's Jump` | Negative | Idiomatic non-medical reaction |
|
|
|
|
Those examples describe the intended learned behavior; they are not hard-coded
|
|
rules or keyword matchers in the inference pipeline.
|
|
|
|
## Plan construction
|
|
|
|
For each category, the service runs the positive pass and then the negative
|
|
pass. It partitions patterns into:
|
|
|
|
- **value-bound:** contains `[x]`;
|
|
- **generic:** does not contain `[x]`.
|
|
|
|
It rejects values without any value-bound pattern and value-bound patterns when
|
|
the category has no values. It then:
|
|
|
|
1. shuffles values, value-bound patterns, and generic patterns;
|
|
2. creates shuffled/cycling subject and tense decks;
|
|
3. emits every generic pattern once;
|
|
4. emits `max(value-bound pattern count, value count)` value-bound plans;
|
|
5. reshuffles a deck whenever it is exhausted.
|
|
|
|
The tense deck is `null`, `past`, `present`, and `future`. The canonical category
|
|
value is placed in `[x]`; variants are not independently randomly sampled.
|
|
|
|
Because each polarity is planned from its own pattern/value counts, positive and
|
|
negative totals are not guaranteed to be equal.
|
|
|
|
## Subject and fragment behavior
|
|
|
|
The precise behavior is:
|
|
|
|
- A pattern containing `[sub]` gets one shuffled explicit subject.
|
|
- A pattern without `[sub]` is marked as a fragment and gets no inserted
|
|
subject.
|
|
- `Member` is one of the four explicit subject choices; it is not a fallback
|
|
value.
|
|
- For a fragment, the LM system prompt uses an implied singular member/patient
|
|
only to choose grammar. It explicitly forbids returning or inventing that
|
|
subject.
|
|
- Some fragments are completed locally without an LM call when no unresolved
|
|
choice, protected-form decision, or meaningful tense rewrite remains.
|
|
|
|
Therefore the safe summary is: **subjectless patterns remain subjectless, while
|
|
grammar assumes an implied singular member/patient.**
|
|
|
|
## LM Studio normalization
|
|
|
|
Default settings:
|
|
|
|
| Setting | Value |
|
|
|---|---|
|
|
| Endpoint | `http://localhost:1234/v1/chat/completions` |
|
|
| Model | `qwen/qwen3.5-9b` |
|
|
| Temperature | `0.0` |
|
|
| Top-p | `1.0` |
|
|
| Max tokens | `96` |
|
|
| Seed | `42` |
|
|
| Timeout | `300` seconds |
|
|
|
|
Each request carries fragment status, optional protected subject, protected
|
|
medical term and allowed forms, requested tense, and the sentence/template. The
|
|
system prompt requires exactly one unwrapped line, preserves meaning and
|
|
polarity, resolves all bracket expressions, and forbids invented fragment
|
|
predicates.
|
|
|
|
Generation rejects empty output, newlines, unresolved square brackets,
|
|
quotation/JSON/code-fence wrappers, and common response labels.
|
|
|
|
## Concurrency and durability
|
|
|
|
Defaults are seed 42, parallelism 8, save batches of 24, and progress logs every
|
|
5 seconds.
|
|
|
|
Workers generate concurrently into a bounded channel. A single persistence
|
|
writer places results into a sorted buffer and commits only the next contiguous
|
|
plan ordinals. Each transaction inserts the batch and advances the generation
|
|
checkpoint together.
|
|
|
|
A run can resume only when its seed, input fingerprint, configuration
|
|
fingerprint, expected count, committed row count, and last ordinal agree.
|
|
Changed inputs/configuration or inconsistent persisted state are rejected.
|
|
|
|
## Keyword catalog
|
|
|
|
Keyword loading uses each category file's canonical value and variants. Loading
|
|
is additive, case-insensitive, and idempotent. It does not update existing
|
|
training items. Backfill is a separate operation:
|
|
|
|
- dry-run is available;
|
|
- only unique matches are assigned;
|
|
- unmatched or ambiguous rows remain without a keyword.
|
|
|
|
## Database migrations
|
|
|
|
The migrations progress through:
|
|
|
|
1. initial `Category` and `TrainingItem` schema;
|
|
2. durable generation-run/checkpoint support;
|
|
3. `Keyword`, `KeywordVariant`, and training-item keyword linkage.
|
|
|
|
`AddKeywordCatalog.sql` provides a SQL alternative for the keyword phase.
|
|
|
|
## Configuration changes that invalidate resume
|
|
|
|
Treat these as generation identity:
|
|
|
|
- category, pattern, or subject file contents/paths;
|
|
- random seed;
|
|
- concurrency and save/progress options included in the fingerprint;
|
|
- LM endpoint/model/prompt and generation settings.
|
|
|
|
Start a clean run rather than bypassing fingerprint checks.
|