# Generator ## Purpose `PHI.Data.Generator` creates labeled contextual medical-language examples. It does not generate a graph of nodes and edges. Its central domain concepts are categories, category values/keywords, patterns, subjects, and training items. ## Projects | Project | Role | |---|---| | `RLHF.Generator` | Catalog parsing, planning, LM client, persistence, generation, keyword services | | `RLHF.Generator.Api` | ASP.NET Core host and HTTP endpoints | | `RLHF.Generator.Tests` | xUnit unit/integration tests using SQLite for test persistence | All three target .NET 10. Production persistence uses EF Core SQL Server. ## Taxonomy The fixed categories are: 1. `MEDICAL_ALLERGY` 2. `MEDICAL_DIAGNOSIS` 3. `MEDICAL_FAMILY_HISTORY` 4. `MEDICAL_MEDICATION` 5. `MEDICAL_MENTAL_HEALTH` 6. `MEDICAL_OTHER` 7. `MEDICAL_PHYSICAL_HEALTH` 8. `MEDICAL_PROCEDURE` 9. `MEDICAL_SUBSTANCE_USE` 10. `MEDICAL_SYMPTOM` 11. `MEDICAL_TEST` 12. `MEDICAL_TREATMENT` Lookup is case-insensitive; stored/returned names use these exact values. ## Input layout and grammar ```text RLHF.Generator/Data/ Subjects.txt Categories/ MEDICAL_ALLERGY.txt ... Patterns/ MEDICAL_ALLERGY.positive.txt MEDICAL_ALLERGY.negative.txt ... ``` Blank lines and lines beginning with `#` are ignored. A category line is: ```text canonical form|allowed variant|another allowed variant ``` Forms must be non-empty and case-insensitively unique within a category. The first is canonical. The remainder are alternative forms available to the normalization model. The exact subjects are `He`, `She`, `They`, and `Member`. Pattern tokens include: | Token | Meaning | |---|---| | `[x]` | Canonical category value; allowed variants are supplied to LM Studio | | `[sub]` | A subject selected from the shuffled subject deck | | `[is\|are\|was\|were]` | Copula selected/rephrased for subject and tense | | `[has\|have\|had]` | Possession/auxiliary selected for subject and tense | | `[a\|an]` | Article resolved locally from the following word | The legacy `[s]` placeholder is rejected. ## Positive and negative examples Every category has an independent positive and negative pattern set. A positive pattern describes real medical context; a negative pattern uses the same domain language without asserting medical information about the person being analyzed. Representative intended behavior: | Input | Expected label | Why | |---|---:|---| | `Has diabetes` | Positive | Subjectless clinical fragment with an implied member/patient | | `He had a reaction to shellfish` | Positive | Actual allergy/reaction assertion | | `She wrote a book about diabetes` | Negative | Creative/educational context, not her diagnosis | | `Had a reaction to the horrible cover of Van Halen's Jump` | Negative | Idiomatic non-medical reaction | Those examples describe the intended learned behavior; they are not hard-coded rules or keyword matchers in the inference pipeline. ## Plan construction For each category, the service runs the positive pass and then the negative pass. It partitions patterns into: - **value-bound:** contains `[x]`; - **generic:** does not contain `[x]`. It rejects values without any value-bound pattern and value-bound patterns when the category has no values. It then: 1. shuffles values, value-bound patterns, and generic patterns; 2. creates shuffled/cycling subject and tense decks; 3. emits every generic pattern once; 4. emits `max(value-bound pattern count, value count)` value-bound plans; 5. reshuffles a deck whenever it is exhausted. The tense deck is `null`, `past`, `present`, and `future`. The canonical category value is placed in `[x]`; variants are not independently randomly sampled. Because each polarity is planned from its own pattern/value counts, positive and negative totals are not guaranteed to be equal. ## Subject and fragment behavior The precise behavior is: - A pattern containing `[sub]` gets one shuffled explicit subject. - A pattern without `[sub]` is marked as a fragment and gets no inserted subject. - `Member` is one of the four explicit subject choices; it is not a fallback value. - For a fragment, the LM system prompt uses an implied singular member/patient only to choose grammar. It explicitly forbids returning or inventing that subject. - Some fragments are completed locally without an LM call when no unresolved choice, protected-form decision, or meaningful tense rewrite remains. Therefore the safe summary is: **subjectless patterns remain subjectless, while grammar assumes an implied singular member/patient.** ## LM Studio normalization Default settings: | Setting | Value | |---|---| | Endpoint | `http://localhost:1234/v1/chat/completions` | | Model | `qwen/qwen3.5-9b` | | Temperature | `0.0` | | Top-p | `1.0` | | Max tokens | `96` | | Seed | `42` | | Timeout | `300` seconds | Each request carries fragment status, optional protected subject, protected medical term and allowed forms, requested tense, and the sentence/template. The system prompt requires exactly one unwrapped line, preserves meaning and polarity, resolves all bracket expressions, and forbids invented fragment predicates. Generation rejects empty output, newlines, unresolved square brackets, quotation/JSON/code-fence wrappers, and common response labels. ## Concurrency and durability Defaults are seed 42, parallelism 8, save batches of 24, and progress logs every 5 seconds. Workers generate concurrently into a bounded channel. A single persistence writer places results into a sorted buffer and commits only the next contiguous plan ordinals. Each transaction inserts the batch and advances the generation checkpoint together. A run can resume only when its seed, input fingerprint, configuration fingerprint, expected count, committed row count, and last ordinal agree. Changed inputs/configuration or inconsistent persisted state are rejected. ## Keyword catalog Keyword loading uses each category file's canonical value and variants. Loading is additive, case-insensitive, and idempotent. It does not update existing training items. Backfill is a separate operation: - dry-run is available; - only unique matches are assigned; - unmatched or ambiguous rows remain without a keyword. ## Database migrations The migrations progress through: 1. initial `Category` and `TrainingItem` schema; 2. durable generation-run/checkpoint support; 3. `Keyword`, `KeywordVariant`, and training-item keyword linkage. `AddKeywordCatalog.sql` provides a SQL alternative for the keyword phase. ## Configuration changes that invalidate resume Treat these as generation identity: - category, pattern, or subject file contents/paths; - random seed; - concurrency and save/progress options included in the fingerprint; - LM endpoint/model/prompt and generation settings. Start a clean run rather than bypassing fingerprint checks.