added mmd and pdf

This commit is contained in:
GAJ3T 2026-10-15 10:38:18 -04:00
parent 42ee4c2605
commit 55065d88ff
9 changed files with 145 additions and 5 deletions

View File

@ -45,13 +45,27 @@ whether the input contains medical PHI and, for positives, its category.
The editable draw.io documents are:
- [Level 1: System context](diagrams/01-system-context.drawio)
- [Level 2: Containers](diagrams/02-containers.drawio)
- [Level 3: Components](diagrams/03-generator-components.drawio)
- [Level 4: Code](diagrams/04-generator-code.drawio)
- Level 1: System context
([draw.io](diagrams/01-system-context.drawio),
[Mermaid](diagrams/01-system-context.mmd),
[PDF](diagrams/01-system-context.pdf))
- Level 2: Containers
([draw.io](diagrams/02-containers.drawio),
[Mermaid](diagrams/02-containers.mmd),
[PDF](diagrams/02-containers.pdf))
- Level 3: Components
([draw.io](diagrams/03-generator-components.drawio),
[Mermaid](diagrams/03-generator-components.mmd),
[PDF](diagrams/03-generator-components.pdf))
- Level 4: Code
([draw.io](diagrams/04-generator-code.drawio),
[Mermaid](diagrams/04-generator-code.mmd),
[PDF](diagrams/04-generator-code.pdf))
Open the files with diagrams.net, the draw.io desktop application, or the VS
Code Draw.io Integration extension.
Code Draw.io Integration extension. The `.drawio` files are the authoritative
editable diagrams. Mermaid companions provide text-based, diff-friendly views;
PDF companions provide portable rendered views.
## Documentation conventions

View File

@ -0,0 +1,23 @@
flowchart LR
operator["ML / Data Operator<br/>Curates inputs, runs generation and training,<br/>and reviews metrics"]
sources["NIH / WHO / CDC Sources<br/>Owner-reported origins of<br/>medical terminology catalogs"]
system["Medical PHI Model System<br/>Generates contextual examples and fine-tunes<br/>an 8B model to classify medical PHI<br/>and emit category-based JSON"]
runtime["Model Runtime<br/>Hosts the merged Hugging Face model<br/>or a separately converted GGUF artifact"]
app["Application / Reviewer<br/>Submits text and consumes<br/>strict JSON classification results"]
presidio["Downstream Presidio Integration<br/>Consumes compatible MEDICAL_* category/entity types<br/>Implementation is outside this repository"]
operator -->|Operates and reviews| system
sources -->|Curated terminology; provenance manifest needed| system
system -->|Produces model artifacts| runtime
system -.->|Intended category contract| presidio
app -->|Inference requests| runtime
runtime -->|Strict JSON decisions| app
classDef person fill:#fff2cc,stroke:#d6b656,color:#000;
classDef system fill:#1168bd,stroke:#0b4884,color:#fff;
classDef external fill:#d5e8d4,stroke:#82b366,color:#000;
classDef source fill:#f8cecc,stroke:#b85450,color:#000;
class operator,app person;
class system system;
class runtime,presidio external;
class sources source;

Binary file not shown.

View File

@ -0,0 +1,31 @@
flowchart LR
operator["Operator"]
lm["LM Studio<br/>OpenAI-compatible API<br/>qwen/qwen3.5-9b normalization"]
base["Base Model<br/>Local Llama3-OpenBioLLM-8B"]
subgraph repository["Medical PHI Model Repository"]
catalogs["Catalogs and Patterns<br/>Text files<br/>12 categories, variants, subjects,<br/>positive/negative templates"]
api["Generator API<br/>ASP.NET Core / .NET 10<br/>Validates, plans, normalizes,<br/>and persists ordered batches"]
sql[("Training Database<br/>SQL Server<br/>Categories, keywords,<br/>generation runs, training items")]
jsonl[("Training Data<br/>JSONL<br/>Source, train, validation,<br/>development")]
pipeline["Training Pipeline<br/>Python 3.12 / PyTorch / TRL / PEFT<br/>Validate, split, SFT, train,<br/>evaluate, merge"]
artifacts[("Model Artifacts<br/>LoRA adapters, metrics,<br/>merged BF16 model, GGUF files")]
catalogs -->|Reads| api
api -->|Persists and checkpoints| sql
sql -->|PowerShell export| jsonl
jsonl -->|Reads and rewrites| pipeline
pipeline -->|Writes| artifacts
end
operator -->|Runs| api
operator -->|Operates and reviews| pipeline
api -->|Normalizes text| lm
base -->|Weights and tokenizer| pipeline
classDef person fill:#fff2cc,stroke:#d6b656,color:#000;
classDef container fill:#438dd5,stroke:#0b4884,color:#fff;
classDef external fill:#999,stroke:#666,color:#fff;
class operator person;
class catalogs,api,sql,jsonl,pipeline,artifacts container;
class lm,base external;

Binary file not shown.

View File

@ -0,0 +1,37 @@
flowchart LR
data[("Catalog and pattern files")]
lm["LM Studio API"]
sql[("SQL Server")]
subgraph generator["RLHF.Generator.Api + RLHF.Generator"]
routes["HTTP Endpoints<br/>Program.cs<br/>Validation, contracts,<br/>status mapping"]
reader["TrainingDataReader<br/>Validates and parses subjects,<br/>categories, and patterns"]
generation["TrainingGenerationService<br/>Plans, fingerprints, resumes,<br/>parallelizes, orders, validates,<br/>and checkpoints"]
lmclient["LmStudioClient<br/>Local token resolution,<br/>prompt construction,<br/>HTTP completion"]
keyword["Keyword Services<br/>Load canonical values/variants;<br/>dry-run or apply unique backfill"]
reset["TrainingItemResetService<br/>Clears generated rows<br/>and run checkpoints"]
db["RlhfGeneratorDbContext<br/>EF Core mappings, seeds, indexes,<br/>relationships, and transactions"]
options["Options<br/>GenerationOptions<br/>LmStudioOptions<br/>SentenceTenses"]
routes -->|Dispatches| generation
routes -->|Dispatches| keyword
routes -->|Dispatches| reset
generation -->|Reads definitions| reader
generation -->|Normalizes| lmclient
generation -->|Persists batches| db
keyword -->|Uses| db
reset -->|Uses| db
options -->|Configures| generation
options -->|Configures| lmclient
end
data -->|Reads| reader
lmclient -->|HTTP chat completion| lm
db -->|EF Core SQL| sql
classDef component fill:#85bbf0,stroke:#0b4884,color:#000;
classDef primary fill:#1168bd,stroke:#0b4884,color:#fff;
classDef external fill:#e1d5e7,stroke:#9673a6,color:#000;
class routes,reader,lmclient,keyword,reset,db,options component;
class generation primary;
class data,lm,sql external;

Binary file not shown.

View File

@ -0,0 +1,35 @@
flowchart TB
reader["TrainingDataReader<br/>+ ValidateAsync()<br/>+ ReadCategoryValuesAsync()<br/>+ ReadPatternsAsync()"]
definitions["Training definitions<br/>TrainingCategories<br/>TrainingSubjects<br/>TrainingCategoryDefinition<br/>TrainingCategoryValue"]
channel["Bounded Channel<br/>Parallel workers write completed plan items;<br/>ordered writer consumes"]
service["TrainingGenerationService<br/>+ GenerateAsync()<br/>- GenerateCategoryAsync()<br/>- CreatePlan()<br/>- GeneratePassAsync()<br/>- PersistOrderedAsync()<br/>- FindResumableRunAsync()<br/>- ValidateGeneratedValue()"]
lmclient["LmStudioClient<br/>+ CompleteAsync()<br/>- ResolveFragmentTokens()<br/>- ResolveArticles()<br/>- SendCompletionAsync()"]
config["Configuration<br/>GenerationOptions<br/>LmStudioOptions<br/>SentenceTenses"]
context["RlhfGeneratorDbContext<br/>DbSet Category<br/>DbSet TrainingItem<br/>DbSet GenerationRun<br/>DbSet Keyword<br/>DbSet KeywordVariant"]
run["GenerationRun<br/>Status, seed, fingerprints,<br/>checkpoint and counts,<br/>failure/cancellation data"]
item["TrainingItem<br/>RecordKey, PlanOrdinal,<br/>Category, Keyword,<br/>IsPositive, Value"]
service -->|Validates and reads inputs| reader
reader -->|Returns| definitions
service -->|Plans with| definitions
service -->|Coordinates producers and consumer| channel
service -->|Calls concurrently| lmclient
config -->|Binds| service
config -->|Binds| lmclient
service -->|Queries and transacts| context
context -->|Tracks| run
context -->|Tracks| item
run -->|Owns ordered items| item
classDef core fill:#dae8fc,stroke:#6c8ebf,color:#000;
classDef input fill:#d5e8d4,stroke:#82b366,color:#000;
classDef integration fill:#e1d5e7,stroke:#9673a6,color:#000;
classDef persistence fill:#f8cecc,stroke:#b85450,color:#000;
classDef config fill:#fff2cc,stroke:#d6b656,color:#000;
classDef coordination fill:#ffe6cc,stroke:#d79b00,color:#000;
class service core;
class reader,definitions input;
class lmclient integration;
class context,run,item persistence;
class config config;
class channel coordination;

Binary file not shown.