4.7 KiB
Data and provenance
Provenance statement
The project owner reports that the medical-term, condition, medication, and related terminology datasets used to assemble the category catalogs came from the U.S. National Institutes of Health (NIH), the World Health Organization (WHO), and the U.S. Centers for Disease Control and Prevention (CDC).
The repository does not currently include a source manifest proving which catalog rows came from which organization. It also does not record source URLs, dataset/product names, versions, retrieval dates, licenses, notices, or transformation history. Accordingly, NIH/WHO/CDC origin is documented here as owner-provided provenance, not independently verifiable row-level provenance.
Before redistribution or production use, create a provenance manifest and review the terms for every underlying source. Public availability does not automatically mean unrestricted redistribution, and naming an organization does not imply its endorsement.
Recommended source manifest
Track at least:
| Field | Example purpose |
|---|---|
source_organization |
NIH, WHO, or CDC |
dataset_name |
Exact source product |
source_url |
Stable landing/download URL |
version |
Release/version/date |
retrieved_at |
UTC retrieval timestamp |
license_or_terms_url |
Governing terms |
source_checksum |
Integrity and reproducibility |
category_files |
Derived files |
transformation |
Filtering, normalization, deduplication |
attribution_required |
Required notice |
reviewed_by / reviewed_at |
Governance approval |
Generator source data
Categories
Twelve text files define the fixed medical taxonomy. Each active row is a pipe-separated canonical term followed by grammatical or surface variants. These catalogs are the terminology/keyword source for generation and optional keyword persistence.
Patterns
Each category has a positive and negative pattern file. Patterns are project authored/curated contextual templates; they determine whether generated text is labeled positive or negative. Catalog membership alone does not set polarity.
Subjects
Subjects.txt is required to contain exactly:
He
She
They
Member
SQL schema used by training export
The export reads dbo.TrainingItem joined to dbo.Category and writes:
{
"id": 123,
"category": "MEDICAL_TREATMENT",
"is_positive": true,
"value": "He is receiving physical therapy."
}
Validation requires exactly those fields, a unique positive integer ID, one of the 12 category names, a real JSON Boolean, and a non-empty string value.
Split data
create_splits.py groups by (category, is_positive), shuffles each group with
a stable derived seed, allocates rounded 10% validation and 10% development
counts, and sends the remainder to training. It then independently shuffles each
whole split and verifies uniqueness, disjointness, and complete assignment.
Small strata can have materially different proportions or no training row after rounding. The script guarantees partition integrity, not minimum representation in every split.
SFT data
build_sft.py overwrites each split with:
{
"id": 123,
"messages": [
{"role": "system", "content": "...strict JSON policy..."},
{"role": "user", "content": "He is receiving physical therapy."},
{"role": "assistant", "content": "{\"is_medical_phi\":true,...}"}
]
}
For positives, the expected result contains the source category and uses the
entire input for both text and evidence. For negatives, results is empty.
This means the current task is whole-input contextual classification rather than
precise entity-span extraction.
The conversion is destructive and non-idempotent. Recreate source-shaped splits before rerunning it.
Data lineage
Owner-curated NIH/WHO/CDC-derived catalogs
+ project-authored polarity patterns
+ subject and tense decks
|
v
normalized generated TrainingItem rows
|
v
data/source.jsonl
|
v
source-shaped train/validation/development JSONL
|
v
SFT-shaped train/validation/development JSONL
|
v
adapter, predictions, metrics, merged model
Privacy and governance
The checked-in source is synthetic/generated according to the system design, but that assumption should be verified operationally. Do not import real patient records without a separate privacy, consent, security, retention, and access control review. The model is designed to detect sensitive medical context; it is not itself a compliance determination or a substitute for human review.