146 lines
4.7 KiB
Markdown
146 lines
4.7 KiB
Markdown
# Data and provenance
|
|
|
|
## Provenance statement
|
|
|
|
The project owner reports that the medical-term, condition, medication, and
|
|
related terminology datasets used to assemble the category catalogs came from
|
|
the **U.S. National Institutes of Health (NIH)**, the **World Health
|
|
Organization (WHO)**, and the **U.S. Centers for Disease Control and Prevention
|
|
(CDC)**.
|
|
|
|
The repository does not currently include a source manifest proving which
|
|
catalog rows came from which organization. It also does not record source URLs,
|
|
dataset/product names, versions, retrieval dates, licenses, notices, or
|
|
transformation history. Accordingly, NIH/WHO/CDC origin is documented here as
|
|
owner-provided provenance, not independently verifiable row-level provenance.
|
|
|
|
Before redistribution or production use, create a provenance manifest and
|
|
review the terms for every underlying source. Public availability does not
|
|
automatically mean unrestricted redistribution, and naming an organization does
|
|
not imply its endorsement.
|
|
|
|
## Recommended source manifest
|
|
|
|
Track at least:
|
|
|
|
| Field | Example purpose |
|
|
|---|---|
|
|
| `source_organization` | NIH, WHO, or CDC |
|
|
| `dataset_name` | Exact source product |
|
|
| `source_url` | Stable landing/download URL |
|
|
| `version` | Release/version/date |
|
|
| `retrieved_at` | UTC retrieval timestamp |
|
|
| `license_or_terms_url` | Governing terms |
|
|
| `source_checksum` | Integrity and reproducibility |
|
|
| `category_files` | Derived files |
|
|
| `transformation` | Filtering, normalization, deduplication |
|
|
| `attribution_required` | Required notice |
|
|
| `reviewed_by` / `reviewed_at` | Governance approval |
|
|
|
|
## Generator source data
|
|
|
|
### Categories
|
|
|
|
Twelve text files define the fixed medical taxonomy. Each active row is a
|
|
pipe-separated canonical term followed by grammatical or surface variants.
|
|
These catalogs are the terminology/keyword source for generation and optional
|
|
keyword persistence.
|
|
|
|
### Patterns
|
|
|
|
Each category has a positive and negative pattern file. Patterns are project
|
|
authored/curated contextual templates; they determine whether generated text is
|
|
labeled positive or negative. Catalog membership alone does not set polarity.
|
|
|
|
### Subjects
|
|
|
|
`Subjects.txt` is required to contain exactly:
|
|
|
|
```text
|
|
He
|
|
She
|
|
They
|
|
Member
|
|
```
|
|
|
|
## SQL schema used by training export
|
|
|
|
The export reads `dbo.TrainingItem` joined to `dbo.Category` and writes:
|
|
|
|
```json
|
|
{
|
|
"id": 123,
|
|
"category": "MEDICAL_TREATMENT",
|
|
"is_positive": true,
|
|
"value": "He is receiving physical therapy."
|
|
}
|
|
```
|
|
|
|
Validation requires exactly those fields, a unique positive integer ID, one of
|
|
the 12 category names, a real JSON Boolean, and a non-empty string value.
|
|
|
|
## Split data
|
|
|
|
`create_splits.py` groups by `(category, is_positive)`, shuffles each group with
|
|
a stable derived seed, allocates rounded 10% validation and 10% development
|
|
counts, and sends the remainder to training. It then independently shuffles each
|
|
whole split and verifies uniqueness, disjointness, and complete assignment.
|
|
|
|
Small strata can have materially different proportions or no training row after
|
|
rounding. The script guarantees partition integrity, not minimum representation
|
|
in every split.
|
|
|
|
## SFT data
|
|
|
|
`build_sft.py` overwrites each split with:
|
|
|
|
```json
|
|
{
|
|
"id": 123,
|
|
"messages": [
|
|
{"role": "system", "content": "...strict JSON policy..."},
|
|
{"role": "user", "content": "He is receiving physical therapy."},
|
|
{"role": "assistant", "content": "{\"is_medical_phi\":true,...}"}
|
|
]
|
|
}
|
|
```
|
|
|
|
For positives, the expected result contains the source category and uses the
|
|
entire input for both `text` and `evidence`. For negatives, `results` is empty.
|
|
This means the current task is whole-input contextual classification rather than
|
|
precise entity-span extraction.
|
|
|
|
The conversion is destructive and non-idempotent. Recreate source-shaped splits
|
|
before rerunning it.
|
|
|
|
## Data lineage
|
|
|
|
```text
|
|
Owner-curated NIH/WHO/CDC-derived catalogs
|
|
+ project-authored polarity patterns
|
|
+ subject and tense decks
|
|
|
|
|
v
|
|
normalized generated TrainingItem rows
|
|
|
|
|
v
|
|
data/source.jsonl
|
|
|
|
|
v
|
|
source-shaped train/validation/development JSONL
|
|
|
|
|
v
|
|
SFT-shaped train/validation/development JSONL
|
|
|
|
|
v
|
|
adapter, predictions, metrics, merged model
|
|
```
|
|
|
|
## Privacy and governance
|
|
|
|
The checked-in source is synthetic/generated according to the system design,
|
|
but that assumption should be verified operationally. Do not import real patient
|
|
records without a separate privacy, consent, security, retention, and access
|
|
control review. The model is designed to detect sensitive medical context; it is
|
|
not itself a compliance determination or a substitute for human review.
|