Models-Phi-8B/docs/data-and-provenance.md

146 lines
4.7 KiB
Markdown

# Data and provenance
## Provenance statement
The project owner reports that the medical-term, condition, medication, and
related terminology datasets used to assemble the category catalogs came from
the **U.S. National Institutes of Health (NIH)**, the **World Health
Organization (WHO)**, and the **U.S. Centers for Disease Control and Prevention
(CDC)**.
The repository does not currently include a source manifest proving which
catalog rows came from which organization. It also does not record source URLs,
dataset/product names, versions, retrieval dates, licenses, notices, or
transformation history. Accordingly, NIH/WHO/CDC origin is documented here as
owner-provided provenance, not independently verifiable row-level provenance.
Before redistribution or production use, create a provenance manifest and
review the terms for every underlying source. Public availability does not
automatically mean unrestricted redistribution, and naming an organization does
not imply its endorsement.
## Recommended source manifest
Track at least:
| Field | Example purpose |
|---|---|
| `source_organization` | NIH, WHO, or CDC |
| `dataset_name` | Exact source product |
| `source_url` | Stable landing/download URL |
| `version` | Release/version/date |
| `retrieved_at` | UTC retrieval timestamp |
| `license_or_terms_url` | Governing terms |
| `source_checksum` | Integrity and reproducibility |
| `category_files` | Derived files |
| `transformation` | Filtering, normalization, deduplication |
| `attribution_required` | Required notice |
| `reviewed_by` / `reviewed_at` | Governance approval |
## Generator source data
### Categories
Twelve text files define the fixed medical taxonomy. Each active row is a
pipe-separated canonical term followed by grammatical or surface variants.
These catalogs are the terminology/keyword source for generation and optional
keyword persistence.
### Patterns
Each category has a positive and negative pattern file. Patterns are project
authored/curated contextual templates; they determine whether generated text is
labeled positive or negative. Catalog membership alone does not set polarity.
### Subjects
`Subjects.txt` is required to contain exactly:
```text
He
She
They
Member
```
## SQL schema used by training export
The export reads `dbo.TrainingItem` joined to `dbo.Category` and writes:
```json
{
"id": 123,
"category": "MEDICAL_TREATMENT",
"is_positive": true,
"value": "He is receiving physical therapy."
}
```
Validation requires exactly those fields, a unique positive integer ID, one of
the 12 category names, a real JSON Boolean, and a non-empty string value.
## Split data
`create_splits.py` groups by `(category, is_positive)`, shuffles each group with
a stable derived seed, allocates rounded 10% validation and 10% development
counts, and sends the remainder to training. It then independently shuffles each
whole split and verifies uniqueness, disjointness, and complete assignment.
Small strata can have materially different proportions or no training row after
rounding. The script guarantees partition integrity, not minimum representation
in every split.
## SFT data
`build_sft.py` overwrites each split with:
```json
{
"id": 123,
"messages": [
{"role": "system", "content": "...strict JSON policy..."},
{"role": "user", "content": "He is receiving physical therapy."},
{"role": "assistant", "content": "{\"is_medical_phi\":true,...}"}
]
}
```
For positives, the expected result contains the source category and uses the
entire input for both `text` and `evidence`. For negatives, `results` is empty.
This means the current task is whole-input contextual classification rather than
precise entity-span extraction.
The conversion is destructive and non-idempotent. Recreate source-shaped splits
before rerunning it.
## Data lineage
```text
Owner-curated NIH/WHO/CDC-derived catalogs
+ project-authored polarity patterns
+ subject and tense decks
|
v
normalized generated TrainingItem rows
|
v
data/source.jsonl
|
v
source-shaped train/validation/development JSONL
|
v
SFT-shaped train/validation/development JSONL
|
v
adapter, predictions, metrics, merged model
```
## Privacy and governance
The checked-in source is synthetic/generated according to the system design,
but that assumption should be verified operationally. Do not import real patient
records without a separate privacy, consent, security, retention, and access
control review. The model is designed to detect sensitive medical context; it is
not itself a compliance determination or a substitute for human review.