PHI-Model-8B/docs/data-and-provenance.md

4.7 KiB

Data and provenance

Provenance statement

The project owner reports that the medical-term, condition, medication, and related terminology datasets used to assemble the category catalogs came from the U.S. National Institutes of Health (NIH), the World Health Organization (WHO), and the U.S. Centers for Disease Control and Prevention (CDC).

The repository does not currently include a source manifest proving which catalog rows came from which organization. It also does not record source URLs, dataset/product names, versions, retrieval dates, licenses, notices, or transformation history. Accordingly, NIH/WHO/CDC origin is documented here as owner-provided provenance, not independently verifiable row-level provenance.

Before redistribution or production use, create a provenance manifest and review the terms for every underlying source. Public availability does not automatically mean unrestricted redistribution, and naming an organization does not imply its endorsement.

Track at least:

Field Example purpose
source_organization NIH, WHO, or CDC
dataset_name Exact source product
source_url Stable landing/download URL
version Release/version/date
retrieved_at UTC retrieval timestamp
license_or_terms_url Governing terms
source_checksum Integrity and reproducibility
category_files Derived files
transformation Filtering, normalization, deduplication
attribution_required Required notice
reviewed_by / reviewed_at Governance approval

Generator source data

Categories

Twelve text files define the fixed medical taxonomy. Each active row is a pipe-separated canonical term followed by grammatical or surface variants. These catalogs are the terminology/keyword source for generation and optional keyword persistence.

Patterns

Each category has a positive and negative pattern file. Patterns are project authored/curated contextual templates; they determine whether generated text is labeled positive or negative. Catalog membership alone does not set polarity.

Subjects

Subjects.txt is required to contain exactly:

He
She
They
Member

SQL schema used by training export

The export reads dbo.TrainingItem joined to dbo.Category and writes:

{
  "id": 123,
  "category": "MEDICAL_TREATMENT",
  "is_positive": true,
  "value": "He is receiving physical therapy."
}

Validation requires exactly those fields, a unique positive integer ID, one of the 12 category names, a real JSON Boolean, and a non-empty string value.

Split data

create_splits.py groups by (category, is_positive), shuffles each group with a stable derived seed, allocates rounded 10% validation and 10% development counts, and sends the remainder to training. It then independently shuffles each whole split and verifies uniqueness, disjointness, and complete assignment.

Small strata can have materially different proportions or no training row after rounding. The script guarantees partition integrity, not minimum representation in every split.

SFT data

build_sft.py overwrites each split with:

{
  "id": 123,
  "messages": [
    {"role": "system", "content": "...strict JSON policy..."},
    {"role": "user", "content": "He is receiving physical therapy."},
    {"role": "assistant", "content": "{\"is_medical_phi\":true,...}"}
  ]
}

For positives, the expected result contains the source category and uses the entire input for both text and evidence. For negatives, results is empty. This means the current task is whole-input contextual classification rather than precise entity-span extraction.

The conversion is destructive and non-idempotent. Recreate source-shaped splits before rerunning it.

Data lineage

Owner-curated NIH/WHO/CDC-derived catalogs
        + project-authored polarity patterns
        + subject and tense decks
                    |
                    v
       normalized generated TrainingItem rows
                    |
                    v
              data/source.jsonl
                    |
                    v
  source-shaped train/validation/development JSONL
                    |
                    v
      SFT-shaped train/validation/development JSONL
                    |
                    v
     adapter, predictions, metrics, merged model

Privacy and governance

The checked-in source is synthetic/generated according to the system design, but that assumption should be verified operationally. Do not import real patient records without a separate privacy, consent, security, retention, and access control review. The model is designed to detect sensitive medical context; it is not itself a compliance determination or a substitute for human review.