Models-Phi-8B/docs/testing-and-validation.md

3.5 KiB

Testing and validation

.NET automated tests

Run:

dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx

The xUnit suite covers:

Test area Coverage
LM output validation Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming
Committed batch generation Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output
Resume safety Changed inputs/config, durable count/ordinal checks, completed and incomplete runs
Endpoint behavior Generation endpoint waits for actual generation and commit
Throughput logging Rates, concurrency, ETA, cancellation, failures, resumed runs
Keyword catalog EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity
Provider isolation SQL Server production DI and SQLite-only test setup

Tests use SQLite for integration persistence while explicitly verifying that the production service registers SQL Server.

Python verification stages

There is no Python unit-test suite. Verification is operational:

  1. validate_source.py validates exported schema and taxonomy.
  2. create_splits.py checks uniqueness, disjointness, and complete assignment.
  3. build_sft.py checks message shape and parses generated assistant JSON.
  4. model_smoke_test.py loads the base model and rejects non-finite logits.
  5. train_pilot.py verifies a small QLoRA training path.
  6. evaluate.py validates generated JSON/schema and computes metrics.
  7. verify_generation_stop.py checks exact JSON parsing on five examples.
  8. merge_model.py verifies merged weights and configuration files.

Current tracked evaluation snapshot

outputs/development-evaluation/summary.json currently reports:

Metric Value
Development records 2,027
Valid JSON 100%
Valid schema 100%
Classification accuracy 99.852%
True positives / false negatives 996 / 0
True negatives / false positives 1,028 / 3
Precision 99.700%
Recall 100%
F1 99.850%
Entity category accuracy 98.394%
Entity text accuracy 100%
Evidence accuracy 100%

These values describe the checked-in development artifact, not an independent benchmark or prospective production guarantee.

Before publishing a new model:

  • all .NET tests pass;
  • source validation and split integrity pass;
  • the pilot finishes with finite loss;
  • full training selects/saves a valid adapter;
  • JSON and schema validity remain 100%;
  • classification/category metrics meet a pre-agreed threshold;
  • false positives and false negatives are manually reviewed;
  • generation-stop verification passes;
  • merged-model loading/inference is added and passes;
  • provenance and license manifest is complete;
  • a Presidio integration test is run in the downstream repository;
  • checksums, configuration, Git revision, base model revision, and environment versions are archived.

Test gaps

  • No direct Python unit tests for schema, splitting edge cases, SFT conversion, JSON stopping, scoring, or merge logic.
  • No inference smoke test for the merged model.
  • No tracked GGUF conversion/quantization verification.
  • No end-to-end test from generator API through SQL export to training data.
  • No checked-in Presidio integration or category mapping test.
  • No fixed golden tests for the four representative contextual sentences.

Adding those contextual examples as regression tests would make the intended positive/negative distinction executable rather than documentary.