PHI-Model-8B/docs/testing-and-validation.md

91 lines
3.5 KiB
Markdown

# Testing and validation
## .NET automated tests
Run:
```powershell
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
```
The xUnit suite covers:
| Test area | Coverage |
|---|---|
| LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming |
| Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output |
| Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs |
| Endpoint behavior | Generation endpoint waits for actual generation and commit |
| Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs |
| Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity |
| Provider isolation | SQL Server production DI and SQLite-only test setup |
Tests use SQLite for integration persistence while explicitly verifying that the
production service registers SQL Server.
## Python verification stages
There is no Python unit-test suite. Verification is operational:
1. `validate_source.py` validates exported schema and taxonomy.
2. `create_splits.py` checks uniqueness, disjointness, and complete assignment.
3. `build_sft.py` checks message shape and parses generated assistant JSON.
4. `model_smoke_test.py` loads the base model and rejects non-finite logits.
5. `train_pilot.py` verifies a small QLoRA training path.
6. `evaluate.py` validates generated JSON/schema and computes metrics.
7. `verify_generation_stop.py` checks exact JSON parsing on five examples.
8. `merge_model.py` verifies merged weights and configuration files.
## Current tracked evaluation snapshot
`outputs/development-evaluation/summary.json` currently reports:
| Metric | Value |
|---|---:|
| Development records | 2,027 |
| Valid JSON | 100% |
| Valid schema | 100% |
| Classification accuracy | 99.852% |
| True positives / false negatives | 996 / 0 |
| True negatives / false positives | 1,028 / 3 |
| Precision | 99.700% |
| Recall | 100% |
| F1 | 99.850% |
| Entity category accuracy | 98.394% |
| Entity text accuracy | 100% |
| Evidence accuracy | 100% |
These values describe the checked-in development artifact, not an independent
benchmark or prospective production guarantee.
## Recommended acceptance gates
Before publishing a new model:
- all .NET tests pass;
- source validation and split integrity pass;
- the pilot finishes with finite loss;
- full training selects/saves a valid adapter;
- JSON and schema validity remain 100%;
- classification/category metrics meet a pre-agreed threshold;
- false positives and false negatives are manually reviewed;
- generation-stop verification passes;
- merged-model loading/inference is added and passes;
- provenance and license manifest is complete;
- a Presidio integration test is run in the downstream repository;
- checksums, configuration, Git revision, base model revision, and environment
versions are archived.
## Test gaps
- No direct Python unit tests for schema, splitting edge cases, SFT conversion,
JSON stopping, scoring, or merge logic.
- No inference smoke test for the merged model.
- No tracked GGUF conversion/quantization verification.
- No end-to-end test from generator API through SQL export to training data.
- No checked-in Presidio integration or category mapping test.
- No fixed golden tests for the four representative contextual sentences.
Adding those contextual examples as regression tests would make the intended
positive/negative distinction executable rather than documentary.