# Testing and validation ## .NET automated tests Run: ```powershell dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx ``` The xUnit suite covers: | Test area | Coverage | |---|---| | LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming | | Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output | | Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs | | Endpoint behavior | Generation endpoint waits for actual generation and commit | | Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs | | Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity | | Provider isolation | SQL Server production DI and SQLite-only test setup | Tests use SQLite for integration persistence while explicitly verifying that the production service registers SQL Server. ## Python verification stages There is no Python unit-test suite. Verification is operational: 1. `validate_source.py` validates exported schema and taxonomy. 2. `create_splits.py` checks uniqueness, disjointness, and complete assignment. 3. `build_sft.py` checks message shape and parses generated assistant JSON. 4. `model_smoke_test.py` loads the base model and rejects non-finite logits. 5. `train_pilot.py` verifies a small QLoRA training path. 6. `evaluate.py` validates generated JSON/schema and computes metrics. 7. `verify_generation_stop.py` checks exact JSON parsing on five examples. 8. `merge_model.py` verifies merged weights and configuration files. ## Current tracked evaluation snapshot `outputs/development-evaluation/summary.json` currently reports: | Metric | Value | |---|---:| | Development records | 2,027 | | Valid JSON | 100% | | Valid schema | 100% | | Classification accuracy | 99.852% | | True positives / false negatives | 996 / 0 | | True negatives / false positives | 1,028 / 3 | | Precision | 99.700% | | Recall | 100% | | F1 | 99.850% | | Entity category accuracy | 98.394% | | Entity text accuracy | 100% | | Evidence accuracy | 100% | These values describe the checked-in development artifact, not an independent benchmark or prospective production guarantee. ## Recommended acceptance gates Before publishing a new model: - all .NET tests pass; - source validation and split integrity pass; - the pilot finishes with finite loss; - full training selects/saves a valid adapter; - JSON and schema validity remain 100%; - classification/category metrics meet a pre-agreed threshold; - false positives and false negatives are manually reviewed; - generation-stop verification passes; - merged-model loading/inference is added and passes; - provenance and license manifest is complete; - a Presidio integration test is run in the downstream repository; - checksums, configuration, Git revision, base model revision, and environment versions are archived. ## Test gaps - No direct Python unit tests for schema, splitting edge cases, SFT conversion, JSON stopping, scoring, or merge logic. - No inference smoke test for the merged model. - No tracked GGUF conversion/quantization verification. - No end-to-end test from generator API through SQL export to training data. - No checked-in Presidio integration or category mapping test. - No fixed golden tests for the four representative contextual sentences. Adding those contextual examples as regression tests would make the intended positive/negative distinction executable rather than documentary.