3.5 KiB
3.5 KiB
Testing and validation
.NET automated tests
Run:
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
The xUnit suite covers:
| Test area | Coverage |
|---|---|
| LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming |
| Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output |
| Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs |
| Endpoint behavior | Generation endpoint waits for actual generation and commit |
| Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs |
| Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity |
| Provider isolation | SQL Server production DI and SQLite-only test setup |
Tests use SQLite for integration persistence while explicitly verifying that the production service registers SQL Server.
Python verification stages
There is no Python unit-test suite. Verification is operational:
validate_source.pyvalidates exported schema and taxonomy.create_splits.pychecks uniqueness, disjointness, and complete assignment.build_sft.pychecks message shape and parses generated assistant JSON.model_smoke_test.pyloads the base model and rejects non-finite logits.train_pilot.pyverifies a small QLoRA training path.evaluate.pyvalidates generated JSON/schema and computes metrics.verify_generation_stop.pychecks exact JSON parsing on five examples.merge_model.pyverifies merged weights and configuration files.
Current tracked evaluation snapshot
outputs/development-evaluation/summary.json currently reports:
| Metric | Value |
|---|---|
| Development records | 2,027 |
| Valid JSON | 100% |
| Valid schema | 100% |
| Classification accuracy | 99.852% |
| True positives / false negatives | 996 / 0 |
| True negatives / false positives | 1,028 / 3 |
| Precision | 99.700% |
| Recall | 100% |
| F1 | 99.850% |
| Entity category accuracy | 98.394% |
| Entity text accuracy | 100% |
| Evidence accuracy | 100% |
These values describe the checked-in development artifact, not an independent benchmark or prospective production guarantee.
Recommended acceptance gates
Before publishing a new model:
- all .NET tests pass;
- source validation and split integrity pass;
- the pilot finishes with finite loss;
- full training selects/saves a valid adapter;
- JSON and schema validity remain 100%;
- classification/category metrics meet a pre-agreed threshold;
- false positives and false negatives are manually reviewed;
- generation-stop verification passes;
- merged-model loading/inference is added and passes;
- provenance and license manifest is complete;
- a Presidio integration test is run in the downstream repository;
- checksums, configuration, Git revision, base model revision, and environment versions are archived.
Test gaps
- No direct Python unit tests for schema, splitting edge cases, SFT conversion, JSON stopping, scoring, or merge logic.
- No inference smoke test for the merged model.
- No tracked GGUF conversion/quantization verification.
- No end-to-end test from generator API through SQL export to training data.
- No checked-in Presidio integration or category mapping test.
- No fixed golden tests for the four representative contextual sentences.
Adding those contextual examples as regression tests would make the intended positive/negative distinction executable rather than documentary.