91 lines
3.5 KiB
Markdown
91 lines
3.5 KiB
Markdown
# Testing and validation
|
|
|
|
## .NET automated tests
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
dotnet test .\PHI.Data.Generator\RLHF.Generator.slnx
|
|
```
|
|
|
|
The xUnit suite covers:
|
|
|
|
| Test area | Coverage |
|
|
|---|---|
|
|
| LM output validation | Empty/multiline/wrapped/labeled/unresolved output, case behavior, trimming |
|
|
| Committed batch generation | Ordering, batch visibility, rollback, timeout, cancellation, invalid LM output |
|
|
| Resume safety | Changed inputs/config, durable count/ordinal checks, completed and incomplete runs |
|
|
| Endpoint behavior | Generation endpoint waits for actual generation and commit |
|
|
| Throughput logging | Rates, concurrency, ETA, cancellation, failures, resumed runs |
|
|
| Keyword catalog | EF constraints, loading, idempotency, dry-run/apply backfill, ambiguity |
|
|
| Provider isolation | SQL Server production DI and SQLite-only test setup |
|
|
|
|
Tests use SQLite for integration persistence while explicitly verifying that the
|
|
production service registers SQL Server.
|
|
|
|
## Python verification stages
|
|
|
|
There is no Python unit-test suite. Verification is operational:
|
|
|
|
1. `validate_source.py` validates exported schema and taxonomy.
|
|
2. `create_splits.py` checks uniqueness, disjointness, and complete assignment.
|
|
3. `build_sft.py` checks message shape and parses generated assistant JSON.
|
|
4. `model_smoke_test.py` loads the base model and rejects non-finite logits.
|
|
5. `train_pilot.py` verifies a small QLoRA training path.
|
|
6. `evaluate.py` validates generated JSON/schema and computes metrics.
|
|
7. `verify_generation_stop.py` checks exact JSON parsing on five examples.
|
|
8. `merge_model.py` verifies merged weights and configuration files.
|
|
|
|
## Current tracked evaluation snapshot
|
|
|
|
`outputs/development-evaluation/summary.json` currently reports:
|
|
|
|
| Metric | Value |
|
|
|---|---:|
|
|
| Development records | 2,027 |
|
|
| Valid JSON | 100% |
|
|
| Valid schema | 100% |
|
|
| Classification accuracy | 99.852% |
|
|
| True positives / false negatives | 996 / 0 |
|
|
| True negatives / false positives | 1,028 / 3 |
|
|
| Precision | 99.700% |
|
|
| Recall | 100% |
|
|
| F1 | 99.850% |
|
|
| Entity category accuracy | 98.394% |
|
|
| Entity text accuracy | 100% |
|
|
| Evidence accuracy | 100% |
|
|
|
|
These values describe the checked-in development artifact, not an independent
|
|
benchmark or prospective production guarantee.
|
|
|
|
## Recommended acceptance gates
|
|
|
|
Before publishing a new model:
|
|
|
|
- all .NET tests pass;
|
|
- source validation and split integrity pass;
|
|
- the pilot finishes with finite loss;
|
|
- full training selects/saves a valid adapter;
|
|
- JSON and schema validity remain 100%;
|
|
- classification/category metrics meet a pre-agreed threshold;
|
|
- false positives and false negatives are manually reviewed;
|
|
- generation-stop verification passes;
|
|
- merged-model loading/inference is added and passes;
|
|
- provenance and license manifest is complete;
|
|
- a Presidio integration test is run in the downstream repository;
|
|
- checksums, configuration, Git revision, base model revision, and environment
|
|
versions are archived.
|
|
|
|
## Test gaps
|
|
|
|
- No direct Python unit tests for schema, splitting edge cases, SFT conversion,
|
|
JSON stopping, scoring, or merge logic.
|
|
- No inference smoke test for the merged model.
|
|
- No tracked GGUF conversion/quantization verification.
|
|
- No end-to-end test from generator API through SQL export to training data.
|
|
- No checked-in Presidio integration or category mapping test.
|
|
- No fixed golden tests for the four representative contextual sentences.
|
|
|
|
Adding those contextual examples as regression tests would make the intended
|
|
positive/negative distinction executable rather than documentary.
|