RAG-DEMO/docs/source-contract.md

73 lines
2.1 KiB
Markdown

# External source contract
## JSON Lines
Each UTF-8 line must be a JSON object with schema version `1.0`.
```json
{
"schema_version": "1.0",
"external_id": "stable-source-record-id",
"corpus": "product-knowledge",
"url": "https://docs.example.com/article",
"body": "<html><body><main>Article text</main></body></html>",
"content_type": "text/html",
"status_code": 200,
"retrieved_at": "2026-10-13T18:00:00Z",
"metadata": {
"collector": "Web-Access",
"stored_by": "Data-Store"
}
}
```
Required fields:
- `schema_version`: exactly `1.0`;
- `external_id`: stable and unique within a corpus;
- `url`;
- `body`;
- `retrieved_at`: ISO 8601 timestamp with an offset.
Defaulted fields:
- `content_type`: `text/html`;
- `status_code`: `200`;
- `metadata`: `{}`.
`corpus`, when provided, must equal the selected runtime corpus. Unknown fields
and unsupported schema versions are rejected. Validation errors include the
file and line number. See the canonical template at
[`src/templates/mcp_documents.jsonl`](../src/templates/mcp_documents.jsonl) and
the acceptance fixture at [`tests/fixtures/external_documents.jsonl`](../tests/fixtures/external_documents.jsonl).
## PostgreSQL source table
Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to use a PostgreSQL handoff.
The table must contain:
```sql
CREATE TABLE external_documents (
external_id text PRIMARY KEY,
schema_version text NOT NULL DEFAULT '1.0',
corpus text,
url text NOT NULL,
body text NOT NULL,
content_type text NOT NULL DEFAULT 'text/html',
status_code integer NOT NULL DEFAULT 200,
retrieved_at timestamptz NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);
```
Additional columns are ignored. Give the application account read-only access.
Use either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both.
## Normalization behavior
Stage 1 stores raw provenance and a normalized document. HTML is extracted with
Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes
whitespace, detects duplicate content by SHA-256 hash, and creates overlapping
chunks.