73 lines
2.1 KiB
Markdown
73 lines
2.1 KiB
Markdown
# External source contract
|
|
|
|
## JSON Lines
|
|
|
|
Each UTF-8 line must be a JSON object with schema version `1.0`.
|
|
|
|
```json
|
|
{
|
|
"schema_version": "1.0",
|
|
"external_id": "stable-source-record-id",
|
|
"corpus": "product-knowledge",
|
|
"url": "https://docs.example.com/article",
|
|
"body": "<html><body><main>Article text</main></body></html>",
|
|
"content_type": "text/html",
|
|
"status_code": 200,
|
|
"retrieved_at": "2026-10-13T18:00:00Z",
|
|
"metadata": {
|
|
"collector": "Web-Access",
|
|
"stored_by": "Data-Store"
|
|
}
|
|
}
|
|
```
|
|
|
|
Required fields:
|
|
|
|
- `schema_version`: exactly `1.0`;
|
|
- `external_id`: stable and unique within a corpus;
|
|
- `url`;
|
|
- `body`;
|
|
- `retrieved_at`: ISO 8601 timestamp with an offset.
|
|
|
|
Defaulted fields:
|
|
|
|
- `content_type`: `text/html`;
|
|
- `status_code`: `200`;
|
|
- `metadata`: `{}`.
|
|
|
|
`corpus`, when provided, must equal the selected runtime corpus. Unknown fields
|
|
and unsupported schema versions are rejected. Validation errors include the
|
|
file and line number. See the canonical template at
|
|
[`src/templates/mcp_documents.jsonl`](../src/templates/mcp_documents.jsonl) and
|
|
the acceptance fixture at [`tests/fixtures/external_documents.jsonl`](../tests/fixtures/external_documents.jsonl).
|
|
|
|
## PostgreSQL source table
|
|
|
|
Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to use a PostgreSQL handoff.
|
|
The table must contain:
|
|
|
|
```sql
|
|
CREATE TABLE external_documents (
|
|
external_id text PRIMARY KEY,
|
|
schema_version text NOT NULL DEFAULT '1.0',
|
|
corpus text,
|
|
url text NOT NULL,
|
|
body text NOT NULL,
|
|
content_type text NOT NULL DEFAULT 'text/html',
|
|
status_code integer NOT NULL DEFAULT 200,
|
|
retrieved_at timestamptz NOT NULL,
|
|
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
|
|
);
|
|
```
|
|
|
|
Additional columns are ignored. Give the application account read-only access.
|
|
Use either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both.
|
|
|
|
## Normalization behavior
|
|
|
|
Stage 1 stores raw provenance and a normalized document. HTML is extracted with
|
|
Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes
|
|
whitespace, detects duplicate content by SHA-256 hash, and creates overlapping
|
|
chunks.
|
|
|