2.1 KiB
External source contract
JSON Lines
Each UTF-8 line must be a JSON object with schema version 1.0.
{
"schema_version": "1.0",
"external_id": "stable-source-record-id",
"corpus": "product-knowledge",
"url": "https://docs.example.com/article",
"body": "<html><body><main>Article text</main></body></html>",
"content_type": "text/html",
"status_code": 200,
"retrieved_at": "2026-10-13T18:00:00Z",
"metadata": {
"collector": "Web-Access",
"stored_by": "Data-Store"
}
}
Required fields:
schema_version: exactly1.0;external_id: stable and unique within a corpus;url;body;retrieved_at: ISO 8601 timestamp with an offset.
Defaulted fields:
content_type:text/html;status_code:200;metadata:{}.
corpus, when provided, must equal the selected runtime corpus. Unknown fields
and unsupported schema versions are rejected. Validation errors include the
file and line number. See the canonical template at
src/templates/mcp_documents.jsonl and
the acceptance fixture at tests/fixtures/external_documents.jsonl.
PostgreSQL source table
Set SOURCE_DATABASE_URL and SOURCE_TABLE_NAME to use a PostgreSQL handoff.
The table must contain:
CREATE TABLE external_documents (
external_id text PRIMARY KEY,
schema_version text NOT NULL DEFAULT '1.0',
corpus text,
url text NOT NULL,
body text NOT NULL,
content_type text NOT NULL DEFAULT 'text/html',
status_code integer NOT NULL DEFAULT 200,
retrieved_at timestamptz NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);
Additional columns are ignored. Give the application account read-only access.
Use either --input-jsonl or SOURCE_DATABASE_URL, not both.
Normalization behavior
Stage 1 stores raw provenance and a normalized document. HTML is extracted with Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes whitespace, detects duplicate content by SHA-256 hash, and creates overlapping chunks.