# External source contract ## JSON Lines Each UTF-8 line must be a JSON object with schema version `1.0`. ```json { "schema_version": "1.0", "external_id": "stable-source-record-id", "corpus": "product-knowledge", "url": "https://docs.example.com/article", "body": "
Article text
", "content_type": "text/html", "status_code": 200, "retrieved_at": "2026-10-13T18:00:00Z", "metadata": { "collector": "Web-Access", "stored_by": "Data-Store" } } ``` Required fields: - `schema_version`: exactly `1.0`; - `external_id`: stable and unique within a corpus; - `url`; - `body`; - `retrieved_at`: ISO 8601 timestamp with an offset. Defaulted fields: - `content_type`: `text/html`; - `status_code`: `200`; - `metadata`: `{}`. `corpus`, when provided, must equal the selected runtime corpus. Unknown fields and unsupported schema versions are rejected. Validation errors include the file and line number. See the canonical template at [`src/templates/mcp_documents.jsonl`](../src/templates/mcp_documents.jsonl) and the acceptance fixture at [`tests/fixtures/external_documents.jsonl`](../tests/fixtures/external_documents.jsonl). ## PostgreSQL source table Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to use a PostgreSQL handoff. The table must contain: ```sql CREATE TABLE external_documents ( external_id text PRIMARY KEY, schema_version text NOT NULL DEFAULT '1.0', corpus text, url text NOT NULL, body text NOT NULL, content_type text NOT NULL DEFAULT 'text/html', status_code integer NOT NULL DEFAULT 200, retrieved_at timestamptz NOT NULL, metadata jsonb NOT NULL DEFAULT '{}'::jsonb ); ``` Additional columns are ignored. Give the application account read-only access. Use either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both. ## Normalization behavior Stage 1 stores raw provenance and a normalized document. HTML is extracted with Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes whitespace, detects duplicate content by SHA-256 hash, and creates overlapping chunks.