RAG-DEMO/docs/source-contract.md

2.1 KiB

External source contract

JSON Lines

Each UTF-8 line must be a JSON object with schema version 1.0.

{
  "schema_version": "1.0",
  "external_id": "stable-source-record-id",
  "corpus": "product-knowledge",
  "url": "https://docs.example.com/article",
  "body": "<html><body><main>Article text</main></body></html>",
  "content_type": "text/html",
  "status_code": 200,
  "retrieved_at": "2026-10-13T18:00:00Z",
  "metadata": {
    "collector": "Web-Access",
    "stored_by": "Data-Store"
  }
}

Required fields:

  • schema_version: exactly 1.0;
  • external_id: stable and unique within a corpus;
  • url;
  • body;
  • retrieved_at: ISO 8601 timestamp with an offset.

Defaulted fields:

  • content_type: text/html;
  • status_code: 200;
  • metadata: {}.

corpus, when provided, must equal the selected runtime corpus. Unknown fields and unsupported schema versions are rejected. Validation errors include the file and line number. See the canonical template at src/templates/mcp_documents.jsonl and the acceptance fixture at tests/fixtures/external_documents.jsonl.

PostgreSQL source table

Set SOURCE_DATABASE_URL and SOURCE_TABLE_NAME to use a PostgreSQL handoff. The table must contain:

CREATE TABLE external_documents (
    external_id text PRIMARY KEY,
    schema_version text NOT NULL DEFAULT '1.0',
    corpus text,
    url text NOT NULL,
    body text NOT NULL,
    content_type text NOT NULL DEFAULT 'text/html',
    status_code integer NOT NULL DEFAULT 200,
    retrieved_at timestamptz NOT NULL,
    metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);

Additional columns are ignored. Give the application account read-only access. Use either --input-jsonl or SOURCE_DATABASE_URL, not both.

Normalization behavior

Stage 1 stores raw provenance and a normalized document. HTML is extracted with Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes whitespace, detects duplicate content by SHA-256 hash, and creates overlapping chunks.