Go to file
GAJ3T 193ca094b5 added mmd and pdf 2026-10-15 10:38:33 -04:00
.github/workflows moved to this repo 2026-10-13 19:21:52 -04:00
docs added mmd and pdf 2026-10-15 10:38:33 -04:00
migrations moved to this repo 2026-10-13 19:21:52 -04:00
src moved to this repo 2026-10-13 19:21:52 -04:00
tests moved to this repo 2026-10-13 19:21:52 -04:00
.dockerignore moved to this repo 2026-10-13 19:21:52 -04:00
.env.example moved to this repo 2026-10-13 19:21:52 -04:00
.gitignore moved to this repo 2026-10-13 19:21:52 -04:00
Containerfile moved to this repo 2026-10-13 19:21:52 -04:00
README.md docs updated 2026-10-15 10:20:38 -04:00
alembic.ini moved to this repo 2026-10-13 19:21:52 -04:00
compose.yaml moved to this repo 2026-10-13 19:21:52 -04:00
pyproject.toml moved to this repo 2026-10-13 19:21:52 -04:00

README.md

KnowledgeGraph GraphRAG

An internal, restartable GraphRAG pipeline that normalizes externally collected content, extracts and reconciles knowledge, builds a graph, indexes graph and source evidence with PostgreSQL/PgVector, retrieves graph-aware context, and generates source-cited answers.

The collection process is outside this project. Existing Web-Access MCP and Data-Store MCP processes discover and persist source material. This project starts by validating and normalizing their versioned handoff.

Architecture

Web-Access MCP + Data-Store MCP
        |
        | versioned JSONL or PostgreSQL handoff
        v
01 normalize and parse
        v
02 clean, deduplicate, and chunk
        v
03 define knowledge model
        v
04 extract entities and relationships
        v
05 reconcile aliases and duplicates
        v
06 materialize graph nodes and edges
        v
07 detect and summarize communities
        v
08 build PgVector HNSW retrieval index
        v
09 retrieve vector + graph + source context
        v
10 generate a cited answer

PostgreSQL is the canonical checkpoint store. Every completed build stage persists its output, so later stages can be rebuilt without repeating valid upstream work. PgVector stores embeddings and performs server-side HNSW cosine search. Indexed relational edge tables provide bounded graph expansion.

The deployment consists of:

  • PostgreSQL 16 with PgVector
  • Alembic migration job
  • Internal FastAPI service
  • PostgreSQL-backed asynchronous worker
  • Existing BFF as the policy enforcement point

The API is internal only and is not intended for direct customer access.

Repository layout

Path Purpose
src/01_normalize_content.py through src/10_generate_answer.py Complete numbered stage implementations
src/graphrag_platform Configuration, models, storage, orchestration, API, auth, worker, and provider clients
migrations Alembic database migrations
tests Unit, API/worker, pipeline, migration, and real PgVector integration tests
compose.yaml Reproducible PostgreSQL, migration, API, and worker environment
Containerfile Non-root application image
.env.example Runtime configuration template

Detailed architecture, API, runbook, configuration, data model, testing, and editable C4 diagrams are available in docs/.

Prerequisites

  • Python 3.11 or newer
  • Podman 5 or newer
  • podman machine running on Windows
  • An OpenAI-compatible chat and embeddings endpoint
  • External Web-Access MCP/Data-Store MCP collection process

Quick start with Compose

Create the local environment:

Copy-Item .env.example .env

Edit .env and set at minimum:

POSTGRES_PASSWORD=choose-a-local-password
DATABASE_URL=postgresql+psycopg://knowledgegraph_app:choose-a-local-password@localhost:5432/knowledgegraph

LLM_BASE_URL=https://api.openai.com/v1
LLM_API_KEY=replace-with-provider-key
LLM_CHAT_MODEL=gpt-4.1-mini
LLM_EMBEDDING_MODEL=text-embedding-3-small
LLM_EMBEDDING_DIMENSIONS=1536

INTERNAL_JWT_SECRET=replace-with-at-least-32-random-bytes

POSTGRES_PASSWORD and the password in DATABASE_URL must match. Compose replaces the database host with postgres inside application containers.

Start the complete environment:

podman machine start
podman compose up --detach --build

Compose:

  1. Starts docker.io/pgvector/pgvector:pg16.
  2. Waits for PostgreSQL readiness.
  3. Runs alembic upgrade head.
  4. Starts the API and worker.
  5. Waits for API dependency readiness.

Verify it:

podman compose ps
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready

View logs:

podman compose logs --follow api worker

Stop services while retaining PostgreSQL data:

podman compose down

Delete services and all local database data:

podman compose down --volumes

Volume removal is destructive.

External source handoff

JSON Lines contract

Each UTF-8 JSONL line must satisfy schema version 1.0:

{
  "schema_version": "1.0",
  "external_id": "stable-source-record-id",
  "corpus": "product-knowledge",
  "url": "https://docs.example.com/article",
  "body": "<html><body><main>Article text</main></body></html>",
  "content_type": "text/html",
  "status_code": 200,
  "retrieved_at": "2026-10-13T18:00:00Z",
  "metadata": {
    "collector": "Web-Access",
    "stored_by": "Data-Store"
  }
}

Required fields:

  • schema_version: currently 1.0
  • external_id: stable and unique within a corpus
  • url
  • body
  • retrieved_at: ISO 8601 timestamp with offset

Optional/defaulted fields:

  • corpus: when provided, it must match the selected runtime corpus
  • content_type: defaults to text/html
  • status_code: defaults to 200
  • metadata: defaults to an empty object

Unknown fields and unsupported schema versions are rejected. Invalid records include the file and line number in the error.

The template is src/templates/mcp_documents.jsonl. A larger acceptance corpus is tests/fixtures/external_documents.jsonl.

PostgreSQL source contract

Set SOURCE_DATABASE_URL and SOURCE_TABLE_NAME to read the same contract from an external PostgreSQL table:

SOURCE_DATABASE_URL=postgresql+psycopg://source_reader:password@source-host:5432/source_database
SOURCE_TABLE_NAME=external_documents

Compatible table:

CREATE TABLE external_documents (
    external_id text PRIMARY KEY,
    schema_version text NOT NULL DEFAULT '1.0',
    corpus text,
    url text NOT NULL,
    body text NOT NULL,
    content_type text NOT NULL DEFAULT 'text/html',
    status_code integer NOT NULL DEFAULT 200,
    retrieved_at timestamptz NOT NULL,
    metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);

Additional source-table columns are ignored. The application account should receive read-only access to the source table.

Provide either --input-jsonl or SOURCE_DATABASE_URL, not both.

Durable checkpoints

Build stages persist these boundaries:

raw_documents + documents
    -> chunks
    -> knowledge_models
    -> extractions
    -> resolved_entities + resolved_relationships
    -> nodes + edges
    -> communities
    -> retrieval_items

pipeline_runs and stage_runs separately record:

  • queued/running/succeeded/failed state
  • selected stage range
  • attempt
  • configuration fingerprint
  • caller and correlation ID
  • stage inputs and outputs
  • timestamps and error details

The worker requeues runs abandoned beyond WORKER_STALE_RUN_SECONDS. Set this value above the expected maximum duration of one stage.

PostgreSQL advisory locks allow only one active build for a corpus. If another worker claims a job for that corpus, the job is returned to the queue instead of executing conflicting writes.

DataFrames are not stage checkpoints. A Pandas or Polars DataFrame may be used by a future adapter for one bounded transformation, but every completed result must be persisted before the next stage starts.

Database migrations

Runtime services verify the migrated schema and do not silently mutate it.

Apply migrations:

alembic upgrade head

Inspect the current revision:

alembic current
alembic history

Create a migration after changing SQLAlchemy models:

alembic revision --autogenerate -m "describe schema change"

Review generated migrations before applying them. Compose runs upgrades through the one-shot migrate service before starting API and worker services.

AUTO_CREATE_SCHEMA=true exists only for isolated tests. Do not enable it in shared dev, test, or production environments.

Run stages directly

Install a local editable environment:

py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[test]"

Run migrations before the first command:

alembic upgrade head

1. Normalize external content

python .\src\01_normalize_content.py `
  --corpus product-knowledge `
  --input-jsonl .\artifacts\mcp-documents.jsonl

This validates the external handoff, stores raw provenance, runs Trafilatura for HTML, and persists normalized document records.

2. Clean, deduplicate, and chunk

python .\src\02_clean_content.py --corpus product-knowledge

3. Define the knowledge model

python .\src\03_define_knowledge_model.py `
  --corpus product-knowledge `
  --model-file .\src\templates\knowledge_model.json

4. Extract entities and relationships

python .\src\04_extract_knowledge.py --corpus product-knowledge

LLM calls occur outside long database transactions. Each validated chunk result is committed independently.

5. Reconcile knowledge

python .\src\05_reconcile_knowledge.py --corpus product-knowledge

6. Build graph nodes and edges

python .\src\06_build_knowledge_graph.py --corpus product-knowledge

7. Build communities

python .\src\07_build_communities.py --corpus product-knowledge

8. Build the PgVector index

python .\src\08_build_retrieval_index.py --corpus product-knowledge

This embeds chunks, nodes, edges, and community summaries, validates dimensions, stores vectors, and creates a dimension-specific HNSW cosine index.

9. Retrieve GraphRAG context

python .\src\09_query_graphrag.py `
  --corpus product-knowledge `
  --question "Who created Atlas?" `
  --top-k 8

The result contains vector context, expanded graph_context, and traceable sources.

10. Generate a cited answer

python .\src\10_generate_answer.py `
  --corpus product-knowledge `
  --question "Who created Atlas?"

Run a durable build

Run stages 1 through 8 under one durable pipeline record:

knowledgegraph run `
  --corpus product-knowledge `
  --from-stage 1 `
  --through-stage 8 `
  --input-jsonl .\artifacts\mcp-documents.jsonl `
  --model-file .\src\templates\knowledge_model.json `
  --requested-by local-developer

Use --from-stage to rebuild only downstream state. The --reset option deletes the selected stage and downstream derived corpus data before execution.

BFF authorization contract

The existing BFF authenticates callers and evaluates the KnowledgeGraph feature. Independent permission bits are:

Read    = 1
Add     = 2
Edit    = 4
Delete  = 8
Execute = 16
Operation Required mask
Read job/source/retrieval information Read (1)
Submit a stage-1 source job `Add
Run downstream build stages Execute (16)
Query or answer `Read
Retry a failed job Execute (16)

The BFF forwards a signed internal JWT. Required claims:

{
  "iss": "knowledgegraph-bff",
  "aud": "knowledgegraph-service",
  "sub": "internal-caller-id",
  "iat": 1791930000,
  "exp": 1791930600,
  "jti": "correlation-or-request-id",
  "features": {
    "KnowledgeGraph": 31
  }
}

Dev/test uses HS256 and INTERNAL_JWT_SECRET. Production deployments can replace this verifier with the BFF's workload identity, mTLS, or asymmetric/JWKS contract. Never trust unsigned permission headers.

Internal API

Endpoint Purpose
GET /internal/knowledge-graph/health Process liveness
GET /internal/knowledge-graph/ready Database and PgVector readiness
POST /internal/knowledge-graph/jobs Queue stages 1–8
GET /internal/knowledge-graph/jobs/{jobId} Read durable job/stage status
POST /internal/knowledge-graph/jobs/{jobId}/retry Requeue a failed job
POST /internal/knowledge-graph/query Synchronous bounded retrieval
POST /internal/knowledge-graph/answer Synchronous cited answer

Example job request:

{
  "corpus": "product-knowledge",
  "from_stage": 1,
  "through_stage": 8,
  "input_jsonl": "/app/artifacts/mcp-documents.jsonl",
  "model_file": "/app/src/templates/knowledge_model.json",
  "reset": false
}

Compose bind-mounts local artifacts at /app/artifacts, allowing the worker to read JSONL handoffs placed there. In deployed environments, prefer the PostgreSQL source contract or a shared managed file volume.

Long-running stages return 202 Accepted with a job ID. Query and answer requests remain synchronous and bounded.

Retry and logging behavior

Model-provider calls retry:

  • HTTP 429
  • HTTP 500, 502, 503, and 504
  • connection failures
  • timeouts

Retries honor numeric Retry-After, otherwise use bounded exponential backoff with jitter. Authentication and validation errors are not retried.

Logs are emitted as one JSON object per line and include event names, run IDs, corpus/stage context where available, attempts, status, and error details. Secrets are not logged.

Tests

Portable tests:

pytest --ignore=.\tests\integration\test_pgvector_postgres.py

Real PostgreSQL/PgVector migration and retrieval test:

$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
pytest .\tests\integration\test_pgvector_postgres.py
Remove-Item Env:TEST_POSTGRES_URL

The suite covers:

  • Versioned source validation
  • Nested metadata and quoted/comma-containing content
  • Configuration parsing
  • Chunk validation
  • Cross-chunk relationship reconciliation
  • Provider retry policy
  • Full ten-stage behavior
  • Signed BFF authorization and permission failures
  • Durable API job and worker execution
  • Alembic migration against PostgreSQL
  • PgVector extension and HNSW index creation
  • Server-side vector retrieval, graph expansion, and citations

Opt-in vector load test:

$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
$env:RUN_PERFORMANCE_TESTS = "1"
$env:PERF_VECTOR_ITEM_COUNT = "10000"
$env:PERF_MAX_QUERY_SECONDS = "2.0"
pytest .\tests\performance\test_vector_query_load.py

Set the item count and latency threshold to the capacity target for the dev/test environment. Performance checks are excluded by default because machine resources vary.

CI is defined in .github/workflows/ci.yml and compiles, tests, validates Compose, exercises real PgVector, and builds the application image.

Persistent format policy

  • PostgreSQL is the canonical operational store.
  • JSON Lines is the preferred file handoff/checkpoint format.
  • CSV is allowed only for explicitly defined flat import/export views.

CSV safely supports commas, quotes, and line breaks when written by a compliant CSV library. It remains unsuitable as the canonical graph format because aliases, metadata, source ID arrays, embeddings, and graph properties are nested and typed. Never construct CSV by joining strings with commas.

RAG database alternatives

PostgreSQL with PgVector

Implemented default. Choose it for SQL transactions, colocated provenance and vectors, self-managed deployment, and tunable HNSW retrieval. Graph traversal uses indexed relational edges rather than a native property-graph language.

Official reference: PgVector.

Azure Cosmos DB for NoSQL

Choose it for managed/serverless document storage, vector indexing, and first-party vector plus BM25 hybrid search. Cosmos DB's Gremlin API is separate from the NoSQL vector feature; use document adjacency or a separate graph store.

References:

Neo4j

Choose it when graph topology, paths, and multi-hop traversal are primary retrieval signals. Neo4j combines native property graphs, Cypher, vector indexes, and graph-aware hybrid retrieval. It requires a dedicated backend adapter; changing DATABASE_URL is not sufficient. Confirm version and edition requirements for the selected vector and unified-search features.

References:

Other credible options:

Operational guidance

  • Keep the API and worker on internal networks accessible only to the BFF and operators.
  • Replace all dev secrets before shared deployment.
  • Use a separate PostgreSQL role for migrations and a least-privilege runtime role.
  • Back up PostgreSQL before destructive resets or migrations.
  • Monitor queued/running/failed run counts, stage duration, LLM retries, database pool usage, vector latency, and retrieval result counts.
  • Set WORKER_STALE_RUN_SECONDS above the longest expected single stage.
  • Rebuild stage 8 after changing embedding model or dimensions.
  • Maintain a golden question/citation set to detect retrieval-quality regressions.

Troubleshooting

Database migrations are required

alembic current
alembic upgrade head

LLM_API_KEY is required

Set LLM_API_KEY in .env and restart API/worker containers.

Embedding dimension mismatch

Set LLM_EMBEDDING_DIMENSIONS to the exact provider output dimension, then reset and rerun stage 8.

Source handoff rejected

Validate every JSONL line against schema version 1.0. Confirm corpus, external_id, URL, body, and retrieved_at.

Job remains queued

podman compose ps worker
podman compose logs --tail 100 worker

PostgreSQL/PgVector is not ready

podman compose ps postgres
podman compose logs --tail 100 postgres migrate

Port conflict

Change POSTGRES_PORT or API_PORT in .env, then recreate the Compose stack.