18 KiB
KnowledgeGraph GraphRAG
An internal, restartable GraphRAG pipeline that normalizes externally collected content, extracts and reconciles knowledge, builds a graph, indexes graph and source evidence with PostgreSQL/PgVector, retrieves graph-aware context, and generates source-cited answers.
The collection process is outside this project. Existing Web-Access MCP and Data-Store MCP processes discover and persist source material. This project starts by validating and normalizing their versioned handoff.
Architecture
Web-Access MCP + Data-Store MCP
|
| versioned JSONL or PostgreSQL handoff
v
01 normalize and parse
v
02 clean, deduplicate, and chunk
v
03 define knowledge model
v
04 extract entities and relationships
v
05 reconcile aliases and duplicates
v
06 materialize graph nodes and edges
v
07 detect and summarize communities
v
08 build PgVector HNSW retrieval index
v
09 retrieve vector + graph + source context
v
10 generate a cited answer
PostgreSQL is the canonical checkpoint store. Every completed build stage persists its output, so later stages can be rebuilt without repeating valid upstream work. PgVector stores embeddings and performs server-side HNSW cosine search. Indexed relational edge tables provide bounded graph expansion.
The deployment consists of:
- PostgreSQL 16 with PgVector
- Alembic migration job
- Internal FastAPI service
- PostgreSQL-backed asynchronous worker
- Existing BFF as the policy enforcement point
The API is internal only and is not intended for direct customer access.
Repository layout
| Path | Purpose |
|---|---|
src/01_normalize_content.py through src/10_generate_answer.py |
Complete numbered stage implementations |
src/graphrag_platform |
Configuration, models, storage, orchestration, API, auth, worker, and provider clients |
migrations |
Alembic database migrations |
tests |
Unit, API/worker, pipeline, migration, and real PgVector integration tests |
compose.yaml |
Reproducible PostgreSQL, migration, API, and worker environment |
Containerfile |
Non-root application image |
.env.example |
Runtime configuration template |
Detailed architecture, API, runbook, configuration, data model, testing, and
editable C4 diagrams are available in docs/.
Prerequisites
- Python 3.11 or newer
- Podman 5 or newer
podman machinerunning on Windows- An OpenAI-compatible chat and embeddings endpoint
- External Web-Access MCP/Data-Store MCP collection process
Quick start with Compose
Create the local environment:
Copy-Item .env.example .env
Edit .env and set at minimum:
POSTGRES_PASSWORD=choose-a-local-password
DATABASE_URL=postgresql+psycopg://knowledgegraph_app:choose-a-local-password@localhost:5432/knowledgegraph
LLM_BASE_URL=https://api.openai.com/v1
LLM_API_KEY=replace-with-provider-key
LLM_CHAT_MODEL=gpt-4.1-mini
LLM_EMBEDDING_MODEL=text-embedding-3-small
LLM_EMBEDDING_DIMENSIONS=1536
INTERNAL_JWT_SECRET=replace-with-at-least-32-random-bytes
POSTGRES_PASSWORD and the password in DATABASE_URL must match. Compose replaces
the database host with postgres inside application containers.
Start the complete environment:
podman machine start
podman compose up --detach --build
Compose:
- Starts
docker.io/pgvector/pgvector:pg16. - Waits for PostgreSQL readiness.
- Runs
alembic upgrade head. - Starts the API and worker.
- Waits for API dependency readiness.
Verify it:
podman compose ps
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready
View logs:
podman compose logs --follow api worker
Stop services while retaining PostgreSQL data:
podman compose down
Delete services and all local database data:
podman compose down --volumes
Volume removal is destructive.
External source handoff
JSON Lines contract
Each UTF-8 JSONL line must satisfy schema version 1.0:
{
"schema_version": "1.0",
"external_id": "stable-source-record-id",
"corpus": "product-knowledge",
"url": "https://docs.example.com/article",
"body": "<html><body><main>Article text</main></body></html>",
"content_type": "text/html",
"status_code": 200,
"retrieved_at": "2026-10-13T18:00:00Z",
"metadata": {
"collector": "Web-Access",
"stored_by": "Data-Store"
}
}
Required fields:
schema_version: currently1.0external_id: stable and unique within a corpusurlbodyretrieved_at: ISO 8601 timestamp with offset
Optional/defaulted fields:
corpus: when provided, it must match the selected runtime corpuscontent_type: defaults totext/htmlstatus_code: defaults to200metadata: defaults to an empty object
Unknown fields and unsupported schema versions are rejected. Invalid records include the file and line number in the error.
The template is src/templates/mcp_documents.jsonl.
A larger acceptance corpus is
tests/fixtures/external_documents.jsonl.
PostgreSQL source contract
Set SOURCE_DATABASE_URL and SOURCE_TABLE_NAME to read the same contract from an
external PostgreSQL table:
SOURCE_DATABASE_URL=postgresql+psycopg://source_reader:password@source-host:5432/source_database
SOURCE_TABLE_NAME=external_documents
Compatible table:
CREATE TABLE external_documents (
external_id text PRIMARY KEY,
schema_version text NOT NULL DEFAULT '1.0',
corpus text,
url text NOT NULL,
body text NOT NULL,
content_type text NOT NULL DEFAULT 'text/html',
status_code integer NOT NULL DEFAULT 200,
retrieved_at timestamptz NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);
Additional source-table columns are ignored. The application account should receive read-only access to the source table.
Provide either --input-jsonl or SOURCE_DATABASE_URL, not both.
Durable checkpoints
Build stages persist these boundaries:
raw_documents + documents
-> chunks
-> knowledge_models
-> extractions
-> resolved_entities + resolved_relationships
-> nodes + edges
-> communities
-> retrieval_items
pipeline_runs and stage_runs separately record:
- queued/running/succeeded/failed state
- selected stage range
- attempt
- configuration fingerprint
- caller and correlation ID
- stage inputs and outputs
- timestamps and error details
The worker requeues runs abandoned beyond WORKER_STALE_RUN_SECONDS. Set this value
above the expected maximum duration of one stage.
PostgreSQL advisory locks allow only one active build for a corpus. If another worker claims a job for that corpus, the job is returned to the queue instead of executing conflicting writes.
DataFrames are not stage checkpoints. A Pandas or Polars DataFrame may be used by a future adapter for one bounded transformation, but every completed result must be persisted before the next stage starts.
Database migrations
Runtime services verify the migrated schema and do not silently mutate it.
Apply migrations:
alembic upgrade head
Inspect the current revision:
alembic current
alembic history
Create a migration after changing SQLAlchemy models:
alembic revision --autogenerate -m "describe schema change"
Review generated migrations before applying them. Compose runs upgrades through the
one-shot migrate service before starting API and worker services.
AUTO_CREATE_SCHEMA=true exists only for isolated tests. Do not enable it in shared
dev, test, or production environments.
Run stages directly
Install a local editable environment:
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[test]"
Run migrations before the first command:
alembic upgrade head
1. Normalize external content
python .\src\01_normalize_content.py `
--corpus product-knowledge `
--input-jsonl .\artifacts\mcp-documents.jsonl
This validates the external handoff, stores raw provenance, runs Trafilatura for HTML, and persists normalized document records.
2. Clean, deduplicate, and chunk
python .\src\02_clean_content.py --corpus product-knowledge
3. Define the knowledge model
python .\src\03_define_knowledge_model.py `
--corpus product-knowledge `
--model-file .\src\templates\knowledge_model.json
4. Extract entities and relationships
python .\src\04_extract_knowledge.py --corpus product-knowledge
LLM calls occur outside long database transactions. Each validated chunk result is committed independently.
5. Reconcile knowledge
python .\src\05_reconcile_knowledge.py --corpus product-knowledge
6. Build graph nodes and edges
python .\src\06_build_knowledge_graph.py --corpus product-knowledge
7. Build communities
python .\src\07_build_communities.py --corpus product-knowledge
8. Build the PgVector index
python .\src\08_build_retrieval_index.py --corpus product-knowledge
This embeds chunks, nodes, edges, and community summaries, validates dimensions, stores vectors, and creates a dimension-specific HNSW cosine index.
9. Retrieve GraphRAG context
python .\src\09_query_graphrag.py `
--corpus product-knowledge `
--question "Who created Atlas?" `
--top-k 8
The result contains vector context, expanded graph_context, and traceable
sources.
10. Generate a cited answer
python .\src\10_generate_answer.py `
--corpus product-knowledge `
--question "Who created Atlas?"
Run a durable build
Run stages 1 through 8 under one durable pipeline record:
knowledgegraph run `
--corpus product-knowledge `
--from-stage 1 `
--through-stage 8 `
--input-jsonl .\artifacts\mcp-documents.jsonl `
--model-file .\src\templates\knowledge_model.json `
--requested-by local-developer
Use --from-stage to rebuild only downstream state. The --reset option deletes
the selected stage and downstream derived corpus data before execution.
BFF authorization contract
The existing BFF authenticates callers and evaluates the KnowledgeGraph feature.
Independent permission bits are:
Read = 1
Add = 2
Edit = 4
Delete = 8
Execute = 16
| Operation | Required mask |
|---|---|
| Read job/source/retrieval information | Read (1) |
| Submit a stage-1 source job | `Add |
| Run downstream build stages | Execute (16) |
| Query or answer | `Read |
| Retry a failed job | Execute (16) |
The BFF forwards a signed internal JWT. Required claims:
{
"iss": "knowledgegraph-bff",
"aud": "knowledgegraph-service",
"sub": "internal-caller-id",
"iat": 1791930000,
"exp": 1791930600,
"jti": "correlation-or-request-id",
"features": {
"KnowledgeGraph": 31
}
}
Dev/test uses HS256 and INTERNAL_JWT_SECRET. Production deployments can replace
this verifier with the BFF's workload identity, mTLS, or asymmetric/JWKS contract.
Never trust unsigned permission headers.
Internal API
| Endpoint | Purpose |
|---|---|
GET /internal/knowledge-graph/health |
Process liveness |
GET /internal/knowledge-graph/ready |
Database and PgVector readiness |
POST /internal/knowledge-graph/jobs |
Queue stages 1–8 |
GET /internal/knowledge-graph/jobs/{jobId} |
Read durable job/stage status |
POST /internal/knowledge-graph/jobs/{jobId}/retry |
Requeue a failed job |
POST /internal/knowledge-graph/query |
Synchronous bounded retrieval |
POST /internal/knowledge-graph/answer |
Synchronous cited answer |
Example job request:
{
"corpus": "product-knowledge",
"from_stage": 1,
"through_stage": 8,
"input_jsonl": "/app/artifacts/mcp-documents.jsonl",
"model_file": "/app/src/templates/knowledge_model.json",
"reset": false
}
Compose bind-mounts local artifacts at /app/artifacts, allowing the
worker to read JSONL handoffs placed there. In deployed environments, prefer the
PostgreSQL source contract or a shared managed file volume.
Long-running stages return 202 Accepted with a job ID. Query and answer requests
remain synchronous and bounded.
Retry and logging behavior
Model-provider calls retry:
- HTTP 429
- HTTP 500, 502, 503, and 504
- connection failures
- timeouts
Retries honor numeric Retry-After, otherwise use bounded exponential backoff with
jitter. Authentication and validation errors are not retried.
Logs are emitted as one JSON object per line and include event names, run IDs, corpus/stage context where available, attempts, status, and error details. Secrets are not logged.
Tests
Portable tests:
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
Real PostgreSQL/PgVector migration and retrieval test:
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
pytest .\tests\integration\test_pgvector_postgres.py
Remove-Item Env:TEST_POSTGRES_URL
The suite covers:
- Versioned source validation
- Nested metadata and quoted/comma-containing content
- Configuration parsing
- Chunk validation
- Cross-chunk relationship reconciliation
- Provider retry policy
- Full ten-stage behavior
- Signed BFF authorization and permission failures
- Durable API job and worker execution
- Alembic migration against PostgreSQL
- PgVector extension and HNSW index creation
- Server-side vector retrieval, graph expansion, and citations
Opt-in vector load test:
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
$env:RUN_PERFORMANCE_TESTS = "1"
$env:PERF_VECTOR_ITEM_COUNT = "10000"
$env:PERF_MAX_QUERY_SECONDS = "2.0"
pytest .\tests\performance\test_vector_query_load.py
Set the item count and latency threshold to the capacity target for the dev/test environment. Performance checks are excluded by default because machine resources vary.
CI is defined in .github/workflows/ci.yml and compiles,
tests, validates Compose, exercises real PgVector, and builds the application image.
Persistent format policy
- PostgreSQL is the canonical operational store.
- JSON Lines is the preferred file handoff/checkpoint format.
- CSV is allowed only for explicitly defined flat import/export views.
CSV safely supports commas, quotes, and line breaks when written by a compliant CSV library. It remains unsuitable as the canonical graph format because aliases, metadata, source ID arrays, embeddings, and graph properties are nested and typed. Never construct CSV by joining strings with commas.
RAG database alternatives
PostgreSQL with PgVector
Implemented default. Choose it for SQL transactions, colocated provenance and vectors, self-managed deployment, and tunable HNSW retrieval. Graph traversal uses indexed relational edges rather than a native property-graph language.
Official reference: PgVector.
Azure Cosmos DB for NoSQL
Choose it for managed/serverless document storage, vector indexing, and first-party vector plus BM25 hybrid search. Cosmos DB's Gremlin API is separate from the NoSQL vector feature; use document adjacency or a separate graph store.
References:
Neo4j
Choose it when graph topology, paths, and multi-hop traversal are primary retrieval
signals. Neo4j combines native property graphs, Cypher, vector indexes, and
graph-aware hybrid retrieval. It requires a dedicated backend adapter; changing
DATABASE_URL is not sufficient. Confirm version and edition requirements for the
selected vector and unified-search features.
References:
Other credible options:
- Amazon Neptune Analytics for AWS-managed graph/vector analytics
- Azure AI Search as a secondary managed hybrid index without a native graph model
Operational guidance
- Keep the API and worker on internal networks accessible only to the BFF and operators.
- Replace all dev secrets before shared deployment.
- Use a separate PostgreSQL role for migrations and a least-privilege runtime role.
- Back up PostgreSQL before destructive resets or migrations.
- Monitor queued/running/failed run counts, stage duration, LLM retries, database pool usage, vector latency, and retrieval result counts.
- Set
WORKER_STALE_RUN_SECONDSabove the longest expected single stage. - Rebuild stage 8 after changing embedding model or dimensions.
- Maintain a golden question/citation set to detect retrieval-quality regressions.
Troubleshooting
Database migrations are required
alembic current
alembic upgrade head
LLM_API_KEY is required
Set LLM_API_KEY in .env and restart API/worker containers.
Embedding dimension mismatch
Set LLM_EMBEDDING_DIMENSIONS to the exact provider output dimension, then reset
and rerun stage 8.
Source handoff rejected
Validate every JSONL line against schema version 1.0. Confirm corpus,
external_id, URL, body, and retrieved_at.
Job remains queued
podman compose ps worker
podman compose logs --tail 100 worker
PostgreSQL/PgVector is not ready
podman compose ps postgres
podman compose logs --tail 100 postgres migrate
Port conflict
Change POSTGRES_PORT or API_PORT in .env, then recreate the Compose stack.