# KnowledgeGraph GraphRAG An internal, restartable GraphRAG pipeline that normalizes externally collected content, extracts and reconciles knowledge, builds a graph, indexes graph and source evidence with PostgreSQL/PgVector, retrieves graph-aware context, and generates source-cited answers. The collection process is outside this project. Existing **Web-Access MCP** and **Data-Store MCP** processes discover and persist source material. This project starts by validating and normalizing their versioned handoff. ## Architecture ```text Web-Access MCP + Data-Store MCP | | versioned JSONL or PostgreSQL handoff v 01 normalize and parse v 02 clean, deduplicate, and chunk v 03 define knowledge model v 04 extract entities and relationships v 05 reconcile aliases and duplicates v 06 materialize graph nodes and edges v 07 detect and summarize communities v 08 build PgVector HNSW retrieval index v 09 retrieve vector + graph + source context v 10 generate a cited answer ``` PostgreSQL is the canonical checkpoint store. Every completed build stage persists its output, so later stages can be rebuilt without repeating valid upstream work. PgVector stores embeddings and performs server-side HNSW cosine search. Indexed relational edge tables provide bounded graph expansion. The deployment consists of: - PostgreSQL 16 with PgVector - Alembic migration job - Internal FastAPI service - PostgreSQL-backed asynchronous worker - Existing BFF as the policy enforcement point The API is internal only and is not intended for direct customer access. ## Repository layout | Path | Purpose | |---|---| | [`src/01_normalize_content.py`](src/01_normalize_content.py) through [`src/10_generate_answer.py`](src/10_generate_answer.py) | Complete numbered stage implementations | | [`src/graphrag_platform`](src/graphrag_platform) | Configuration, models, storage, orchestration, API, auth, worker, and provider clients | | [`migrations`](migrations) | Alembic database migrations | | [`tests`](tests) | Unit, API/worker, pipeline, migration, and real PgVector integration tests | | [`compose.yaml`](compose.yaml) | Reproducible PostgreSQL, migration, API, and worker environment | | [`Containerfile`](Containerfile) | Non-root application image | | [`.env.example`](.env.example) | Runtime configuration template | ## Prerequisites - Python 3.11 or newer - Podman 5 or newer - `podman machine` running on Windows - An OpenAI-compatible chat and embeddings endpoint - External Web-Access MCP/Data-Store MCP collection process ## Quick start with Compose Create the local environment: ```powershell Copy-Item .env.example .env ``` Edit `.env` and set at minimum: ```dotenv POSTGRES_PASSWORD=choose-a-local-password DATABASE_URL=postgresql+psycopg://knowledgegraph_app:choose-a-local-password@localhost:5432/knowledgegraph LLM_BASE_URL=https://api.openai.com/v1 LLM_API_KEY=replace-with-provider-key LLM_CHAT_MODEL=gpt-4.1-mini LLM_EMBEDDING_MODEL=text-embedding-3-small LLM_EMBEDDING_DIMENSIONS=1536 INTERNAL_JWT_SECRET=replace-with-at-least-32-random-bytes ``` `POSTGRES_PASSWORD` and the password in `DATABASE_URL` must match. Compose replaces the database host with `postgres` inside application containers. Start the complete environment: ```powershell podman machine start podman compose up --detach --build ``` Compose: 1. Starts `docker.io/pgvector/pgvector:pg16`. 2. Waits for PostgreSQL readiness. 3. Runs `alembic upgrade head`. 4. Starts the API and worker. 5. Waits for API dependency readiness. Verify it: ```powershell podman compose ps Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready ``` View logs: ```powershell podman compose logs --follow api worker ``` Stop services while retaining PostgreSQL data: ```powershell podman compose down ``` Delete services and all local database data: ```powershell podman compose down --volumes ``` Volume removal is destructive. ## External source handoff ### JSON Lines contract Each UTF-8 JSONL line must satisfy schema version `1.0`: ```json { "schema_version": "1.0", "external_id": "stable-source-record-id", "corpus": "product-knowledge", "url": "https://docs.example.com/article", "body": "
Article text
", "content_type": "text/html", "status_code": 200, "retrieved_at": "2026-10-13T18:00:00Z", "metadata": { "collector": "Web-Access", "stored_by": "Data-Store" } } ``` Required fields: - `schema_version`: currently `1.0` - `external_id`: stable and unique within a corpus - `url` - `body` - `retrieved_at`: ISO 8601 timestamp with offset Optional/defaulted fields: - `corpus`: when provided, it must match the selected runtime corpus - `content_type`: defaults to `text/html` - `status_code`: defaults to `200` - `metadata`: defaults to an empty object Unknown fields and unsupported schema versions are rejected. Invalid records include the file and line number in the error. The template is [`src/templates/mcp_documents.jsonl`](src/templates/mcp_documents.jsonl). A larger acceptance corpus is [`tests/fixtures/external_documents.jsonl`](tests/fixtures/external_documents.jsonl). ### PostgreSQL source contract Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to read the same contract from an external PostgreSQL table: ```dotenv SOURCE_DATABASE_URL=postgresql+psycopg://source_reader:password@source-host:5432/source_database SOURCE_TABLE_NAME=external_documents ``` Compatible table: ```sql CREATE TABLE external_documents ( external_id text PRIMARY KEY, schema_version text NOT NULL DEFAULT '1.0', corpus text, url text NOT NULL, body text NOT NULL, content_type text NOT NULL DEFAULT 'text/html', status_code integer NOT NULL DEFAULT 200, retrieved_at timestamptz NOT NULL, metadata jsonb NOT NULL DEFAULT '{}'::jsonb ); ``` Additional source-table columns are ignored. The application account should receive read-only access to the source table. Provide either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both. ## Durable checkpoints Build stages persist these boundaries: ```text raw_documents + documents -> chunks -> knowledge_models -> extractions -> resolved_entities + resolved_relationships -> nodes + edges -> communities -> retrieval_items ``` `pipeline_runs` and `stage_runs` separately record: - queued/running/succeeded/failed state - selected stage range - attempt - configuration fingerprint - caller and correlation ID - stage inputs and outputs - timestamps and error details The worker requeues runs abandoned beyond `WORKER_STALE_RUN_SECONDS`. Set this value above the expected maximum duration of one stage. PostgreSQL advisory locks allow only one active build for a corpus. If another worker claims a job for that corpus, the job is returned to the queue instead of executing conflicting writes. DataFrames are not stage checkpoints. A Pandas or Polars DataFrame may be used by a future adapter for one bounded transformation, but every completed result must be persisted before the next stage starts. ## Database migrations Runtime services verify the migrated schema and do not silently mutate it. Apply migrations: ```powershell alembic upgrade head ``` Inspect the current revision: ```powershell alembic current alembic history ``` Create a migration after changing SQLAlchemy models: ```powershell alembic revision --autogenerate -m "describe schema change" ``` Review generated migrations before applying them. Compose runs upgrades through the one-shot `migrate` service before starting API and worker services. `AUTO_CREATE_SCHEMA=true` exists only for isolated tests. Do not enable it in shared dev, test, or production environments. ## Run stages directly Install a local editable environment: ```powershell py -3.12 -m venv .venv .\.venv\Scripts\Activate.ps1 python -m pip install --upgrade pip python -m pip install -e ".[test]" ``` Run migrations before the first command: ```powershell alembic upgrade head ``` ### 1. Normalize external content ```powershell python .\src\01_normalize_content.py ` --corpus product-knowledge ` --input-jsonl .\artifacts\mcp-documents.jsonl ``` This validates the external handoff, stores raw provenance, runs Trafilatura for HTML, and persists normalized document records. ### 2. Clean, deduplicate, and chunk ```powershell python .\src\02_clean_content.py --corpus product-knowledge ``` ### 3. Define the knowledge model ```powershell python .\src\03_define_knowledge_model.py ` --corpus product-knowledge ` --model-file .\src\templates\knowledge_model.json ``` ### 4. Extract entities and relationships ```powershell python .\src\04_extract_knowledge.py --corpus product-knowledge ``` LLM calls occur outside long database transactions. Each validated chunk result is committed independently. ### 5. Reconcile knowledge ```powershell python .\src\05_reconcile_knowledge.py --corpus product-knowledge ``` ### 6. Build graph nodes and edges ```powershell python .\src\06_build_knowledge_graph.py --corpus product-knowledge ``` ### 7. Build communities ```powershell python .\src\07_build_communities.py --corpus product-knowledge ``` ### 8. Build the PgVector index ```powershell python .\src\08_build_retrieval_index.py --corpus product-knowledge ``` This embeds chunks, nodes, edges, and community summaries, validates dimensions, stores vectors, and creates a dimension-specific HNSW cosine index. ### 9. Retrieve GraphRAG context ```powershell python .\src\09_query_graphrag.py ` --corpus product-knowledge ` --question "Who created Atlas?" ` --top-k 8 ``` The result contains vector `context`, expanded `graph_context`, and traceable `sources`. ### 10. Generate a cited answer ```powershell python .\src\10_generate_answer.py ` --corpus product-knowledge ` --question "Who created Atlas?" ``` ## Run a durable build Run stages 1 through 8 under one durable pipeline record: ```powershell knowledgegraph run ` --corpus product-knowledge ` --from-stage 1 ` --through-stage 8 ` --input-jsonl .\artifacts\mcp-documents.jsonl ` --model-file .\src\templates\knowledge_model.json ` --requested-by local-developer ``` Use `--from-stage` to rebuild only downstream state. The `--reset` option deletes the selected stage and downstream derived corpus data before execution. ## BFF authorization contract The existing BFF authenticates callers and evaluates the `KnowledgeGraph` feature. Independent permission bits are: ```text Read = 1 Add = 2 Edit = 4 Delete = 8 Execute = 16 ``` | Operation | Required mask | |---|---:| | Read job/source/retrieval information | `Read` (`1`) | | Submit a stage-1 source job | `Add | Execute` (`18`) | | Run downstream build stages | `Execute` (`16`) | | Query or answer | `Read | Execute` (`17`) | | Retry a failed job | `Execute` (`16`) | The BFF forwards a signed internal JWT. Required claims: ```json { "iss": "knowledgegraph-bff", "aud": "knowledgegraph-service", "sub": "internal-caller-id", "iat": 1791930000, "exp": 1791930600, "jti": "correlation-or-request-id", "features": { "KnowledgeGraph": 31 } } ``` Dev/test uses HS256 and `INTERNAL_JWT_SECRET`. Production deployments can replace this verifier with the BFF's workload identity, mTLS, or asymmetric/JWKS contract. Never trust unsigned permission headers. ## Internal API | Endpoint | Purpose | |---|---| | `GET /internal/knowledge-graph/health` | Process liveness | | `GET /internal/knowledge-graph/ready` | Database and PgVector readiness | | `POST /internal/knowledge-graph/jobs` | Queue stages 1–8 | | `GET /internal/knowledge-graph/jobs/{jobId}` | Read durable job/stage status | | `POST /internal/knowledge-graph/jobs/{jobId}/retry` | Requeue a failed job | | `POST /internal/knowledge-graph/query` | Synchronous bounded retrieval | | `POST /internal/knowledge-graph/answer` | Synchronous cited answer | Example job request: ```json { "corpus": "product-knowledge", "from_stage": 1, "through_stage": 8, "input_jsonl": "/app/artifacts/mcp-documents.jsonl", "model_file": "/app/src/templates/knowledge_model.json", "reset": false } ``` Compose bind-mounts local [`artifacts`](artifacts) at `/app/artifacts`, allowing the worker to read JSONL handoffs placed there. In deployed environments, prefer the PostgreSQL source contract or a shared managed file volume. Long-running stages return `202 Accepted` with a job ID. Query and answer requests remain synchronous and bounded. ## Retry and logging behavior Model-provider calls retry: - HTTP 429 - HTTP 500, 502, 503, and 504 - connection failures - timeouts Retries honor numeric `Retry-After`, otherwise use bounded exponential backoff with jitter. Authentication and validation errors are not retried. Logs are emitted as one JSON object per line and include event names, run IDs, corpus/stage context where available, attempts, status, and error details. Secrets are not logged. ## Tests Portable tests: ```powershell pytest --ignore=.\tests\integration\test_pgvector_postgres.py ``` Real PostgreSQL/PgVector migration and retrieval test: ```powershell $env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph" pytest .\tests\integration\test_pgvector_postgres.py Remove-Item Env:TEST_POSTGRES_URL ``` The suite covers: - Versioned source validation - Nested metadata and quoted/comma-containing content - Configuration parsing - Chunk validation - Cross-chunk relationship reconciliation - Provider retry policy - Full ten-stage behavior - Signed BFF authorization and permission failures - Durable API job and worker execution - Alembic migration against PostgreSQL - PgVector extension and HNSW index creation - Server-side vector retrieval, graph expansion, and citations Opt-in vector load test: ```powershell $env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph" $env:RUN_PERFORMANCE_TESTS = "1" $env:PERF_VECTOR_ITEM_COUNT = "10000" $env:PERF_MAX_QUERY_SECONDS = "2.0" pytest .\tests\performance\test_vector_query_load.py ``` Set the item count and latency threshold to the capacity target for the dev/test environment. Performance checks are excluded by default because machine resources vary. CI is defined in [`.github/workflows/ci.yml`](.github/workflows/ci.yml) and compiles, tests, validates Compose, exercises real PgVector, and builds the application image. ## Persistent format policy - PostgreSQL is the canonical operational store. - JSON Lines is the preferred file handoff/checkpoint format. - CSV is allowed only for explicitly defined flat import/export views. CSV safely supports commas, quotes, and line breaks when written by a compliant CSV library. It remains unsuitable as the canonical graph format because aliases, metadata, source ID arrays, embeddings, and graph properties are nested and typed. Never construct CSV by joining strings with commas. ## RAG database alternatives ### PostgreSQL with PgVector Implemented default. Choose it for SQL transactions, colocated provenance and vectors, self-managed deployment, and tunable HNSW retrieval. Graph traversal uses indexed relational edges rather than a native property-graph language. Official reference: [PgVector](https://github.com/pgvector/pgvector). ### Azure Cosmos DB for NoSQL Choose it for managed/serverless document storage, vector indexing, and first-party vector plus BM25 hybrid search. Cosmos DB's Gremlin API is separate from the NoSQL vector feature; use document adjacency or a separate graph store. References: - [Cosmos DB vector search](https://learn.microsoft.com/azure/cosmos-db/vector-search) - [Cosmos DB hybrid search](https://learn.microsoft.com/azure/cosmos-db/gen-ai/hybrid-search) - [Cosmos DB Gremlin API](https://learn.microsoft.com/azure/cosmos-db/gremlin/overview) ### Neo4j Choose it when graph topology, paths, and multi-hop traversal are primary retrieval signals. Neo4j combines native property graphs, Cypher, vector indexes, and graph-aware hybrid retrieval. It requires a dedicated backend adapter; changing `DATABASE_URL` is not sufficient. Confirm version and edition requirements for the selected vector and unified-search features. References: - [Neo4j vector indexes](https://neo4j.com/docs/cypher-manual/current/indexes/search-indexes/vector-indexes/) - [Neo4j hybrid search](https://neo4j.com/developer/genai-ecosystem/hybrid-search/) Other credible options: - [Amazon Neptune Analytics](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html) for AWS-managed graph/vector analytics - [Azure AI Search](https://learn.microsoft.com/azure/search/hybrid-search-overview) as a secondary managed hybrid index without a native graph model ## Operational guidance - Keep the API and worker on internal networks accessible only to the BFF and operators. - Replace all dev secrets before shared deployment. - Use a separate PostgreSQL role for migrations and a least-privilege runtime role. - Back up PostgreSQL before destructive resets or migrations. - Monitor queued/running/failed run counts, stage duration, LLM retries, database pool usage, vector latency, and retrieval result counts. - Set `WORKER_STALE_RUN_SECONDS` above the longest expected single stage. - Rebuild stage 8 after changing embedding model or dimensions. - Maintain a golden question/citation set to detect retrieval-quality regressions. ## Troubleshooting ### Database migrations are required ```powershell alembic current alembic upgrade head ``` ### `LLM_API_KEY is required` Set `LLM_API_KEY` in `.env` and restart API/worker containers. ### Embedding dimension mismatch Set `LLM_EMBEDDING_DIMENSIONS` to the exact provider output dimension, then reset and rerun stage 8. ### Source handoff rejected Validate every JSONL line against schema version `1.0`. Confirm `corpus`, `external_id`, URL, body, and `retrieved_at`. ### Job remains queued ```powershell podman compose ps worker podman compose logs --tail 100 worker ``` ### PostgreSQL/PgVector is not ready ```powershell podman compose ps postgres podman compose logs --tail 100 postgres migrate ``` ### Port conflict Change `POSTGRES_PORT` or `API_PORT` in `.env`, then recreate the Compose stack.