646 lines
18 KiB
Markdown
646 lines
18 KiB
Markdown
# KnowledgeGraph GraphRAG
|
||
|
||
An internal, restartable GraphRAG pipeline that normalizes externally collected
|
||
content, extracts and reconciles knowledge, builds a graph, indexes graph and source
|
||
evidence with PostgreSQL/PgVector, retrieves graph-aware context, and generates
|
||
source-cited answers.
|
||
|
||
The collection process is outside this project. Existing **Web-Access MCP** and
|
||
**Data-Store MCP** processes discover and persist source material. This project
|
||
starts by validating and normalizing their versioned handoff.
|
||
|
||
## Architecture
|
||
|
||
```text
|
||
Web-Access MCP + Data-Store MCP
|
||
|
|
||
| versioned JSONL or PostgreSQL handoff
|
||
v
|
||
01 normalize and parse
|
||
v
|
||
02 clean, deduplicate, and chunk
|
||
v
|
||
03 define knowledge model
|
||
v
|
||
04 extract entities and relationships
|
||
v
|
||
05 reconcile aliases and duplicates
|
||
v
|
||
06 materialize graph nodes and edges
|
||
v
|
||
07 detect and summarize communities
|
||
v
|
||
08 build PgVector HNSW retrieval index
|
||
v
|
||
09 retrieve vector + graph + source context
|
||
v
|
||
10 generate a cited answer
|
||
```
|
||
|
||
PostgreSQL is the canonical checkpoint store. Every completed build stage persists
|
||
its output, so later stages can be rebuilt without repeating valid upstream work.
|
||
PgVector stores embeddings and performs server-side HNSW cosine search. Indexed
|
||
relational edge tables provide bounded graph expansion.
|
||
|
||
The deployment consists of:
|
||
|
||
- PostgreSQL 16 with PgVector
|
||
- Alembic migration job
|
||
- Internal FastAPI service
|
||
- PostgreSQL-backed asynchronous worker
|
||
- Existing BFF as the policy enforcement point
|
||
|
||
The API is internal only and is not intended for direct customer access.
|
||
|
||
## Repository layout
|
||
|
||
| Path | Purpose |
|
||
|---|---|
|
||
| [`src/01_normalize_content.py`](src/01_normalize_content.py) through [`src/10_generate_answer.py`](src/10_generate_answer.py) | Complete numbered stage implementations |
|
||
| [`src/graphrag_platform`](src/graphrag_platform) | Configuration, models, storage, orchestration, API, auth, worker, and provider clients |
|
||
| [`migrations`](migrations) | Alembic database migrations |
|
||
| [`tests`](tests) | Unit, API/worker, pipeline, migration, and real PgVector integration tests |
|
||
| [`compose.yaml`](compose.yaml) | Reproducible PostgreSQL, migration, API, and worker environment |
|
||
| [`Containerfile`](Containerfile) | Non-root application image |
|
||
| [`.env.example`](.env.example) | Runtime configuration template |
|
||
|
||
Detailed architecture, API, runbook, configuration, data model, testing, and
|
||
editable C4 diagrams are available in [`docs/`](docs/README.md).
|
||
|
||
## Prerequisites
|
||
|
||
- Python 3.11 or newer
|
||
- Podman 5 or newer
|
||
- `podman machine` running on Windows
|
||
- An OpenAI-compatible chat and embeddings endpoint
|
||
- External Web-Access MCP/Data-Store MCP collection process
|
||
|
||
## Quick start with Compose
|
||
|
||
Create the local environment:
|
||
|
||
```powershell
|
||
Copy-Item .env.example .env
|
||
```
|
||
|
||
Edit `.env` and set at minimum:
|
||
|
||
```dotenv
|
||
POSTGRES_PASSWORD=choose-a-local-password
|
||
DATABASE_URL=postgresql+psycopg://knowledgegraph_app:choose-a-local-password@localhost:5432/knowledgegraph
|
||
|
||
LLM_BASE_URL=https://api.openai.com/v1
|
||
LLM_API_KEY=replace-with-provider-key
|
||
LLM_CHAT_MODEL=gpt-4.1-mini
|
||
LLM_EMBEDDING_MODEL=text-embedding-3-small
|
||
LLM_EMBEDDING_DIMENSIONS=1536
|
||
|
||
INTERNAL_JWT_SECRET=replace-with-at-least-32-random-bytes
|
||
```
|
||
|
||
`POSTGRES_PASSWORD` and the password in `DATABASE_URL` must match. Compose replaces
|
||
the database host with `postgres` inside application containers.
|
||
|
||
Start the complete environment:
|
||
|
||
```powershell
|
||
podman machine start
|
||
podman compose up --detach --build
|
||
```
|
||
|
||
Compose:
|
||
|
||
1. Starts `docker.io/pgvector/pgvector:pg16`.
|
||
2. Waits for PostgreSQL readiness.
|
||
3. Runs `alembic upgrade head`.
|
||
4. Starts the API and worker.
|
||
5. Waits for API dependency readiness.
|
||
|
||
Verify it:
|
||
|
||
```powershell
|
||
podman compose ps
|
||
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health
|
||
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready
|
||
```
|
||
|
||
View logs:
|
||
|
||
```powershell
|
||
podman compose logs --follow api worker
|
||
```
|
||
|
||
Stop services while retaining PostgreSQL data:
|
||
|
||
```powershell
|
||
podman compose down
|
||
```
|
||
|
||
Delete services and all local database data:
|
||
|
||
```powershell
|
||
podman compose down --volumes
|
||
```
|
||
|
||
Volume removal is destructive.
|
||
|
||
## External source handoff
|
||
|
||
### JSON Lines contract
|
||
|
||
Each UTF-8 JSONL line must satisfy schema version `1.0`:
|
||
|
||
```json
|
||
{
|
||
"schema_version": "1.0",
|
||
"external_id": "stable-source-record-id",
|
||
"corpus": "product-knowledge",
|
||
"url": "https://docs.example.com/article",
|
||
"body": "<html><body><main>Article text</main></body></html>",
|
||
"content_type": "text/html",
|
||
"status_code": 200,
|
||
"retrieved_at": "2026-10-13T18:00:00Z",
|
||
"metadata": {
|
||
"collector": "Web-Access",
|
||
"stored_by": "Data-Store"
|
||
}
|
||
}
|
||
```
|
||
|
||
Required fields:
|
||
|
||
- `schema_version`: currently `1.0`
|
||
- `external_id`: stable and unique within a corpus
|
||
- `url`
|
||
- `body`
|
||
- `retrieved_at`: ISO 8601 timestamp with offset
|
||
|
||
Optional/defaulted fields:
|
||
|
||
- `corpus`: when provided, it must match the selected runtime corpus
|
||
- `content_type`: defaults to `text/html`
|
||
- `status_code`: defaults to `200`
|
||
- `metadata`: defaults to an empty object
|
||
|
||
Unknown fields and unsupported schema versions are rejected. Invalid records include
|
||
the file and line number in the error.
|
||
|
||
The template is [`src/templates/mcp_documents.jsonl`](src/templates/mcp_documents.jsonl).
|
||
A larger acceptance corpus is
|
||
[`tests/fixtures/external_documents.jsonl`](tests/fixtures/external_documents.jsonl).
|
||
|
||
### PostgreSQL source contract
|
||
|
||
Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to read the same contract from an
|
||
external PostgreSQL table:
|
||
|
||
```dotenv
|
||
SOURCE_DATABASE_URL=postgresql+psycopg://source_reader:password@source-host:5432/source_database
|
||
SOURCE_TABLE_NAME=external_documents
|
||
```
|
||
|
||
Compatible table:
|
||
|
||
```sql
|
||
CREATE TABLE external_documents (
|
||
external_id text PRIMARY KEY,
|
||
schema_version text NOT NULL DEFAULT '1.0',
|
||
corpus text,
|
||
url text NOT NULL,
|
||
body text NOT NULL,
|
||
content_type text NOT NULL DEFAULT 'text/html',
|
||
status_code integer NOT NULL DEFAULT 200,
|
||
retrieved_at timestamptz NOT NULL,
|
||
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
|
||
);
|
||
```
|
||
|
||
Additional source-table columns are ignored. The application account should receive
|
||
read-only access to the source table.
|
||
|
||
Provide either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both.
|
||
|
||
## Durable checkpoints
|
||
|
||
Build stages persist these boundaries:
|
||
|
||
```text
|
||
raw_documents + documents
|
||
-> chunks
|
||
-> knowledge_models
|
||
-> extractions
|
||
-> resolved_entities + resolved_relationships
|
||
-> nodes + edges
|
||
-> communities
|
||
-> retrieval_items
|
||
```
|
||
|
||
`pipeline_runs` and `stage_runs` separately record:
|
||
|
||
- queued/running/succeeded/failed state
|
||
- selected stage range
|
||
- attempt
|
||
- configuration fingerprint
|
||
- caller and correlation ID
|
||
- stage inputs and outputs
|
||
- timestamps and error details
|
||
|
||
The worker requeues runs abandoned beyond `WORKER_STALE_RUN_SECONDS`. Set this value
|
||
above the expected maximum duration of one stage.
|
||
|
||
PostgreSQL advisory locks allow only one active build for a corpus. If another
|
||
worker claims a job for that corpus, the job is returned to the queue instead of
|
||
executing conflicting writes.
|
||
|
||
DataFrames are not stage checkpoints. A Pandas or Polars DataFrame may be used by a
|
||
future adapter for one bounded transformation, but every completed result must be
|
||
persisted before the next stage starts.
|
||
|
||
## Database migrations
|
||
|
||
Runtime services verify the migrated schema and do not silently mutate it.
|
||
|
||
Apply migrations:
|
||
|
||
```powershell
|
||
alembic upgrade head
|
||
```
|
||
|
||
Inspect the current revision:
|
||
|
||
```powershell
|
||
alembic current
|
||
alembic history
|
||
```
|
||
|
||
Create a migration after changing SQLAlchemy models:
|
||
|
||
```powershell
|
||
alembic revision --autogenerate -m "describe schema change"
|
||
```
|
||
|
||
Review generated migrations before applying them. Compose runs upgrades through the
|
||
one-shot `migrate` service before starting API and worker services.
|
||
|
||
`AUTO_CREATE_SCHEMA=true` exists only for isolated tests. Do not enable it in shared
|
||
dev, test, or production environments.
|
||
|
||
## Run stages directly
|
||
|
||
Install a local editable environment:
|
||
|
||
```powershell
|
||
py -3.12 -m venv .venv
|
||
.\.venv\Scripts\Activate.ps1
|
||
python -m pip install --upgrade pip
|
||
python -m pip install -e ".[test]"
|
||
```
|
||
|
||
Run migrations before the first command:
|
||
|
||
```powershell
|
||
alembic upgrade head
|
||
```
|
||
|
||
### 1. Normalize external content
|
||
|
||
```powershell
|
||
python .\src\01_normalize_content.py `
|
||
--corpus product-knowledge `
|
||
--input-jsonl .\artifacts\mcp-documents.jsonl
|
||
```
|
||
|
||
This validates the external handoff, stores raw provenance, runs Trafilatura for
|
||
HTML, and persists normalized document records.
|
||
|
||
### 2. Clean, deduplicate, and chunk
|
||
|
||
```powershell
|
||
python .\src\02_clean_content.py --corpus product-knowledge
|
||
```
|
||
|
||
### 3. Define the knowledge model
|
||
|
||
```powershell
|
||
python .\src\03_define_knowledge_model.py `
|
||
--corpus product-knowledge `
|
||
--model-file .\src\templates\knowledge_model.json
|
||
```
|
||
|
||
### 4. Extract entities and relationships
|
||
|
||
```powershell
|
||
python .\src\04_extract_knowledge.py --corpus product-knowledge
|
||
```
|
||
|
||
LLM calls occur outside long database transactions. Each validated chunk result is
|
||
committed independently.
|
||
|
||
### 5. Reconcile knowledge
|
||
|
||
```powershell
|
||
python .\src\05_reconcile_knowledge.py --corpus product-knowledge
|
||
```
|
||
|
||
### 6. Build graph nodes and edges
|
||
|
||
```powershell
|
||
python .\src\06_build_knowledge_graph.py --corpus product-knowledge
|
||
```
|
||
|
||
### 7. Build communities
|
||
|
||
```powershell
|
||
python .\src\07_build_communities.py --corpus product-knowledge
|
||
```
|
||
|
||
### 8. Build the PgVector index
|
||
|
||
```powershell
|
||
python .\src\08_build_retrieval_index.py --corpus product-knowledge
|
||
```
|
||
|
||
This embeds chunks, nodes, edges, and community summaries, validates dimensions,
|
||
stores vectors, and creates a dimension-specific HNSW cosine index.
|
||
|
||
### 9. Retrieve GraphRAG context
|
||
|
||
```powershell
|
||
python .\src\09_query_graphrag.py `
|
||
--corpus product-knowledge `
|
||
--question "Who created Atlas?" `
|
||
--top-k 8
|
||
```
|
||
|
||
The result contains vector `context`, expanded `graph_context`, and traceable
|
||
`sources`.
|
||
|
||
### 10. Generate a cited answer
|
||
|
||
```powershell
|
||
python .\src\10_generate_answer.py `
|
||
--corpus product-knowledge `
|
||
--question "Who created Atlas?"
|
||
```
|
||
|
||
## Run a durable build
|
||
|
||
Run stages 1 through 8 under one durable pipeline record:
|
||
|
||
```powershell
|
||
knowledgegraph run `
|
||
--corpus product-knowledge `
|
||
--from-stage 1 `
|
||
--through-stage 8 `
|
||
--input-jsonl .\artifacts\mcp-documents.jsonl `
|
||
--model-file .\src\templates\knowledge_model.json `
|
||
--requested-by local-developer
|
||
```
|
||
|
||
Use `--from-stage` to rebuild only downstream state. The `--reset` option deletes
|
||
the selected stage and downstream derived corpus data before execution.
|
||
|
||
## BFF authorization contract
|
||
|
||
The existing BFF authenticates callers and evaluates the `KnowledgeGraph` feature.
|
||
Independent permission bits are:
|
||
|
||
```text
|
||
Read = 1
|
||
Add = 2
|
||
Edit = 4
|
||
Delete = 8
|
||
Execute = 16
|
||
```
|
||
|
||
| Operation | Required mask |
|
||
|---|---:|
|
||
| Read job/source/retrieval information | `Read` (`1`) |
|
||
| Submit a stage-1 source job | `Add | Execute` (`18`) |
|
||
| Run downstream build stages | `Execute` (`16`) |
|
||
| Query or answer | `Read | Execute` (`17`) |
|
||
| Retry a failed job | `Execute` (`16`) |
|
||
|
||
The BFF forwards a signed internal JWT. Required claims:
|
||
|
||
```json
|
||
{
|
||
"iss": "knowledgegraph-bff",
|
||
"aud": "knowledgegraph-service",
|
||
"sub": "internal-caller-id",
|
||
"iat": 1791930000,
|
||
"exp": 1791930600,
|
||
"jti": "correlation-or-request-id",
|
||
"features": {
|
||
"KnowledgeGraph": 31
|
||
}
|
||
}
|
||
```
|
||
|
||
Dev/test uses HS256 and `INTERNAL_JWT_SECRET`. Production deployments can replace
|
||
this verifier with the BFF's workload identity, mTLS, or asymmetric/JWKS contract.
|
||
Never trust unsigned permission headers.
|
||
|
||
## Internal API
|
||
|
||
| Endpoint | Purpose |
|
||
|---|---|
|
||
| `GET /internal/knowledge-graph/health` | Process liveness |
|
||
| `GET /internal/knowledge-graph/ready` | Database and PgVector readiness |
|
||
| `POST /internal/knowledge-graph/jobs` | Queue stages 1–8 |
|
||
| `GET /internal/knowledge-graph/jobs/{jobId}` | Read durable job/stage status |
|
||
| `POST /internal/knowledge-graph/jobs/{jobId}/retry` | Requeue a failed job |
|
||
| `POST /internal/knowledge-graph/query` | Synchronous bounded retrieval |
|
||
| `POST /internal/knowledge-graph/answer` | Synchronous cited answer |
|
||
|
||
Example job request:
|
||
|
||
```json
|
||
{
|
||
"corpus": "product-knowledge",
|
||
"from_stage": 1,
|
||
"through_stage": 8,
|
||
"input_jsonl": "/app/artifacts/mcp-documents.jsonl",
|
||
"model_file": "/app/src/templates/knowledge_model.json",
|
||
"reset": false
|
||
}
|
||
```
|
||
|
||
Compose bind-mounts local [`artifacts`](artifacts) at `/app/artifacts`, allowing the
|
||
worker to read JSONL handoffs placed there. In deployed environments, prefer the
|
||
PostgreSQL source contract or a shared managed file volume.
|
||
|
||
Long-running stages return `202 Accepted` with a job ID. Query and answer requests
|
||
remain synchronous and bounded.
|
||
|
||
## Retry and logging behavior
|
||
|
||
Model-provider calls retry:
|
||
|
||
- HTTP 429
|
||
- HTTP 500, 502, 503, and 504
|
||
- connection failures
|
||
- timeouts
|
||
|
||
Retries honor numeric `Retry-After`, otherwise use bounded exponential backoff with
|
||
jitter. Authentication and validation errors are not retried.
|
||
|
||
Logs are emitted as one JSON object per line and include event names, run IDs,
|
||
corpus/stage context where available, attempts, status, and error details. Secrets
|
||
are not logged.
|
||
|
||
## Tests
|
||
|
||
Portable tests:
|
||
|
||
```powershell
|
||
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
|
||
```
|
||
|
||
Real PostgreSQL/PgVector migration and retrieval test:
|
||
|
||
```powershell
|
||
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
|
||
pytest .\tests\integration\test_pgvector_postgres.py
|
||
Remove-Item Env:TEST_POSTGRES_URL
|
||
```
|
||
|
||
The suite covers:
|
||
|
||
- Versioned source validation
|
||
- Nested metadata and quoted/comma-containing content
|
||
- Configuration parsing
|
||
- Chunk validation
|
||
- Cross-chunk relationship reconciliation
|
||
- Provider retry policy
|
||
- Full ten-stage behavior
|
||
- Signed BFF authorization and permission failures
|
||
- Durable API job and worker execution
|
||
- Alembic migration against PostgreSQL
|
||
- PgVector extension and HNSW index creation
|
||
- Server-side vector retrieval, graph expansion, and citations
|
||
|
||
Opt-in vector load test:
|
||
|
||
```powershell
|
||
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
|
||
$env:RUN_PERFORMANCE_TESTS = "1"
|
||
$env:PERF_VECTOR_ITEM_COUNT = "10000"
|
||
$env:PERF_MAX_QUERY_SECONDS = "2.0"
|
||
pytest .\tests\performance\test_vector_query_load.py
|
||
```
|
||
|
||
Set the item count and latency threshold to the capacity target for the dev/test
|
||
environment. Performance checks are excluded by default because machine resources
|
||
vary.
|
||
|
||
CI is defined in [`.github/workflows/ci.yml`](.github/workflows/ci.yml) and compiles,
|
||
tests, validates Compose, exercises real PgVector, and builds the application image.
|
||
|
||
## Persistent format policy
|
||
|
||
- PostgreSQL is the canonical operational store.
|
||
- JSON Lines is the preferred file handoff/checkpoint format.
|
||
- CSV is allowed only for explicitly defined flat import/export views.
|
||
|
||
CSV safely supports commas, quotes, and line breaks when written by a compliant CSV
|
||
library. It remains unsuitable as the canonical graph format because aliases,
|
||
metadata, source ID arrays, embeddings, and graph properties are nested and typed.
|
||
Never construct CSV by joining strings with commas.
|
||
|
||
## RAG database alternatives
|
||
|
||
### PostgreSQL with PgVector
|
||
|
||
Implemented default. Choose it for SQL transactions, colocated provenance and
|
||
vectors, self-managed deployment, and tunable HNSW retrieval. Graph traversal uses
|
||
indexed relational edges rather than a native property-graph language.
|
||
|
||
Official reference: [PgVector](https://github.com/pgvector/pgvector).
|
||
|
||
### Azure Cosmos DB for NoSQL
|
||
|
||
Choose it for managed/serverless document storage, vector indexing, and first-party
|
||
vector plus BM25 hybrid search. Cosmos DB's Gremlin API is separate from the NoSQL
|
||
vector feature; use document adjacency or a separate graph store.
|
||
|
||
References:
|
||
|
||
- [Cosmos DB vector search](https://learn.microsoft.com/azure/cosmos-db/vector-search)
|
||
- [Cosmos DB hybrid search](https://learn.microsoft.com/azure/cosmos-db/gen-ai/hybrid-search)
|
||
- [Cosmos DB Gremlin API](https://learn.microsoft.com/azure/cosmos-db/gremlin/overview)
|
||
|
||
### Neo4j
|
||
|
||
Choose it when graph topology, paths, and multi-hop traversal are primary retrieval
|
||
signals. Neo4j combines native property graphs, Cypher, vector indexes, and
|
||
graph-aware hybrid retrieval. It requires a dedicated backend adapter; changing
|
||
`DATABASE_URL` is not sufficient. Confirm version and edition requirements for the
|
||
selected vector and unified-search features.
|
||
|
||
References:
|
||
|
||
- [Neo4j vector indexes](https://neo4j.com/docs/cypher-manual/current/indexes/search-indexes/vector-indexes/)
|
||
- [Neo4j hybrid search](https://neo4j.com/developer/genai-ecosystem/hybrid-search/)
|
||
|
||
Other credible options:
|
||
|
||
- [Amazon Neptune Analytics](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html)
|
||
for AWS-managed graph/vector analytics
|
||
- [Azure AI Search](https://learn.microsoft.com/azure/search/hybrid-search-overview)
|
||
as a secondary managed hybrid index without a native graph model
|
||
|
||
## Operational guidance
|
||
|
||
- Keep the API and worker on internal networks accessible only to the BFF and
|
||
operators.
|
||
- Replace all dev secrets before shared deployment.
|
||
- Use a separate PostgreSQL role for migrations and a least-privilege runtime role.
|
||
- Back up PostgreSQL before destructive resets or migrations.
|
||
- Monitor queued/running/failed run counts, stage duration, LLM retries, database
|
||
pool usage, vector latency, and retrieval result counts.
|
||
- Set `WORKER_STALE_RUN_SECONDS` above the longest expected single stage.
|
||
- Rebuild stage 8 after changing embedding model or dimensions.
|
||
- Maintain a golden question/citation set to detect retrieval-quality regressions.
|
||
|
||
## Troubleshooting
|
||
|
||
### Database migrations are required
|
||
|
||
```powershell
|
||
alembic current
|
||
alembic upgrade head
|
||
```
|
||
|
||
### `LLM_API_KEY is required`
|
||
|
||
Set `LLM_API_KEY` in `.env` and restart API/worker containers.
|
||
|
||
### Embedding dimension mismatch
|
||
|
||
Set `LLM_EMBEDDING_DIMENSIONS` to the exact provider output dimension, then reset
|
||
and rerun stage 8.
|
||
|
||
### Source handoff rejected
|
||
|
||
Validate every JSONL line against schema version `1.0`. Confirm `corpus`,
|
||
`external_id`, URL, body, and `retrieved_at`.
|
||
|
||
### Job remains queued
|
||
|
||
```powershell
|
||
podman compose ps worker
|
||
podman compose logs --tail 100 worker
|
||
```
|
||
|
||
### PostgreSQL/PgVector is not ready
|
||
|
||
```powershell
|
||
podman compose ps postgres
|
||
podman compose logs --tail 100 postgres migrate
|
||
```
|
||
|
||
### Port conflict
|
||
|
||
Change `POSTGRES_PORT` or `API_PORT` in `.env`, then recreate the Compose stack.
|