2.4 KiB
2.4 KiB
System overview
Purpose
KnowledgeGraph GraphRAG turns externally collected documents into a source-grounded knowledge graph and vector retrieval index. It then combines semantic retrieval with bounded graph expansion to produce context and cited answers.
The collection boundary is deliberate: Web-Access MCP and Data-Store MCP discover and persist source material. This repository starts at their versioned handoff.
End-to-end flow
external source handoff
-> 1 normalize
-> 2 clean, deduplicate, chunk
-> 3 persist knowledge model
-> 4 extract entities and relationships
-> 5 reconcile aliases and duplicates
-> 6 materialize nodes and edges
-> 7 detect and summarize communities
-> 8 embed and index
-> 9 retrieve vector + graph + source context
-> 10 generate cited answer
Stages 1-8 persist durable checkpoints in PostgreSQL. A later stage can be re-run without repeating valid upstream work. A durable run records stage selection, attempts, configuration fingerprint, status, timestamps, and errors.
Runtime boundary
The deployment contains:
- PostgreSQL 16 with the PgVector extension.
- An Alembic migration job.
- An internal FastAPI service.
- A PostgreSQL-backed worker.
- An existing BFF that authenticates callers and enforces feature permissions.
- An OpenAI-compatible chat and embedding provider.
The API is an internal service and is not intended for direct customer access.
Important design decisions
- PostgreSQL is the canonical checkpoint store.
- PgVector performs server-side cosine search using a dimension-specific HNSW index.
- SQLite is a local-test fallback with JSON embeddings and in-process search.
- LLM calls happen outside long database transactions; validated chunk results are committed independently.
- A PostgreSQL advisory lock permits one active build per corpus.
- The worker can recover stale runs and resume after the last successful stage.
- The BFF supplies a short-lived signed internal JWT; unsigned permission headers are not trusted.
Non-goals and boundaries
- Crawling and source discovery are not implemented here.
- The API does not provide customer-facing authentication or authorization.
- Query and answer are not queued worker jobs.
- Citation correctness is requested through the answer prompt, not independently verified.
- Entity reconciliation is normalized-name based, not semantic entity matching.