69 lines
2.4 KiB
Markdown
69 lines
2.4 KiB
Markdown
# System overview
|
|
|
|
## Purpose
|
|
|
|
KnowledgeGraph GraphRAG turns externally collected documents into a
|
|
source-grounded knowledge graph and vector retrieval index. It then combines
|
|
semantic retrieval with bounded graph expansion to produce context and cited
|
|
answers.
|
|
|
|
The collection boundary is deliberate: Web-Access MCP and Data-Store MCP
|
|
discover and persist source material. This repository starts at their versioned
|
|
handoff.
|
|
|
|
## End-to-end flow
|
|
|
|
```text
|
|
external source handoff
|
|
-> 1 normalize
|
|
-> 2 clean, deduplicate, chunk
|
|
-> 3 persist knowledge model
|
|
-> 4 extract entities and relationships
|
|
-> 5 reconcile aliases and duplicates
|
|
-> 6 materialize nodes and edges
|
|
-> 7 detect and summarize communities
|
|
-> 8 embed and index
|
|
-> 9 retrieve vector + graph + source context
|
|
-> 10 generate cited answer
|
|
```
|
|
|
|
Stages 1-8 persist durable checkpoints in PostgreSQL. A later stage can be
|
|
re-run without repeating valid upstream work. A durable run records stage
|
|
selection, attempts, configuration fingerprint, status, timestamps, and errors.
|
|
|
|
## Runtime boundary
|
|
|
|
The deployment contains:
|
|
|
|
1. PostgreSQL 16 with the PgVector extension.
|
|
2. An Alembic migration job.
|
|
3. An internal FastAPI service.
|
|
4. A PostgreSQL-backed worker.
|
|
5. An existing BFF that authenticates callers and enforces feature permissions.
|
|
6. An OpenAI-compatible chat and embedding provider.
|
|
|
|
The API is an internal service and is not intended for direct customer access.
|
|
|
|
## Important design decisions
|
|
|
|
- PostgreSQL is the canonical checkpoint store.
|
|
- PgVector performs server-side cosine search using a dimension-specific HNSW
|
|
index.
|
|
- SQLite is a local-test fallback with JSON embeddings and in-process search.
|
|
- LLM calls happen outside long database transactions; validated chunk results
|
|
are committed independently.
|
|
- A PostgreSQL advisory lock permits one active build per corpus.
|
|
- The worker can recover stale runs and resume after the last successful stage.
|
|
- The BFF supplies a short-lived signed internal JWT; unsigned permission
|
|
headers are not trusted.
|
|
|
|
## Non-goals and boundaries
|
|
|
|
- Crawling and source discovery are not implemented here.
|
|
- The API does not provide customer-facing authentication or authorization.
|
|
- Query and answer are not queued worker jobs.
|
|
- Citation correctness is requested through the answer prompt, not independently
|
|
verified.
|
|
- Entity reconciliation is normalized-name based, not semantic entity matching.
|
|
|