# System overview ## Purpose KnowledgeGraph GraphRAG turns externally collected documents into a source-grounded knowledge graph and vector retrieval index. It then combines semantic retrieval with bounded graph expansion to produce context and cited answers. The collection boundary is deliberate: Web-Access MCP and Data-Store MCP discover and persist source material. This repository starts at their versioned handoff. ## End-to-end flow ```text external source handoff -> 1 normalize -> 2 clean, deduplicate, chunk -> 3 persist knowledge model -> 4 extract entities and relationships -> 5 reconcile aliases and duplicates -> 6 materialize nodes and edges -> 7 detect and summarize communities -> 8 embed and index -> 9 retrieve vector + graph + source context -> 10 generate cited answer ``` Stages 1-8 persist durable checkpoints in PostgreSQL. A later stage can be re-run without repeating valid upstream work. A durable run records stage selection, attempts, configuration fingerprint, status, timestamps, and errors. ## Runtime boundary The deployment contains: 1. PostgreSQL 16 with the PgVector extension. 2. An Alembic migration job. 3. An internal FastAPI service. 4. A PostgreSQL-backed worker. 5. An existing BFF that authenticates callers and enforces feature permissions. 6. An OpenAI-compatible chat and embedding provider. The API is an internal service and is not intended for direct customer access. ## Important design decisions - PostgreSQL is the canonical checkpoint store. - PgVector performs server-side cosine search using a dimension-specific HNSW index. - SQLite is a local-test fallback with JSON embeddings and in-process search. - LLM calls happen outside long database transactions; validated chunk results are committed independently. - A PostgreSQL advisory lock permits one active build per corpus. - The worker can recover stale runs and resume after the last successful stage. - The BFF supplies a short-lived signed internal JWT; unsigned permission headers are not trusted. ## Non-goals and boundaries - Crawling and source discovery are not implemented here. - The API does not provide customer-facing authentication or authorization. - Query and answer are not queued worker jobs. - Citation correctness is requested through the answer prompt, not independently verified. - Entity reconciliation is normalized-name based, not semantic entity matching.