docs updated

This commit is contained in:
GAJ3T 2026-10-15 10:20:38 -04:00
parent 964b043a94
commit 82a5a0e557
16 changed files with 768 additions and 0 deletions

View File

@ -64,6 +64,9 @@ The API is internal only and is not intended for direct customer access.
| [`Containerfile`](Containerfile) | Non-root application image | | [`Containerfile`](Containerfile) | Non-root application image |
| [`.env.example`](.env.example) | Runtime configuration template | | [`.env.example`](.env.example) | Runtime configuration template |
Detailed architecture, API, runbook, configuration, data model, testing, and
editable C4 diagrams are available in [`docs/`](docs/README.md).
## Prerequisites ## Prerequisites
- Python 3.11 or newer - Python 3.11 or newer

45
docs/README.md Normal file
View File

@ -0,0 +1,45 @@
# KnowledgeGraph GraphRAG documentation
This folder is the operational and technical reference for the KnowledgeGraph
GraphRAG proof of concept. The repository README remains the shortest getting
started path; these documents explain the contracts behind it.
## Read this first
| Document | Use it for |
|---|---|
| [System overview](./overview.md) | Scope, boundaries, stage flow, and key design decisions |
| [Architecture](./architecture.md) | Runtime topology, stage contracts, persistence, and retrieval |
| [Runbook](./runbook.md) | Step-by-step local setup, ingestion, rebuilds, recovery, and shutdown |
| [Configuration](./configuration.md) | Environment variables, defaults, and production guidance |
| [Source contract](./source-contract.md) | JSONL and PostgreSQL handoff schemas |
| [API reference](./api.md) | Internal endpoints, JWT claims, permissions, and examples |
| [Data model](./data-model.md) | Checkpoint tables and stage ownership |
| [Testing](./testing.md) | Test commands, prerequisites, and performance checks |
| [Operations and troubleshooting](./operations.md) | Logs, health checks, failures, retries, and known limitations |
| [Module reference](./module-reference.md) | Python modules, stage entry points, and responsibilities |
## C4 diagrams
The diagrams are editable in [diagrams.net/draw.io](https://app.diagrams.net/).
| C4 level | Diagram |
|---|---|
| Context | [c4-context.drawio](./diagrams/c4-context.drawio) |
| Container | [c4-container.drawio](./diagrams/c4-container.drawio) |
| Component | [c4-component.drawio](./diagrams/c4-component.drawio) |
| Code | [c4-code.drawio](./diagrams/c4-code.drawio) |
## Documentation scope
The documentation is based on the implementation, tests, migration, Compose
definition, CI workflow, templates, and repository README. It documents current
behavior rather than promising features that are not implemented. In particular:
- source collection is external to this repository;
- stages 1-8 are durable worker jobs;
- query and answer are synchronous API/stage operations;
- PostgreSQL with PgVector is the deployment target;
- SQLite is supported for isolated tests, not production retrieval;
- LLM output is schema-validated but not independently fact-checked.

71
docs/api.md Normal file
View File

@ -0,0 +1,71 @@
# Internal API reference
All routes are under `/internal/knowledge-graph`. Health is unauthenticated;
other routes require an internal signed JWT from the BFF.
## Endpoints
| Method and path | Permission | Behavior |
|---|---:|---|
| `GET /health` | none | process liveness |
| `GET /ready` | none | database and PgVector readiness |
| `POST /jobs` | stage 1: `18`; downstream: `16` | queue stages 1-8 |
| `GET /jobs/{jobId}` | `1` | durable job and stage status |
| `POST /jobs/{jobId}/retry` | `16` | requeue a failed job |
| `POST /query` | `17` | synchronous retrieval |
| `POST /answer` | `17` | synchronous retrieval plus cited answer |
Permission bits are `Read=1`, `Add=2`, `Edit=4`, `Delete=8`, and `Execute=16`.
## Job request
```json
{
"corpus": "product-knowledge",
"from_stage": 1,
"through_stage": 8,
"input_jsonl": "/app/artifacts/mcp-documents.jsonl",
"model_file": "/app/src/templates/knowledge_model.json",
"reset": false
}
```
Long-running job submission returns `202 Accepted` with a `job_id`. Query and
answer remain synchronous and bounded.
## Query and answer requests
```json
{
"corpus": "product-knowledge",
"question": "Who created Atlas?",
"top_k": 8
}
```
Retrieval returns vector `context`, bounded `graph_context`, and traceable
`sources`. Answer generation asks the LLM to cite those sources as `[S1]`,
`[S2]`, and so on.
## JWT contract
The verifier expects HS256 in the current implementation and validates `iss`,
`aud`, `sub`, `iat`, `exp`, `jti`, and a feature permission mask:
```json
{
"iss": "knowledgegraph-bff",
"aud": "knowledgegraph-service",
"sub": "internal-caller-id",
"iat": 1791930000,
"exp": 1791930600,
"jti": "request-id",
"features": {
"KnowledgeGraph": 31
}
}
```
The BFF remains the policy enforcement point. Do not substitute unsigned
headers for the token.

88
docs/architecture.md Normal file
View File

@ -0,0 +1,88 @@
# Architecture
## Logical architecture
```text
Web-Access/Data-Store MCP
|
| JSONL file or PostgreSQL source table
v
StageContext + numbered stages
|
+--> PostgreSQL checkpoint tables
+--> PgVector retrieval_items + HNSW index
+--> structured JSON logs and artifacts
|
+--> OpenAI-compatible chat/embedding provider
BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages
|
+--> query/answer stages
```
See the editable [C4 context diagram](./diagrams/c4-context.drawio) and
[C4 container diagram](./diagrams/c4-container.drawio) for the system boundary.
## Stage contracts
| Stage | Reads | Writes | Result |
|---:|---|---|---|
| 1 | JSONL or source table | `raw_documents`, `documents` | normalized documents |
| 2 | documents | `chunks` | cleaned, deduplicated chunks |
| 3 | settings/model file | `knowledge_models` | entity and relationship vocabulary |
| 4 | chunks, model | `extractions` | validated LLM extraction payloads |
| 5 | extractions | `resolved_entities`, `resolved_relationships` | canonical entities and links |
| 6 | resolved data | `nodes`, `edges` | graph materialization |
| 7 | graph | `communities` | detected communities and LLM summaries |
| 8 | chunks, graph, communities | `search_items` | embeddings and HNSW index |
| 9 | search index, graph, documents | none | vector context, graph context, sources |
| 10 | stage 9 result | none | cited answer text and sources |
Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds
the question, retrieves `top_k` items, expands graph neighbors for the
configured number of hops and limit, and resolves source URLs.
## Durable execution
`pipeline_runs` and `stage_runs` are separate from stage data. A run can be:
```text
queued -> running -> succeeded
\-> failed -> queued (retry)
```
The worker:
1. initializes and verifies the schema;
2. requeues stale runs older than `WORKER_STALE_RUN_SECONDS`;
3. claims one queued run using database locking;
4. takes a per-corpus PostgreSQL advisory lock;
5. executes the selected stage range;
6. persists stage status and outputs;
7. releases the lock.
If a corpus lock is unavailable, execution is deferred and the job remains
queued. Retry resumes after the last successful stage unless the caller uses
`--reset` or otherwise clears downstream state.
## Deployment topology
`compose.yaml` starts PostgreSQL, waits for its health check, runs the one-shot
`migrate` service, then starts `api` and `worker`. The API and worker share the
application environment and bind-mount `./artifacts` into `/app/artifacts`.
The [C4 container diagram](./diagrams/c4-container.drawio) shows these
containers and external dependencies. The [component diagram](./diagrams/c4-component.drawio)
maps the application package, and the [code diagram](./diagrams/c4-code.drawio)
maps the numbered stage pipeline.
## Failure and consistency model
- Source records are rejected before they become normalized documents if their
schema, required fields, or corpus are invalid.
- A completed stage is a checkpoint, not a transaction spanning the whole build.
- A failed stage may leave partial stage side effects; reset the affected stage
and downstream state when a clean rebuild is required.
- PostgreSQL is required to exercise PgVector and HNSW behavior.
- Embedding vector length must equal `LLM_EMBEDDING_DIMENSIONS`.

79
docs/configuration.md Normal file
View File

@ -0,0 +1,79 @@
# Configuration reference
`Settings` reads environment variables and `.env` values. The complete starter
set is in [`.env.example`](../.env.example). Values below are grouped by
responsibility; names are case-insensitive through Pydantic settings.
## Database and source
| Variable | Purpose | Default |
|---|---|---|
| `DATABASE_URL` | canonical checkpoint database | PostgreSQL on localhost |
| `SOURCE_DATABASE_URL` | optional external source database | empty |
| `SOURCE_TABLE_NAME` | source table | `external_documents` |
| `ANALYTICS_DATABASE_URL` | reserved analytics setting | empty |
| `AUTO_CREATE_SCHEMA` | test-only schema creation | `false` |
| `ARTIFACT_DIR` | local handoff/artifact directory | `artifacts` |
## Corpus and model
| Variable | Purpose | Default |
|---|---|---|
| `CORPUS_NAME` | corpus namespace | `default-corpus` |
| `ENTITY_TYPES` | comma-separated allowed entity types | Person, Organization, Product, Concept, Event, Location |
| `RELATIONSHIP_TYPES` | comma-separated allowed edge types | AFFILIATED_WITH, CREATED, DEPENDS_ON, MENTIONS, PART_OF, RELATED_TO |
| `NODE_TEMPLATE` | JSON extraction shape | template in `.env.example` |
| `EDGE_TEMPLATE` | JSON extraction shape | template in `.env.example` |
| `EXTRACTION_INSTRUCTIONS` | additional extraction guardrail | source-grounded extraction |
Use [the model template](../src/templates/knowledge_model.json) when invoking
stage 3 with `--model-file`.
## LLM provider
| Variable | Purpose | Default |
|---|---|---|
| `LLM_BASE_URL` | OpenAI-compatible API root | `https://api.openai.com/v1` |
| `LLM_API_KEY` | provider credential | empty |
| `LLM_CHAT_MODEL` | chat/completion model | `gpt-4.1-mini` |
| `LLM_EMBEDDING_MODEL` | embedding model | `text-embedding-3-small` |
| `LLM_EMBEDDING_DIMENSIONS` | expected vector length | `1536` |
| `LLM_TIMEOUT_SECONDS` | request timeout | `60` |
| `LLM_MAX_RETRIES` | retry count | `4` |
| `LLM_RETRY_INITIAL_SECONDS` | initial backoff | `1` |
| `LLM_RETRY_MAX_SECONDS` | maximum backoff | `30` |
Retries cover 429, 500/502/503/504, connection failures, and timeouts.
Numeric `Retry-After` is honored. Authentication and validation failures are
not retried.
## Processing and retrieval
| Variable | Purpose | Default |
|---|---|---:|
| `CHUNK_SIZE` | chunk target size | `1200` |
| `CHUNK_OVERLAP` | chunk overlap | `150` |
| `COMMUNITY_RESOLUTION` | NetworkX community resolution | `1.0` |
| `RETRIEVAL_TOP_K` | vector result count | `8` |
| `RETRIEVAL_GRAPH_HOPS` | graph expansion depth | `2` |
| `RETRIEVAL_GRAPH_LIMIT` | graph expansion cap | `50` |
| `RETRIEVAL_HNSW_EF_SEARCH` | PgVector search breadth | `100` |
`CHUNK_OVERLAP` must be smaller than `CHUNK_SIZE`. Embedding dimensions must
match the provider output and the PgVector index dimension.
## Security and worker
| Variable | Purpose |
|---|---|
| `INTERNAL_JWT_SECRET` | HS256 verification secret |
| `INTERNAL_JWT_ISSUER` | expected JWT issuer |
| `INTERNAL_JWT_AUDIENCE` | expected JWT audience |
| `INTERNAL_FEATURE_NAME` | feature key containing permission bits |
| `LOG_LEVEL` | structured log level |
| `WORKER_POLL_SECONDS` | idle worker polling interval |
| `WORKER_STALE_RUN_SECONDS` | stale-run recovery threshold |
Change every development secret before shared or production use. Prefer workload
identity, mTLS, or asymmetric/JWKS verification in production.

38
docs/data-model.md Normal file
View File

@ -0,0 +1,38 @@
# Data model and checkpoint ownership
The initial Alembic revision creates the SQLAlchemy metadata and the PgVector
extension. The models live in [`src/graphrag_platform/models.py`](../src/graphrag_platform/models.py).
## Pipeline data
| Table/model | Owner stage | Meaning |
|---|---:|---|
| `raw_documents` | 1 | original source payload and provenance |
| `documents` | 1 | normalized text and source identity |
| `chunks` | 2 | cleaned, overlapping retrieval units |
| `knowledge_models` | 3 | configured entity/relationship vocabulary and templates |
| `extractions` | 4 | validated LLM payload per chunk |
| `resolved_entities` | 5 | canonical entities and aliases |
| `resolved_relationships` | 5 | relationships linked to canonical keys |
| `nodes` | 6 | graph nodes |
| `edges` | 6 | graph edges |
| `communities` | 7 | detected groups and summaries |
| `search_items` | 8 | text, metadata, source URL, and embedding |
## Run tracking
| Table/model | Purpose |
|---|---|
| `pipeline_runs` | requested range, status, attempt, caller, correlation, and errors |
| `stage_runs` | per-stage status, input/output, timestamps, and error details |
Stage data is corpus-scoped. Clear or rebuild from the first invalid stage
through downstream derived data; do not manually delete unrelated corpora.
## Indexing
PostgreSQL creates a dimension-specific HNSW cosine index on retrieval items.
SQLite stores vectors as JSON and uses in-process cosine similarity. The latter
is useful for tests but does not validate production query plans or PgVector
behavior.

View File

@ -0,0 +1 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="code" name="Code"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="s1" value="01 normalize&#xa;raw + documents" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="60" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s2" value="02 clean/chunk&#xa;chunks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="250" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s3" value="03 model&#xa;knowledge_models" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="440" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s4" value="04 extract&#xa;extractions" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="630" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s5" value="05 reconcile&#xa;resolved data" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="820" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s6" value="06 graph&#xa;nodes + edges" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="155" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s7" value="07 communities&#xa;summaries" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="345" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s8" value="08 index&#xa;search_items + HNSW" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="535" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s9" value="09 retrieve&#xa;vector + graph context" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="725" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s10" value="10 answer&#xa;cited response" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="915" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="e1" edge="1" parent="1" source="s1" target="s2"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" edge="1" parent="1" source="s2" target="s3"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" edge="1" parent="1" source="s3" target="s4"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e4" edge="1" parent="1" source="s4" target="s5"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e5" edge="1" parent="1" source="s5" target="s6"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e6" edge="1" parent="1" source="s6" target="s7"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e7" edge="1" parent="1" source="s7" target="s8"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e8" edge="1" parent="1" source="s8" target="s9"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e9" edge="1" parent="1" source="s9" target="s10"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>

View File

@ -0,0 +1 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="component" name="Component"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="api" value="api.py&#xa;Routes and request models" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="80" y="90" width="210" height="70" as="geometry"/></mxCell><mxCell id="auth" value="auth.py&#xa;JWT and permission checks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="80" y="220" width="210" height="70" as="geometry"/></mxCell><mxCell id="orch" value="orchestrator.py&#xa;Durable runs and locks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="380" y="90" width="210" height="70" as="geometry"/></mxCell><mxCell id="runtime" value="runtime.py + stage_loader.py&#xa;Context and stage dispatch" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="380" y="220" width="210" height="70" as="geometry"/></mxCell><mxCell id="storage" value="storage.py + models.py&#xa;SQLAlchemy and PgVector" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="680" y="90" width="220" height="70" as="geometry"/></mxCell><mxCell id="llm" value="llm.py&#xa;Provider client and retries" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="680" y="220" width="220" height="70" as="geometry"/></mxCell><mxCell id="e1" value="authorize" edge="1" parent="1" source="api" target="auth"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" value="queue" edge="1" parent="1" source="api" target="orch"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" value="dispatch" edge="1" parent="1" source="orch" target="runtime"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e4" value="persist" edge="1" parent="1" source="runtime" target="storage"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e5" value="complete" edge="1" parent="1" source="runtime" target="llm"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>

View File

@ -0,0 +1 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="container" name="Container"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="api" value="FastAPI API&#xa;Internal query, answer, and job routes" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="460" y="80" width="240" height="80" as="geometry"/></mxCell><mxCell id="worker" value="Worker&#xa;Claims and executes durable runs" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="460" y="230" width="240" height="80" as="geometry"/></mxCell><mxCell id="db" value="PostgreSQL + PgVector&#xa;Checkpoints, graph, vectors, HNSW" style="shape=cylinder;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="800" y="180" width="250" height="100" as="geometry"/></mxCell><mxCell id="stages" value="Numbered stage modules&#xa;Normalize through answer" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="100" y="155" width="240" height="80" as="geometry"/></mxCell><mxCell id="llm" value="OpenAI-compatible provider&#xa;Chat and embeddings" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="100" y="320" width="240" height="80" as="geometry"/></mxCell><mxCell id="e1" value="dispatch" edge="1" parent="1" source="api" target="worker"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" value="execute" edge="1" parent="1" source="worker" target="stages"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" value="persist/retrieve" edge="1" parent="1" source="stages" target="db"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e4" value="prompts/vectors" edge="1" parent="1" source="stages" target="llm"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e5" value="status and query" edge="1" parent="1" source="api" target="db"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>

View File

@ -0,0 +1 @@
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="context" name="Context"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="kg" value="KnowledgeGraph GraphRAG" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="520" y="220" width="240" height="80" as="geometry"/></mxCell><mxCell id="mcp" value="Web-Access MCP + Data-Store MCP&#xa;External source collection" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="80" y="220" width="260" height="80" as="geometry"/></mxCell><mxCell id="bff" value="Existing BFF&#xa;Authentication and policy" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="870" y="100" width="220" height="80" as="geometry"/></mxCell><mxCell id="llm" value="OpenAI-compatible provider&#xa;Chat and embeddings" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="870" y="340" width="220" height="80" as="geometry"/></mxCell><mxCell id="e1" value="Versioned source handoff" edge="1" parent="1" source="mcp" target="kg"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" value="Signed internal JWT&#xa;and API calls" edge="1" parent="1" source="bff" target="kg"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" value="Prompts and embeddings" edge="1" parent="1" source="kg" target="llm"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>

46
docs/module-reference.md Normal file
View File

@ -0,0 +1,46 @@
# Module reference
## Platform package
| Module | Responsibility |
|---|---|
| `config.py` | typed settings, environment parsing, model definition |
| `models.py` | SQLAlchemy entities and checkpoint schema |
| `storage.py` | engine/session, schema checks, persistence, vector index |
| `sources.py` | strict JSONL and PostgreSQL source readers |
| `llm.py` | OpenAI-compatible completion and embedding client |
| `runtime.py` | `StageContext` construction and initialization |
| `stage_loader.py` | numbered stage module lookup |
| `pipeline.py` | compatibility facade for stage methods |
| `orchestrator.py` | durable runs, claiming, resume, stale recovery, locks |
| `auth.py` | JWT verification and permission masks |
| `api.py` | FastAPI app, request models, routes, authorization |
| `worker.py` | polling worker and one-run execution |
| `cli.py` | stage and durable-run command line entry points |
| `observability.py` | structured JSON logging |
## Numbered stages
| Module | Main responsibility |
|---|---|
| `01_normalize_content.py` | validate handoff, extract text, persist raw/normalized documents |
| `02_clean_content.py` | clean, deduplicate, and chunk |
| `03_define_knowledge_model.py` | persist entity/relationship vocabulary and templates |
| `04_extract_knowledge.py` | call LLM, validate extraction, persist per-chunk results |
| `05_reconcile_knowledge.py` | canonicalize names and resolve cross-chunk relationships |
| `06_build_knowledge_graph.py` | materialize nodes and edges |
| `07_build_communities.py` | detect communities and summarize them |
| `08_build_retrieval_index.py` | embed chunks/graph/community items and build HNSW |
| `09_query_graphrag.py` | vector retrieval, graph expansion, source resolution |
| `10_generate_answer.py` | generate answer with source citations |
Each numbered module exposes a `run` function and command-line adapter. Stages
1-5 also expose focused validation/transformation helpers; stage 9 exposes
retrieval helpers such as cosine ranking and graph expansion.
## Entry points
- `knowledgegraph` -> `graphrag_platform.cli:main_all`
- `knowledgegraph-worker` -> `graphrag_platform.worker:main`
- `uvicorn graphrag_platform.api:app`

76
docs/operations.md Normal file
View File

@ -0,0 +1,76 @@
# Operations and troubleshooting
## Health and readiness
- `/internal/knowledge-graph/health` checks process liveness.
- `/internal/knowledge-graph/ready` checks database and PgVector readiness.
- `podman compose ps` shows service and health state.
- `podman compose logs --follow api worker` streams structured JSON logs.
Logs include event names, run IDs, corpus/stage context, attempts, status, and
error details. Secrets are not intended to be logged.
## Common failures
### Migration does not complete
Check PostgreSQL health, credentials, and `DATABASE_URL`. Confirm the PgVector
image is used and inspect:
```powershell
podman compose logs migrate postgres
alembic current
```
Do not enable automatic schema creation in a shared environment.
### Source validation fails
Check schema version, required fields, ISO timestamp offset, unique
`external_id`, and corpus. The error identifies the source file and line.
### LLM requests fail
Verify `LLM_BASE_URL`, key, model names, and embedding dimensions. 429 and
transient server/network errors retry; authentication and validation errors fail
immediately. Check provider quota and `Retry-After` behavior before increasing
retry settings.
### Vector index or retrieval fails
Confirm the `vector` extension exists, the provider returns exactly
`LLM_EMBEDDING_DIMENSIONS` values, and stage 8 completed for the selected
corpus. SQLite cannot validate PgVector behavior.
### Job remains queued
Check worker logs and the corpus advisory lock. Another build may be active.
The worker requeues stale runs after `WORKER_STALE_RUN_SECONDS`; set this above
the maximum expected stage duration.
### Job fails after partial progress
Read the job and stage error, repair the source/configuration, then retry. Retry
resumes after the last successful stage. Use a reset rebuild if partial side
effects or changed model settings make the checkpoint unsafe.
## Known limitations
- Only source schema `1.0` is accepted.
- `ANALYTICS_DATABASE_URL` is configured but unused by the current pipeline.
- Source table names are simple unqualified names.
- Stage-level recovery is not an all-stage transaction.
- Name normalization can raise on conflicting entity types.
- LLM shape validation does not establish factual correctness.
- Answer citations are prompt-enforced, not independently checked.
- No observed API rate limiting or pagination middleware.
- The migration set currently contains one initial revision.
## Operational safety
- Use read-only credentials for the external source database.
- Keep API and worker on private network paths behind the BFF.
- Rotate all development secrets before deployment.
- Back up PostgreSQL before destructive resets or migration changes.
- Treat `podman compose down --volumes` as destructive.

68
docs/overview.md Normal file
View File

@ -0,0 +1,68 @@
# System overview
## Purpose
KnowledgeGraph GraphRAG turns externally collected documents into a
source-grounded knowledge graph and vector retrieval index. It then combines
semantic retrieval with bounded graph expansion to produce context and cited
answers.
The collection boundary is deliberate: Web-Access MCP and Data-Store MCP
discover and persist source material. This repository starts at their versioned
handoff.
## End-to-end flow
```text
external source handoff
-> 1 normalize
-> 2 clean, deduplicate, chunk
-> 3 persist knowledge model
-> 4 extract entities and relationships
-> 5 reconcile aliases and duplicates
-> 6 materialize nodes and edges
-> 7 detect and summarize communities
-> 8 embed and index
-> 9 retrieve vector + graph + source context
-> 10 generate cited answer
```
Stages 1-8 persist durable checkpoints in PostgreSQL. A later stage can be
re-run without repeating valid upstream work. A durable run records stage
selection, attempts, configuration fingerprint, status, timestamps, and errors.
## Runtime boundary
The deployment contains:
1. PostgreSQL 16 with the PgVector extension.
2. An Alembic migration job.
3. An internal FastAPI service.
4. A PostgreSQL-backed worker.
5. An existing BFF that authenticates callers and enforces feature permissions.
6. An OpenAI-compatible chat and embedding provider.
The API is an internal service and is not intended for direct customer access.
## Important design decisions
- PostgreSQL is the canonical checkpoint store.
- PgVector performs server-side cosine search using a dimension-specific HNSW
index.
- SQLite is a local-test fallback with JSON embeddings and in-process search.
- LLM calls happen outside long database transactions; validated chunk results
are committed independently.
- A PostgreSQL advisory lock permits one active build per corpus.
- The worker can recover stale runs and resume after the last successful stage.
- The BFF supplies a short-lived signed internal JWT; unsigned permission
headers are not trusted.
## Non-goals and boundaries
- Crawling and source discovery are not implemented here.
- The API does not provide customer-facing authentication or authorization.
- Query and answer are not queued worker jobs.
- Citation correctness is requested through the answer prompt, not independently
verified.
- Entity reconciliation is normalized-name based, not semantic entity matching.

131
docs/runbook.md Normal file
View File

@ -0,0 +1,131 @@
# Step-by-step runbook
Commands use PowerShell on Windows and assume the repository root is current.
## 1. Prepare tools and environment
Prerequisites: Python 3.11+, Podman 5+, a running `podman machine`, and an
OpenAI-compatible chat/embedding endpoint.
```powershell
Copy-Item .env.example .env
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[test]"
```
Edit `.env`: set `POSTGRES_PASSWORD`, matching `DATABASE_URL` credentials,
`LLM_API_KEY`, model names/dimensions, and a random
`INTERNAL_JWT_SECRET` of at least 32 bytes.
## 2. Start the stack
```powershell
podman machine start
podman compose up --detach --build
podman compose ps
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready
```
Compose starts PostgreSQL, applies migrations, and then starts API and worker.
The migration service must complete before the application services start.
## 3. Supply a source handoff
Place a UTF-8 JSONL file in `.\artifacts`, or configure a PostgreSQL source
table. Validate it against the [source contract](./source-contract.md). For a
file handoff, use `.\artifacts\mcp-documents.jsonl`.
## 4. Run a durable build
```powershell
knowledgegraph run `
--corpus product-knowledge `
--from-stage 1 `
--through-stage 8 `
--input-jsonl .\artifacts\mcp-documents.jsonl `
--model-file .\src\templates\knowledge_model.json `
--requested-by local-developer
```
Watch progress:
```powershell
podman compose logs --follow api worker
```
## 5. Query and answer
For direct local stage execution:
```powershell
python .\src\09_query_graphrag.py `
--corpus product-knowledge `
--question "Who created Atlas?" `
--top-k 8
python .\src\10_generate_answer.py `
--corpus product-knowledge `
--question "Who created Atlas?"
```
For the API, obtain a BFF-issued JWT and call `/query` or `/answer` as described
in the [API reference](./api.md).
## 6. Rebuild downstream data
Use `--from-stage` to avoid repeating valid upstream work. Use `--reset` when
the selected stage or any downstream data is inconsistent:
```powershell
knowledgegraph run `
--corpus product-knowledge `
--from-stage 4 `
--through-stage 8 `
--reset `
--requested-by rebuild
```
Do not run two builds for the same corpus concurrently; the PostgreSQL advisory
lock will defer one, but coordinating callers is clearer.
## 7. Apply or inspect migrations
```powershell
alembic current
alembic history
alembic upgrade head
```
`AUTO_CREATE_SCHEMA=true` is for isolated tests only. Review generated
migrations before applying them.
## 8. Run tests
```powershell
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
```
For the real PgVector integration:
```powershell
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
pytest .\tests\integration\test_pgvector_postgres.py
Remove-Item Env:TEST_POSTGRES_URL
```
## 9. Stop or reset local services
```powershell
podman compose down
```
This retains the database volume. To delete all local PostgreSQL data:
```powershell
podman compose down --volumes
```
Volume removal is destructive.

72
docs/source-contract.md Normal file
View File

@ -0,0 +1,72 @@
# External source contract
## JSON Lines
Each UTF-8 line must be a JSON object with schema version `1.0`.
```json
{
"schema_version": "1.0",
"external_id": "stable-source-record-id",
"corpus": "product-knowledge",
"url": "https://docs.example.com/article",
"body": "<html><body><main>Article text</main></body></html>",
"content_type": "text/html",
"status_code": 200,
"retrieved_at": "2026-10-13T18:00:00Z",
"metadata": {
"collector": "Web-Access",
"stored_by": "Data-Store"
}
}
```
Required fields:
- `schema_version`: exactly `1.0`;
- `external_id`: stable and unique within a corpus;
- `url`;
- `body`;
- `retrieved_at`: ISO 8601 timestamp with an offset.
Defaulted fields:
- `content_type`: `text/html`;
- `status_code`: `200`;
- `metadata`: `{}`.
`corpus`, when provided, must equal the selected runtime corpus. Unknown fields
and unsupported schema versions are rejected. Validation errors include the
file and line number. See the canonical template at
[`src/templates/mcp_documents.jsonl`](../src/templates/mcp_documents.jsonl) and
the acceptance fixture at [`tests/fixtures/external_documents.jsonl`](../tests/fixtures/external_documents.jsonl).
## PostgreSQL source table
Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to use a PostgreSQL handoff.
The table must contain:
```sql
CREATE TABLE external_documents (
external_id text PRIMARY KEY,
schema_version text NOT NULL DEFAULT '1.0',
corpus text,
url text NOT NULL,
body text NOT NULL,
content_type text NOT NULL DEFAULT 'text/html',
status_code integer NOT NULL DEFAULT 200,
retrieved_at timestamptz NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
);
```
Additional columns are ignored. Give the application account read-only access.
Use either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both.
## Normalization behavior
Stage 1 stores raw provenance and a normalized document. HTML is extracted with
Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes
whitespace, detects duplicate content by SHA-256 hash, and creates overlapping
chunks.

47
docs/testing.md Normal file
View File

@ -0,0 +1,47 @@
# Testing and validation
## Test layers
| Layer | Coverage |
|---|---|
| Unit | settings, source validation, chunk constraints, reconciliation, provider retry policy |
| Integration | all ten stages, API/worker jobs, repair/retry/resume |
| PostgreSQL integration | migrations, PgVector extension, HNSW index, graph retrieval, advisory lock |
| Performance | configurable vector query load against real PostgreSQL |
The deterministic `FakeLLM` in `tests/conftest.py` keeps portable tests
repeatable. It is not a quality or latency substitute for a production model.
## Local validation
```powershell
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
python -m compileall -q src tests migrations
python -m pip check
```
The CI workflow also validates Compose configuration and builds the
`Containerfile` image.
## PostgreSQL integration
Set `TEST_POSTGRES_URL` to a PgVector-enabled database. The migration test
resets the schema, creates the external source table, runs stages 1-9, checks
the vector extension and HNSW definition, and verifies advisory-lock behavior.
## Performance benchmark
The benchmark is skipped unless both `TEST_POSTGRES_URL` and
`RUN_PERFORMANCE_TESTS=1` are set:
```powershell
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
$env:RUN_PERFORMANCE_TESTS = "1"
$env:PERF_VECTOR_ITEM_COUNT = "10000"
$env:PERF_MAX_QUERY_SECONDS = "2.0"
pytest .\tests\performance\test_vector_query_load.py -m performance
```
The benchmark inserts deterministic three-dimensional vectors and measures
stage 9 retrieval. Results depend on hardware, database configuration, and load.