docs updated
This commit is contained in:
parent
964b043a94
commit
82a5a0e557
|
|
@ -64,6 +64,9 @@ The API is internal only and is not intended for direct customer access.
|
|||
| [`Containerfile`](Containerfile) | Non-root application image |
|
||||
| [`.env.example`](.env.example) | Runtime configuration template |
|
||||
|
||||
Detailed architecture, API, runbook, configuration, data model, testing, and
|
||||
editable C4 diagrams are available in [`docs/`](docs/README.md).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.11 or newer
|
||||
|
|
|
|||
|
|
@ -0,0 +1,45 @@
|
|||
# KnowledgeGraph GraphRAG documentation
|
||||
|
||||
This folder is the operational and technical reference for the KnowledgeGraph
|
||||
GraphRAG proof of concept. The repository README remains the shortest getting
|
||||
started path; these documents explain the contracts behind it.
|
||||
|
||||
## Read this first
|
||||
|
||||
| Document | Use it for |
|
||||
|---|---|
|
||||
| [System overview](./overview.md) | Scope, boundaries, stage flow, and key design decisions |
|
||||
| [Architecture](./architecture.md) | Runtime topology, stage contracts, persistence, and retrieval |
|
||||
| [Runbook](./runbook.md) | Step-by-step local setup, ingestion, rebuilds, recovery, and shutdown |
|
||||
| [Configuration](./configuration.md) | Environment variables, defaults, and production guidance |
|
||||
| [Source contract](./source-contract.md) | JSONL and PostgreSQL handoff schemas |
|
||||
| [API reference](./api.md) | Internal endpoints, JWT claims, permissions, and examples |
|
||||
| [Data model](./data-model.md) | Checkpoint tables and stage ownership |
|
||||
| [Testing](./testing.md) | Test commands, prerequisites, and performance checks |
|
||||
| [Operations and troubleshooting](./operations.md) | Logs, health checks, failures, retries, and known limitations |
|
||||
| [Module reference](./module-reference.md) | Python modules, stage entry points, and responsibilities |
|
||||
|
||||
## C4 diagrams
|
||||
|
||||
The diagrams are editable in [diagrams.net/draw.io](https://app.diagrams.net/).
|
||||
|
||||
| C4 level | Diagram |
|
||||
|---|---|
|
||||
| Context | [c4-context.drawio](./diagrams/c4-context.drawio) |
|
||||
| Container | [c4-container.drawio](./diagrams/c4-container.drawio) |
|
||||
| Component | [c4-component.drawio](./diagrams/c4-component.drawio) |
|
||||
| Code | [c4-code.drawio](./diagrams/c4-code.drawio) |
|
||||
|
||||
## Documentation scope
|
||||
|
||||
The documentation is based on the implementation, tests, migration, Compose
|
||||
definition, CI workflow, templates, and repository README. It documents current
|
||||
behavior rather than promising features that are not implemented. In particular:
|
||||
|
||||
- source collection is external to this repository;
|
||||
- stages 1-8 are durable worker jobs;
|
||||
- query and answer are synchronous API/stage operations;
|
||||
- PostgreSQL with PgVector is the deployment target;
|
||||
- SQLite is supported for isolated tests, not production retrieval;
|
||||
- LLM output is schema-validated but not independently fact-checked.
|
||||
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# Internal API reference
|
||||
|
||||
All routes are under `/internal/knowledge-graph`. Health is unauthenticated;
|
||||
other routes require an internal signed JWT from the BFF.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method and path | Permission | Behavior |
|
||||
|---|---:|---|
|
||||
| `GET /health` | none | process liveness |
|
||||
| `GET /ready` | none | database and PgVector readiness |
|
||||
| `POST /jobs` | stage 1: `18`; downstream: `16` | queue stages 1-8 |
|
||||
| `GET /jobs/{jobId}` | `1` | durable job and stage status |
|
||||
| `POST /jobs/{jobId}/retry` | `16` | requeue a failed job |
|
||||
| `POST /query` | `17` | synchronous retrieval |
|
||||
| `POST /answer` | `17` | synchronous retrieval plus cited answer |
|
||||
|
||||
Permission bits are `Read=1`, `Add=2`, `Edit=4`, `Delete=8`, and `Execute=16`.
|
||||
|
||||
## Job request
|
||||
|
||||
```json
|
||||
{
|
||||
"corpus": "product-knowledge",
|
||||
"from_stage": 1,
|
||||
"through_stage": 8,
|
||||
"input_jsonl": "/app/artifacts/mcp-documents.jsonl",
|
||||
"model_file": "/app/src/templates/knowledge_model.json",
|
||||
"reset": false
|
||||
}
|
||||
```
|
||||
|
||||
Long-running job submission returns `202 Accepted` with a `job_id`. Query and
|
||||
answer remain synchronous and bounded.
|
||||
|
||||
## Query and answer requests
|
||||
|
||||
```json
|
||||
{
|
||||
"corpus": "product-knowledge",
|
||||
"question": "Who created Atlas?",
|
||||
"top_k": 8
|
||||
}
|
||||
```
|
||||
|
||||
Retrieval returns vector `context`, bounded `graph_context`, and traceable
|
||||
`sources`. Answer generation asks the LLM to cite those sources as `[S1]`,
|
||||
`[S2]`, and so on.
|
||||
|
||||
## JWT contract
|
||||
|
||||
The verifier expects HS256 in the current implementation and validates `iss`,
|
||||
`aud`, `sub`, `iat`, `exp`, `jti`, and a feature permission mask:
|
||||
|
||||
```json
|
||||
{
|
||||
"iss": "knowledgegraph-bff",
|
||||
"aud": "knowledgegraph-service",
|
||||
"sub": "internal-caller-id",
|
||||
"iat": 1791930000,
|
||||
"exp": 1791930600,
|
||||
"jti": "request-id",
|
||||
"features": {
|
||||
"KnowledgeGraph": 31
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The BFF remains the policy enforcement point. Do not substitute unsigned
|
||||
headers for the token.
|
||||
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
# Architecture
|
||||
|
||||
## Logical architecture
|
||||
|
||||
```text
|
||||
Web-Access/Data-Store MCP
|
||||
|
|
||||
| JSONL file or PostgreSQL source table
|
||||
v
|
||||
StageContext + numbered stages
|
||||
|
|
||||
+--> PostgreSQL checkpoint tables
|
||||
+--> PgVector retrieval_items + HNSW index
|
||||
+--> structured JSON logs and artifacts
|
||||
|
|
||||
+--> OpenAI-compatible chat/embedding provider
|
||||
|
||||
BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages
|
||||
|
|
||||
+--> query/answer stages
|
||||
```
|
||||
|
||||
See the editable [C4 context diagram](./diagrams/c4-context.drawio) and
|
||||
[C4 container diagram](./diagrams/c4-container.drawio) for the system boundary.
|
||||
|
||||
## Stage contracts
|
||||
|
||||
| Stage | Reads | Writes | Result |
|
||||
|---:|---|---|---|
|
||||
| 1 | JSONL or source table | `raw_documents`, `documents` | normalized documents |
|
||||
| 2 | documents | `chunks` | cleaned, deduplicated chunks |
|
||||
| 3 | settings/model file | `knowledge_models` | entity and relationship vocabulary |
|
||||
| 4 | chunks, model | `extractions` | validated LLM extraction payloads |
|
||||
| 5 | extractions | `resolved_entities`, `resolved_relationships` | canonical entities and links |
|
||||
| 6 | resolved data | `nodes`, `edges` | graph materialization |
|
||||
| 7 | graph | `communities` | detected communities and LLM summaries |
|
||||
| 8 | chunks, graph, communities | `search_items` | embeddings and HNSW index |
|
||||
| 9 | search index, graph, documents | none | vector context, graph context, sources |
|
||||
| 10 | stage 9 result | none | cited answer text and sources |
|
||||
|
||||
Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds
|
||||
the question, retrieves `top_k` items, expands graph neighbors for the
|
||||
configured number of hops and limit, and resolves source URLs.
|
||||
|
||||
## Durable execution
|
||||
|
||||
`pipeline_runs` and `stage_runs` are separate from stage data. A run can be:
|
||||
|
||||
```text
|
||||
queued -> running -> succeeded
|
||||
\-> failed -> queued (retry)
|
||||
```
|
||||
|
||||
The worker:
|
||||
|
||||
1. initializes and verifies the schema;
|
||||
2. requeues stale runs older than `WORKER_STALE_RUN_SECONDS`;
|
||||
3. claims one queued run using database locking;
|
||||
4. takes a per-corpus PostgreSQL advisory lock;
|
||||
5. executes the selected stage range;
|
||||
6. persists stage status and outputs;
|
||||
7. releases the lock.
|
||||
|
||||
If a corpus lock is unavailable, execution is deferred and the job remains
|
||||
queued. Retry resumes after the last successful stage unless the caller uses
|
||||
`--reset` or otherwise clears downstream state.
|
||||
|
||||
## Deployment topology
|
||||
|
||||
`compose.yaml` starts PostgreSQL, waits for its health check, runs the one-shot
|
||||
`migrate` service, then starts `api` and `worker`. The API and worker share the
|
||||
application environment and bind-mount `./artifacts` into `/app/artifacts`.
|
||||
|
||||
The [C4 container diagram](./diagrams/c4-container.drawio) shows these
|
||||
containers and external dependencies. The [component diagram](./diagrams/c4-component.drawio)
|
||||
maps the application package, and the [code diagram](./diagrams/c4-code.drawio)
|
||||
maps the numbered stage pipeline.
|
||||
|
||||
## Failure and consistency model
|
||||
|
||||
- Source records are rejected before they become normalized documents if their
|
||||
schema, required fields, or corpus are invalid.
|
||||
- A completed stage is a checkpoint, not a transaction spanning the whole build.
|
||||
- A failed stage may leave partial stage side effects; reset the affected stage
|
||||
and downstream state when a clean rebuild is required.
|
||||
- PostgreSQL is required to exercise PgVector and HNSW behavior.
|
||||
- Embedding vector length must equal `LLM_EMBEDDING_DIMENSIONS`.
|
||||
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
# Configuration reference
|
||||
|
||||
`Settings` reads environment variables and `.env` values. The complete starter
|
||||
set is in [`.env.example`](../.env.example). Values below are grouped by
|
||||
responsibility; names are case-insensitive through Pydantic settings.
|
||||
|
||||
## Database and source
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | canonical checkpoint database | PostgreSQL on localhost |
|
||||
| `SOURCE_DATABASE_URL` | optional external source database | empty |
|
||||
| `SOURCE_TABLE_NAME` | source table | `external_documents` |
|
||||
| `ANALYTICS_DATABASE_URL` | reserved analytics setting | empty |
|
||||
| `AUTO_CREATE_SCHEMA` | test-only schema creation | `false` |
|
||||
| `ARTIFACT_DIR` | local handoff/artifact directory | `artifacts` |
|
||||
|
||||
## Corpus and model
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|---|---|---|
|
||||
| `CORPUS_NAME` | corpus namespace | `default-corpus` |
|
||||
| `ENTITY_TYPES` | comma-separated allowed entity types | Person, Organization, Product, Concept, Event, Location |
|
||||
| `RELATIONSHIP_TYPES` | comma-separated allowed edge types | AFFILIATED_WITH, CREATED, DEPENDS_ON, MENTIONS, PART_OF, RELATED_TO |
|
||||
| `NODE_TEMPLATE` | JSON extraction shape | template in `.env.example` |
|
||||
| `EDGE_TEMPLATE` | JSON extraction shape | template in `.env.example` |
|
||||
| `EXTRACTION_INSTRUCTIONS` | additional extraction guardrail | source-grounded extraction |
|
||||
|
||||
Use [the model template](../src/templates/knowledge_model.json) when invoking
|
||||
stage 3 with `--model-file`.
|
||||
|
||||
## LLM provider
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|---|---|---|
|
||||
| `LLM_BASE_URL` | OpenAI-compatible API root | `https://api.openai.com/v1` |
|
||||
| `LLM_API_KEY` | provider credential | empty |
|
||||
| `LLM_CHAT_MODEL` | chat/completion model | `gpt-4.1-mini` |
|
||||
| `LLM_EMBEDDING_MODEL` | embedding model | `text-embedding-3-small` |
|
||||
| `LLM_EMBEDDING_DIMENSIONS` | expected vector length | `1536` |
|
||||
| `LLM_TIMEOUT_SECONDS` | request timeout | `60` |
|
||||
| `LLM_MAX_RETRIES` | retry count | `4` |
|
||||
| `LLM_RETRY_INITIAL_SECONDS` | initial backoff | `1` |
|
||||
| `LLM_RETRY_MAX_SECONDS` | maximum backoff | `30` |
|
||||
|
||||
Retries cover 429, 500/502/503/504, connection failures, and timeouts.
|
||||
Numeric `Retry-After` is honored. Authentication and validation failures are
|
||||
not retried.
|
||||
|
||||
## Processing and retrieval
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|---|---|---:|
|
||||
| `CHUNK_SIZE` | chunk target size | `1200` |
|
||||
| `CHUNK_OVERLAP` | chunk overlap | `150` |
|
||||
| `COMMUNITY_RESOLUTION` | NetworkX community resolution | `1.0` |
|
||||
| `RETRIEVAL_TOP_K` | vector result count | `8` |
|
||||
| `RETRIEVAL_GRAPH_HOPS` | graph expansion depth | `2` |
|
||||
| `RETRIEVAL_GRAPH_LIMIT` | graph expansion cap | `50` |
|
||||
| `RETRIEVAL_HNSW_EF_SEARCH` | PgVector search breadth | `100` |
|
||||
|
||||
`CHUNK_OVERLAP` must be smaller than `CHUNK_SIZE`. Embedding dimensions must
|
||||
match the provider output and the PgVector index dimension.
|
||||
|
||||
## Security and worker
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `INTERNAL_JWT_SECRET` | HS256 verification secret |
|
||||
| `INTERNAL_JWT_ISSUER` | expected JWT issuer |
|
||||
| `INTERNAL_JWT_AUDIENCE` | expected JWT audience |
|
||||
| `INTERNAL_FEATURE_NAME` | feature key containing permission bits |
|
||||
| `LOG_LEVEL` | structured log level |
|
||||
| `WORKER_POLL_SECONDS` | idle worker polling interval |
|
||||
| `WORKER_STALE_RUN_SECONDS` | stale-run recovery threshold |
|
||||
|
||||
Change every development secret before shared or production use. Prefer workload
|
||||
identity, mTLS, or asymmetric/JWKS verification in production.
|
||||
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
# Data model and checkpoint ownership
|
||||
|
||||
The initial Alembic revision creates the SQLAlchemy metadata and the PgVector
|
||||
extension. The models live in [`src/graphrag_platform/models.py`](../src/graphrag_platform/models.py).
|
||||
|
||||
## Pipeline data
|
||||
|
||||
| Table/model | Owner stage | Meaning |
|
||||
|---|---:|---|
|
||||
| `raw_documents` | 1 | original source payload and provenance |
|
||||
| `documents` | 1 | normalized text and source identity |
|
||||
| `chunks` | 2 | cleaned, overlapping retrieval units |
|
||||
| `knowledge_models` | 3 | configured entity/relationship vocabulary and templates |
|
||||
| `extractions` | 4 | validated LLM payload per chunk |
|
||||
| `resolved_entities` | 5 | canonical entities and aliases |
|
||||
| `resolved_relationships` | 5 | relationships linked to canonical keys |
|
||||
| `nodes` | 6 | graph nodes |
|
||||
| `edges` | 6 | graph edges |
|
||||
| `communities` | 7 | detected groups and summaries |
|
||||
| `search_items` | 8 | text, metadata, source URL, and embedding |
|
||||
|
||||
## Run tracking
|
||||
|
||||
| Table/model | Purpose |
|
||||
|---|---|
|
||||
| `pipeline_runs` | requested range, status, attempt, caller, correlation, and errors |
|
||||
| `stage_runs` | per-stage status, input/output, timestamps, and error details |
|
||||
|
||||
Stage data is corpus-scoped. Clear or rebuild from the first invalid stage
|
||||
through downstream derived data; do not manually delete unrelated corpora.
|
||||
|
||||
## Indexing
|
||||
|
||||
PostgreSQL creates a dimension-specific HNSW cosine index on retrieval items.
|
||||
SQLite stores vectors as JSON and uses in-process cosine similarity. The latter
|
||||
is useful for tests but does not validate production query plans or PgVector
|
||||
behavior.
|
||||
|
||||
|
|
@ -0,0 +1 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="code" name="Code"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="s1" value="01 normalize
raw + documents" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="60" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s2" value="02 clean/chunk
chunks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="250" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s3" value="03 model
knowledge_models" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="440" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s4" value="04 extract
extractions" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="630" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s5" value="05 reconcile
resolved data" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="820" y="100" width="150" height="65" as="geometry"/></mxCell><mxCell id="s6" value="06 graph
nodes + edges" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="155" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s7" value="07 communities
summaries" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="345" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s8" value="08 index
search_items + HNSW" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="535" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s9" value="09 retrieve
vector + graph context" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="725" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="s10" value="10 answer
cited response" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="915" y="240" width="150" height="65" as="geometry"/></mxCell><mxCell id="e1" edge="1" parent="1" source="s1" target="s2"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" edge="1" parent="1" source="s2" target="s3"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" edge="1" parent="1" source="s3" target="s4"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e4" edge="1" parent="1" source="s4" target="s5"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e5" edge="1" parent="1" source="s5" target="s6"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e6" edge="1" parent="1" source="s6" target="s7"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e7" edge="1" parent="1" source="s7" target="s8"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e8" edge="1" parent="1" source="s8" target="s9"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e9" edge="1" parent="1" source="s9" target="s10"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>
|
||||
|
|
@ -0,0 +1 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="component" name="Component"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="api" value="api.py
Routes and request models" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="80" y="90" width="210" height="70" as="geometry"/></mxCell><mxCell id="auth" value="auth.py
JWT and permission checks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="80" y="220" width="210" height="70" as="geometry"/></mxCell><mxCell id="orch" value="orchestrator.py
Durable runs and locks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="380" y="90" width="210" height="70" as="geometry"/></mxCell><mxCell id="runtime" value="runtime.py + stage_loader.py
Context and stage dispatch" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="380" y="220" width="210" height="70" as="geometry"/></mxCell><mxCell id="storage" value="storage.py + models.py
SQLAlchemy and PgVector" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="680" y="90" width="220" height="70" as="geometry"/></mxCell><mxCell id="llm" value="llm.py
Provider client and retries" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="680" y="220" width="220" height="70" as="geometry"/></mxCell><mxCell id="e1" value="authorize" edge="1" parent="1" source="api" target="auth"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" value="queue" edge="1" parent="1" source="api" target="orch"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" value="dispatch" edge="1" parent="1" source="orch" target="runtime"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e4" value="persist" edge="1" parent="1" source="runtime" target="storage"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e5" value="complete" edge="1" parent="1" source="runtime" target="llm"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>
|
||||
|
|
@ -0,0 +1 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="container" name="Container"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="api" value="FastAPI API
Internal query, answer, and job routes" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="460" y="80" width="240" height="80" as="geometry"/></mxCell><mxCell id="worker" value="Worker
Claims and executes durable runs" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="460" y="230" width="240" height="80" as="geometry"/></mxCell><mxCell id="db" value="PostgreSQL + PgVector
Checkpoints, graph, vectors, HNSW" style="shape=cylinder;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366" vertex="1" parent="1"><mxGeometry x="800" y="180" width="250" height="100" as="geometry"/></mxCell><mxCell id="stages" value="Numbered stage modules
Normalize through answer" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="100" y="155" width="240" height="80" as="geometry"/></mxCell><mxCell id="llm" value="OpenAI-compatible provider
Chat and embeddings" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="100" y="320" width="240" height="80" as="geometry"/></mxCell><mxCell id="e1" value="dispatch" edge="1" parent="1" source="api" target="worker"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" value="execute" edge="1" parent="1" source="worker" target="stages"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" value="persist/retrieve" edge="1" parent="1" source="stages" target="db"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e4" value="prompts/vectors" edge="1" parent="1" source="stages" target="llm"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e5" value="status and query" edge="1" parent="1" source="api" target="db"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>
|
||||
|
|
@ -0,0 +1 @@
|
|||
<mxfile host="app.diagrams.net" modified="2026-10-15T14:00:00.000Z" agent="Copilot" version="24.7.17"><diagram id="context" name="Context"><mxGraphModel><root><mxCell id="0"/><mxCell id="1" parent="0"/><mxCell id="kg" value="KnowledgeGraph GraphRAG" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf" vertex="1" parent="1"><mxGeometry x="520" y="220" width="240" height="80" as="geometry"/></mxCell><mxCell id="mcp" value="Web-Access MCP + Data-Store MCP
External source collection" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="80" y="220" width="260" height="80" as="geometry"/></mxCell><mxCell id="bff" value="Existing BFF
Authentication and policy" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="870" y="100" width="220" height="80" as="geometry"/></mxCell><mxCell id="llm" value="OpenAI-compatible provider
Chat and embeddings" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656" vertex="1" parent="1"><mxGeometry x="870" y="340" width="220" height="80" as="geometry"/></mxCell><mxCell id="e1" value="Versioned source handoff" edge="1" parent="1" source="mcp" target="kg"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e2" value="Signed internal JWT
and API calls" edge="1" parent="1" source="bff" target="kg"><mxGeometry relative="1" as="geometry"/></mxCell><mxCell id="e3" value="Prompts and embeddings" edge="1" parent="1" source="kg" target="llm"><mxGeometry relative="1" as="geometry"/></mxCell></root></mxGraphModel></diagram></mxfile>
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# Module reference
|
||||
|
||||
## Platform package
|
||||
|
||||
| Module | Responsibility |
|
||||
|---|---|
|
||||
| `config.py` | typed settings, environment parsing, model definition |
|
||||
| `models.py` | SQLAlchemy entities and checkpoint schema |
|
||||
| `storage.py` | engine/session, schema checks, persistence, vector index |
|
||||
| `sources.py` | strict JSONL and PostgreSQL source readers |
|
||||
| `llm.py` | OpenAI-compatible completion and embedding client |
|
||||
| `runtime.py` | `StageContext` construction and initialization |
|
||||
| `stage_loader.py` | numbered stage module lookup |
|
||||
| `pipeline.py` | compatibility facade for stage methods |
|
||||
| `orchestrator.py` | durable runs, claiming, resume, stale recovery, locks |
|
||||
| `auth.py` | JWT verification and permission masks |
|
||||
| `api.py` | FastAPI app, request models, routes, authorization |
|
||||
| `worker.py` | polling worker and one-run execution |
|
||||
| `cli.py` | stage and durable-run command line entry points |
|
||||
| `observability.py` | structured JSON logging |
|
||||
|
||||
## Numbered stages
|
||||
|
||||
| Module | Main responsibility |
|
||||
|---|---|
|
||||
| `01_normalize_content.py` | validate handoff, extract text, persist raw/normalized documents |
|
||||
| `02_clean_content.py` | clean, deduplicate, and chunk |
|
||||
| `03_define_knowledge_model.py` | persist entity/relationship vocabulary and templates |
|
||||
| `04_extract_knowledge.py` | call LLM, validate extraction, persist per-chunk results |
|
||||
| `05_reconcile_knowledge.py` | canonicalize names and resolve cross-chunk relationships |
|
||||
| `06_build_knowledge_graph.py` | materialize nodes and edges |
|
||||
| `07_build_communities.py` | detect communities and summarize them |
|
||||
| `08_build_retrieval_index.py` | embed chunks/graph/community items and build HNSW |
|
||||
| `09_query_graphrag.py` | vector retrieval, graph expansion, source resolution |
|
||||
| `10_generate_answer.py` | generate answer with source citations |
|
||||
|
||||
Each numbered module exposes a `run` function and command-line adapter. Stages
|
||||
1-5 also expose focused validation/transformation helpers; stage 9 exposes
|
||||
retrieval helpers such as cosine ranking and graph expansion.
|
||||
|
||||
## Entry points
|
||||
|
||||
- `knowledgegraph` -> `graphrag_platform.cli:main_all`
|
||||
- `knowledgegraph-worker` -> `graphrag_platform.worker:main`
|
||||
- `uvicorn graphrag_platform.api:app`
|
||||
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
# Operations and troubleshooting
|
||||
|
||||
## Health and readiness
|
||||
|
||||
- `/internal/knowledge-graph/health` checks process liveness.
|
||||
- `/internal/knowledge-graph/ready` checks database and PgVector readiness.
|
||||
- `podman compose ps` shows service and health state.
|
||||
- `podman compose logs --follow api worker` streams structured JSON logs.
|
||||
|
||||
Logs include event names, run IDs, corpus/stage context, attempts, status, and
|
||||
error details. Secrets are not intended to be logged.
|
||||
|
||||
## Common failures
|
||||
|
||||
### Migration does not complete
|
||||
|
||||
Check PostgreSQL health, credentials, and `DATABASE_URL`. Confirm the PgVector
|
||||
image is used and inspect:
|
||||
|
||||
```powershell
|
||||
podman compose logs migrate postgres
|
||||
alembic current
|
||||
```
|
||||
|
||||
Do not enable automatic schema creation in a shared environment.
|
||||
|
||||
### Source validation fails
|
||||
|
||||
Check schema version, required fields, ISO timestamp offset, unique
|
||||
`external_id`, and corpus. The error identifies the source file and line.
|
||||
|
||||
### LLM requests fail
|
||||
|
||||
Verify `LLM_BASE_URL`, key, model names, and embedding dimensions. 429 and
|
||||
transient server/network errors retry; authentication and validation errors fail
|
||||
immediately. Check provider quota and `Retry-After` behavior before increasing
|
||||
retry settings.
|
||||
|
||||
### Vector index or retrieval fails
|
||||
|
||||
Confirm the `vector` extension exists, the provider returns exactly
|
||||
`LLM_EMBEDDING_DIMENSIONS` values, and stage 8 completed for the selected
|
||||
corpus. SQLite cannot validate PgVector behavior.
|
||||
|
||||
### Job remains queued
|
||||
|
||||
Check worker logs and the corpus advisory lock. Another build may be active.
|
||||
The worker requeues stale runs after `WORKER_STALE_RUN_SECONDS`; set this above
|
||||
the maximum expected stage duration.
|
||||
|
||||
### Job fails after partial progress
|
||||
|
||||
Read the job and stage error, repair the source/configuration, then retry. Retry
|
||||
resumes after the last successful stage. Use a reset rebuild if partial side
|
||||
effects or changed model settings make the checkpoint unsafe.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- Only source schema `1.0` is accepted.
|
||||
- `ANALYTICS_DATABASE_URL` is configured but unused by the current pipeline.
|
||||
- Source table names are simple unqualified names.
|
||||
- Stage-level recovery is not an all-stage transaction.
|
||||
- Name normalization can raise on conflicting entity types.
|
||||
- LLM shape validation does not establish factual correctness.
|
||||
- Answer citations are prompt-enforced, not independently checked.
|
||||
- No observed API rate limiting or pagination middleware.
|
||||
- The migration set currently contains one initial revision.
|
||||
|
||||
## Operational safety
|
||||
|
||||
- Use read-only credentials for the external source database.
|
||||
- Keep API and worker on private network paths behind the BFF.
|
||||
- Rotate all development secrets before deployment.
|
||||
- Back up PostgreSQL before destructive resets or migration changes.
|
||||
- Treat `podman compose down --volumes` as destructive.
|
||||
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
# System overview
|
||||
|
||||
## Purpose
|
||||
|
||||
KnowledgeGraph GraphRAG turns externally collected documents into a
|
||||
source-grounded knowledge graph and vector retrieval index. It then combines
|
||||
semantic retrieval with bounded graph expansion to produce context and cited
|
||||
answers.
|
||||
|
||||
The collection boundary is deliberate: Web-Access MCP and Data-Store MCP
|
||||
discover and persist source material. This repository starts at their versioned
|
||||
handoff.
|
||||
|
||||
## End-to-end flow
|
||||
|
||||
```text
|
||||
external source handoff
|
||||
-> 1 normalize
|
||||
-> 2 clean, deduplicate, chunk
|
||||
-> 3 persist knowledge model
|
||||
-> 4 extract entities and relationships
|
||||
-> 5 reconcile aliases and duplicates
|
||||
-> 6 materialize nodes and edges
|
||||
-> 7 detect and summarize communities
|
||||
-> 8 embed and index
|
||||
-> 9 retrieve vector + graph + source context
|
||||
-> 10 generate cited answer
|
||||
```
|
||||
|
||||
Stages 1-8 persist durable checkpoints in PostgreSQL. A later stage can be
|
||||
re-run without repeating valid upstream work. A durable run records stage
|
||||
selection, attempts, configuration fingerprint, status, timestamps, and errors.
|
||||
|
||||
## Runtime boundary
|
||||
|
||||
The deployment contains:
|
||||
|
||||
1. PostgreSQL 16 with the PgVector extension.
|
||||
2. An Alembic migration job.
|
||||
3. An internal FastAPI service.
|
||||
4. A PostgreSQL-backed worker.
|
||||
5. An existing BFF that authenticates callers and enforces feature permissions.
|
||||
6. An OpenAI-compatible chat and embedding provider.
|
||||
|
||||
The API is an internal service and is not intended for direct customer access.
|
||||
|
||||
## Important design decisions
|
||||
|
||||
- PostgreSQL is the canonical checkpoint store.
|
||||
- PgVector performs server-side cosine search using a dimension-specific HNSW
|
||||
index.
|
||||
- SQLite is a local-test fallback with JSON embeddings and in-process search.
|
||||
- LLM calls happen outside long database transactions; validated chunk results
|
||||
are committed independently.
|
||||
- A PostgreSQL advisory lock permits one active build per corpus.
|
||||
- The worker can recover stale runs and resume after the last successful stage.
|
||||
- The BFF supplies a short-lived signed internal JWT; unsigned permission
|
||||
headers are not trusted.
|
||||
|
||||
## Non-goals and boundaries
|
||||
|
||||
- Crawling and source discovery are not implemented here.
|
||||
- The API does not provide customer-facing authentication or authorization.
|
||||
- Query and answer are not queued worker jobs.
|
||||
- Citation correctness is requested through the answer prompt, not independently
|
||||
verified.
|
||||
- Entity reconciliation is normalized-name based, not semantic entity matching.
|
||||
|
||||
|
|
@ -0,0 +1,131 @@
|
|||
# Step-by-step runbook
|
||||
|
||||
Commands use PowerShell on Windows and assume the repository root is current.
|
||||
|
||||
## 1. Prepare tools and environment
|
||||
|
||||
Prerequisites: Python 3.11+, Podman 5+, a running `podman machine`, and an
|
||||
OpenAI-compatible chat/embedding endpoint.
|
||||
|
||||
```powershell
|
||||
Copy-Item .env.example .env
|
||||
py -3.12 -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -e ".[test]"
|
||||
```
|
||||
|
||||
Edit `.env`: set `POSTGRES_PASSWORD`, matching `DATABASE_URL` credentials,
|
||||
`LLM_API_KEY`, model names/dimensions, and a random
|
||||
`INTERNAL_JWT_SECRET` of at least 32 bytes.
|
||||
|
||||
## 2. Start the stack
|
||||
|
||||
```powershell
|
||||
podman machine start
|
||||
podman compose up --detach --build
|
||||
podman compose ps
|
||||
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/health
|
||||
Invoke-RestMethod http://127.0.0.1:8080/internal/knowledge-graph/ready
|
||||
```
|
||||
|
||||
Compose starts PostgreSQL, applies migrations, and then starts API and worker.
|
||||
The migration service must complete before the application services start.
|
||||
|
||||
## 3. Supply a source handoff
|
||||
|
||||
Place a UTF-8 JSONL file in `.\artifacts`, or configure a PostgreSQL source
|
||||
table. Validate it against the [source contract](./source-contract.md). For a
|
||||
file handoff, use `.\artifacts\mcp-documents.jsonl`.
|
||||
|
||||
## 4. Run a durable build
|
||||
|
||||
```powershell
|
||||
knowledgegraph run `
|
||||
--corpus product-knowledge `
|
||||
--from-stage 1 `
|
||||
--through-stage 8 `
|
||||
--input-jsonl .\artifacts\mcp-documents.jsonl `
|
||||
--model-file .\src\templates\knowledge_model.json `
|
||||
--requested-by local-developer
|
||||
```
|
||||
|
||||
Watch progress:
|
||||
|
||||
```powershell
|
||||
podman compose logs --follow api worker
|
||||
```
|
||||
|
||||
## 5. Query and answer
|
||||
|
||||
For direct local stage execution:
|
||||
|
||||
```powershell
|
||||
python .\src\09_query_graphrag.py `
|
||||
--corpus product-knowledge `
|
||||
--question "Who created Atlas?" `
|
||||
--top-k 8
|
||||
|
||||
python .\src\10_generate_answer.py `
|
||||
--corpus product-knowledge `
|
||||
--question "Who created Atlas?"
|
||||
```
|
||||
|
||||
For the API, obtain a BFF-issued JWT and call `/query` or `/answer` as described
|
||||
in the [API reference](./api.md).
|
||||
|
||||
## 6. Rebuild downstream data
|
||||
|
||||
Use `--from-stage` to avoid repeating valid upstream work. Use `--reset` when
|
||||
the selected stage or any downstream data is inconsistent:
|
||||
|
||||
```powershell
|
||||
knowledgegraph run `
|
||||
--corpus product-knowledge `
|
||||
--from-stage 4 `
|
||||
--through-stage 8 `
|
||||
--reset `
|
||||
--requested-by rebuild
|
||||
```
|
||||
|
||||
Do not run two builds for the same corpus concurrently; the PostgreSQL advisory
|
||||
lock will defer one, but coordinating callers is clearer.
|
||||
|
||||
## 7. Apply or inspect migrations
|
||||
|
||||
```powershell
|
||||
alembic current
|
||||
alembic history
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
`AUTO_CREATE_SCHEMA=true` is for isolated tests only. Review generated
|
||||
migrations before applying them.
|
||||
|
||||
## 8. Run tests
|
||||
|
||||
```powershell
|
||||
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
|
||||
```
|
||||
|
||||
For the real PgVector integration:
|
||||
|
||||
```powershell
|
||||
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
|
||||
pytest .\tests\integration\test_pgvector_postgres.py
|
||||
Remove-Item Env:TEST_POSTGRES_URL
|
||||
```
|
||||
|
||||
## 9. Stop or reset local services
|
||||
|
||||
```powershell
|
||||
podman compose down
|
||||
```
|
||||
|
||||
This retains the database volume. To delete all local PostgreSQL data:
|
||||
|
||||
```powershell
|
||||
podman compose down --volumes
|
||||
```
|
||||
|
||||
Volume removal is destructive.
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
# External source contract
|
||||
|
||||
## JSON Lines
|
||||
|
||||
Each UTF-8 line must be a JSON object with schema version `1.0`.
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"external_id": "stable-source-record-id",
|
||||
"corpus": "product-knowledge",
|
||||
"url": "https://docs.example.com/article",
|
||||
"body": "<html><body><main>Article text</main></body></html>",
|
||||
"content_type": "text/html",
|
||||
"status_code": 200,
|
||||
"retrieved_at": "2026-10-13T18:00:00Z",
|
||||
"metadata": {
|
||||
"collector": "Web-Access",
|
||||
"stored_by": "Data-Store"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Required fields:
|
||||
|
||||
- `schema_version`: exactly `1.0`;
|
||||
- `external_id`: stable and unique within a corpus;
|
||||
- `url`;
|
||||
- `body`;
|
||||
- `retrieved_at`: ISO 8601 timestamp with an offset.
|
||||
|
||||
Defaulted fields:
|
||||
|
||||
- `content_type`: `text/html`;
|
||||
- `status_code`: `200`;
|
||||
- `metadata`: `{}`.
|
||||
|
||||
`corpus`, when provided, must equal the selected runtime corpus. Unknown fields
|
||||
and unsupported schema versions are rejected. Validation errors include the
|
||||
file and line number. See the canonical template at
|
||||
[`src/templates/mcp_documents.jsonl`](../src/templates/mcp_documents.jsonl) and
|
||||
the acceptance fixture at [`tests/fixtures/external_documents.jsonl`](../tests/fixtures/external_documents.jsonl).
|
||||
|
||||
## PostgreSQL source table
|
||||
|
||||
Set `SOURCE_DATABASE_URL` and `SOURCE_TABLE_NAME` to use a PostgreSQL handoff.
|
||||
The table must contain:
|
||||
|
||||
```sql
|
||||
CREATE TABLE external_documents (
|
||||
external_id text PRIMARY KEY,
|
||||
schema_version text NOT NULL DEFAULT '1.0',
|
||||
corpus text,
|
||||
url text NOT NULL,
|
||||
body text NOT NULL,
|
||||
content_type text NOT NULL DEFAULT 'text/html',
|
||||
status_code integer NOT NULL DEFAULT 200,
|
||||
retrieved_at timestamptz NOT NULL,
|
||||
metadata jsonb NOT NULL DEFAULT '{}'::jsonb
|
||||
);
|
||||
```
|
||||
|
||||
Additional columns are ignored. Give the application account read-only access.
|
||||
Use either `--input-jsonl` or `SOURCE_DATABASE_URL`, not both.
|
||||
|
||||
## Normalization behavior
|
||||
|
||||
Stage 1 stores raw provenance and a normalized document. HTML is extracted with
|
||||
Trafilatura; non-HTML content is stripped directly. Stage 2 normalizes
|
||||
whitespace, detects duplicate content by SHA-256 hash, and creates overlapping
|
||||
chunks.
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Testing and validation
|
||||
|
||||
## Test layers
|
||||
|
||||
| Layer | Coverage |
|
||||
|---|---|
|
||||
| Unit | settings, source validation, chunk constraints, reconciliation, provider retry policy |
|
||||
| Integration | all ten stages, API/worker jobs, repair/retry/resume |
|
||||
| PostgreSQL integration | migrations, PgVector extension, HNSW index, graph retrieval, advisory lock |
|
||||
| Performance | configurable vector query load against real PostgreSQL |
|
||||
|
||||
The deterministic `FakeLLM` in `tests/conftest.py` keeps portable tests
|
||||
repeatable. It is not a quality or latency substitute for a production model.
|
||||
|
||||
## Local validation
|
||||
|
||||
```powershell
|
||||
pytest --ignore=.\tests\integration\test_pgvector_postgres.py
|
||||
python -m compileall -q src tests migrations
|
||||
python -m pip check
|
||||
```
|
||||
|
||||
The CI workflow also validates Compose configuration and builds the
|
||||
`Containerfile` image.
|
||||
|
||||
## PostgreSQL integration
|
||||
|
||||
Set `TEST_POSTGRES_URL` to a PgVector-enabled database. The migration test
|
||||
resets the schema, creates the external source table, runs stages 1-9, checks
|
||||
the vector extension and HNSW definition, and verifies advisory-lock behavior.
|
||||
|
||||
## Performance benchmark
|
||||
|
||||
The benchmark is skipped unless both `TEST_POSTGRES_URL` and
|
||||
`RUN_PERFORMANCE_TESTS=1` are set:
|
||||
|
||||
```powershell
|
||||
$env:TEST_POSTGRES_URL = "postgresql+psycopg://knowledgegraph_app:password@localhost:5432/knowledgegraph"
|
||||
$env:RUN_PERFORMANCE_TESTS = "1"
|
||||
$env:PERF_VECTOR_ITEM_COUNT = "10000"
|
||||
$env:PERF_MAX_QUERY_SECONDS = "2.0"
|
||||
pytest .\tests\performance\test_vector_query_load.py -m performance
|
||||
```
|
||||
|
||||
The benchmark inserts deterministic three-dimensional vectors and measures
|
||||
stage 9 retrieval. Results depend on hardware, database configuration, and load.
|
||||
|
||||
Loading…
Reference in New Issue