3.5 KiB
Architecture
Logical architecture
Web-Access/Data-Store MCP
|
| JSONL file or PostgreSQL source table
v
StageContext + numbered stages
|
+--> PostgreSQL checkpoint tables
+--> PgVector retrieval_items + HNSW index
+--> structured JSON logs and artifacts
|
+--> OpenAI-compatible chat/embedding provider
BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages
|
+--> query/answer stages
See the editable C4 context diagram and C4 container diagram for the system boundary.
Stage contracts
| Stage | Reads | Writes | Result |
|---|---|---|---|
| 1 | JSONL or source table | raw_documents, documents |
normalized documents |
| 2 | documents | chunks |
cleaned, deduplicated chunks |
| 3 | settings/model file | knowledge_models |
entity and relationship vocabulary |
| 4 | chunks, model | extractions |
validated LLM extraction payloads |
| 5 | extractions | resolved_entities, resolved_relationships |
canonical entities and links |
| 6 | resolved data | nodes, edges |
graph materialization |
| 7 | graph | communities |
detected communities and LLM summaries |
| 8 | chunks, graph, communities | search_items |
embeddings and HNSW index |
| 9 | search index, graph, documents | none | vector context, graph context, sources |
| 10 | stage 9 result | none | cited answer text and sources |
Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds
the question, retrieves top_k items, expands graph neighbors for the
configured number of hops and limit, and resolves source URLs.
Durable execution
pipeline_runs and stage_runs are separate from stage data. A run can be:
queued -> running -> succeeded
\-> failed -> queued (retry)
The worker:
- initializes and verifies the schema;
- requeues stale runs older than
WORKER_STALE_RUN_SECONDS; - claims one queued run using database locking;
- takes a per-corpus PostgreSQL advisory lock;
- executes the selected stage range;
- persists stage status and outputs;
- releases the lock.
If a corpus lock is unavailable, execution is deferred and the job remains
queued. Retry resumes after the last successful stage unless the caller uses
--reset or otherwise clears downstream state.
Deployment topology
compose.yaml starts PostgreSQL, waits for its health check, runs the one-shot
migrate service, then starts api and worker. The API and worker share the
application environment and bind-mount ./artifacts into /app/artifacts.
The C4 container diagram shows these containers and external dependencies. The component diagram maps the application package, and the code diagram maps the numbered stage pipeline.
Failure and consistency model
- Source records are rejected before they become normalized documents if their schema, required fields, or corpus are invalid.
- A completed stage is a checkpoint, not a transaction spanning the whole build.
- A failed stage may leave partial stage side effects; reset the affected stage and downstream state when a clean rebuild is required.
- PostgreSQL is required to exercise PgVector and HNSW behavior.
- Embedding vector length must equal
LLM_EMBEDDING_DIMENSIONS.