# Architecture ## Logical architecture ```text Web-Access/Data-Store MCP | | JSONL file or PostgreSQL source table v StageContext + numbered stages | +--> PostgreSQL checkpoint tables +--> PgVector retrieval_items + HNSW index +--> structured JSON logs and artifacts | +--> OpenAI-compatible chat/embedding provider BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages | +--> query/answer stages ``` See the editable [C4 context diagram](./diagrams/c4-context.drawio) and [C4 container diagram](./diagrams/c4-container.drawio) for the system boundary. ## Stage contracts | Stage | Reads | Writes | Result | |---:|---|---|---| | 1 | JSONL or source table | `raw_documents`, `documents` | normalized documents | | 2 | documents | `chunks` | cleaned, deduplicated chunks | | 3 | settings/model file | `knowledge_models` | entity and relationship vocabulary | | 4 | chunks, model | `extractions` | validated LLM extraction payloads | | 5 | extractions | `resolved_entities`, `resolved_relationships` | canonical entities and links | | 6 | resolved data | `nodes`, `edges` | graph materialization | | 7 | graph | `communities` | detected communities and LLM summaries | | 8 | chunks, graph, communities | `search_items` | embeddings and HNSW index | | 9 | search index, graph, documents | none | vector context, graph context, sources | | 10 | stage 9 result | none | cited answer text and sources | Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds the question, retrieves `top_k` items, expands graph neighbors for the configured number of hops and limit, and resolves source URLs. ## Durable execution `pipeline_runs` and `stage_runs` are separate from stage data. A run can be: ```text queued -> running -> succeeded \-> failed -> queued (retry) ``` The worker: 1. initializes and verifies the schema; 2. requeues stale runs older than `WORKER_STALE_RUN_SECONDS`; 3. claims one queued run using database locking; 4. takes a per-corpus PostgreSQL advisory lock; 5. executes the selected stage range; 6. persists stage status and outputs; 7. releases the lock. If a corpus lock is unavailable, execution is deferred and the job remains queued. Retry resumes after the last successful stage unless the caller uses `--reset` or otherwise clears downstream state. ## Deployment topology `compose.yaml` starts PostgreSQL, waits for its health check, runs the one-shot `migrate` service, then starts `api` and `worker`. The API and worker share the application environment and bind-mount `./artifacts` into `/app/artifacts`. The [C4 container diagram](./diagrams/c4-container.drawio) shows these containers and external dependencies. The [component diagram](./diagrams/c4-component.drawio) maps the application package, and the [code diagram](./diagrams/c4-code.drawio) maps the numbered stage pipeline. ## Failure and consistency model - Source records are rejected before they become normalized documents if their schema, required fields, or corpus are invalid. - A completed stage is a checkpoint, not a transaction spanning the whole build. - A failed stage may leave partial stage side effects; reset the affected stage and downstream state when a clean rebuild is required. - PostgreSQL is required to exercise PgVector and HNSW behavior. - Embedding vector length must equal `LLM_EMBEDDING_DIMENSIONS`.