RAG-DEMO/docs/architecture.md

89 lines
3.5 KiB
Markdown

# Architecture
## Logical architecture
```text
Web-Access/Data-Store MCP
|
| JSONL file or PostgreSQL source table
v
StageContext + numbered stages
|
+--> PostgreSQL checkpoint tables
+--> PgVector retrieval_items + HNSW index
+--> structured JSON logs and artifacts
|
+--> OpenAI-compatible chat/embedding provider
BFF -- signed JWT --> FastAPI API --> orchestrator --> worker --> stages
|
+--> query/answer stages
```
See the editable [C4 context diagram](./diagrams/c4-context.drawio) and
[C4 container diagram](./diagrams/c4-container.drawio) for the system boundary.
## Stage contracts
| Stage | Reads | Writes | Result |
|---:|---|---|---|
| 1 | JSONL or source table | `raw_documents`, `documents` | normalized documents |
| 2 | documents | `chunks` | cleaned, deduplicated chunks |
| 3 | settings/model file | `knowledge_models` | entity and relationship vocabulary |
| 4 | chunks, model | `extractions` | validated LLM extraction payloads |
| 5 | extractions | `resolved_entities`, `resolved_relationships` | canonical entities and links |
| 6 | resolved data | `nodes`, `edges` | graph materialization |
| 7 | graph | `communities` | detected communities and LLM summaries |
| 8 | chunks, graph, communities | `search_items` | embeddings and HNSW index |
| 9 | search index, graph, documents | none | vector context, graph context, sources |
| 10 | stage 9 result | none | cited answer text and sources |
Stage 8 indexes chunks, nodes, edges, and community summaries. Stage 9 embeds
the question, retrieves `top_k` items, expands graph neighbors for the
configured number of hops and limit, and resolves source URLs.
## Durable execution
`pipeline_runs` and `stage_runs` are separate from stage data. A run can be:
```text
queued -> running -> succeeded
\-> failed -> queued (retry)
```
The worker:
1. initializes and verifies the schema;
2. requeues stale runs older than `WORKER_STALE_RUN_SECONDS`;
3. claims one queued run using database locking;
4. takes a per-corpus PostgreSQL advisory lock;
5. executes the selected stage range;
6. persists stage status and outputs;
7. releases the lock.
If a corpus lock is unavailable, execution is deferred and the job remains
queued. Retry resumes after the last successful stage unless the caller uses
`--reset` or otherwise clears downstream state.
## Deployment topology
`compose.yaml` starts PostgreSQL, waits for its health check, runs the one-shot
`migrate` service, then starts `api` and `worker`. The API and worker share the
application environment and bind-mount `./artifacts` into `/app/artifacts`.
The [C4 container diagram](./diagrams/c4-container.drawio) shows these
containers and external dependencies. The [component diagram](./diagrams/c4-component.drawio)
maps the application package, and the [code diagram](./diagrams/c4-code.drawio)
maps the numbered stage pipeline.
## Failure and consistency model
- Source records are rejected before they become normalized documents if their
schema, required fields, or corpus are invalid.
- A completed stage is a checkpoint, not a transaction spanning the whole build.
- A failed stage may leave partial stage side effects; reset the affected stage
and downstream state when a clean rebuild is required.
- PostgreSQL is required to exercise PgVector and HNSW behavior.
- Embedding vector length must equal `LLM_EMBEDDING_DIMENSIONS`.